[Bug] Workflow side signal with start does not work with TelemetryInterceptor

[robcao]

What are you really trying to do?

I'm trying to use workflow side signal with start while also using the TracingInterceptor, and the workflow task scheduling the Nexus operation fails due to the interceptor attaching a header to the command.

Describe the bug

It appears that system nexus endpoints don't accept headers

if attrs.Endpoint == commonnexus.SystemEndpoint {
		if len(attrs.NexusHeader) > 0 {
			return FailWorkflowTaskError{
				Cause:   enumspb.WORKFLOW_TASK_FAILED_CAUSE_BAD_SCHEDULE_NEXUS_OPERATION_ATTRIBUTES,
				Message: fmt.Sprintf("ScheduleNexusOperationCommandAttributes.NexusHeader must be empty when using %s endpoint", commonnexus.SystemEndpoint),
			}
		}

But the SDK has no such special treatment for system nexus endpoints, and sends system nexus endpoints through the interceptor chain like this

private class NexusWorkflowClientImpl : NexusWorkflowClient
        {
            private readonly WorkflowInstance instance;

            public NexusWorkflowClientImpl(WorkflowInstance instance, string service, NexusWorkflowClientOptions options)
            {
                this.instance = instance;
                Service = service;
                Options = options;
            }

            public override string Service { get; }

            public override NexusWorkflowClientOptions Options { get; }

            public override Task<NexusWorkflowOperationHandle<TResult>> StartNexusOperationAsync<TResult>(
                string operationName, object? arg, NexusWorkflowOperationOptions? options = null) =>
                instance.outbound.Value.ScheduleNexusOperationAsync<TResult>(new(
                    Service: Service,
                    ClientOptions: Options,
                    OperationName: operationName,
                    Arg: arg,
                    Options: options ?? new(),
                    Headers: null));
        }

The tracing interceptor attaches headers to most intercepted call sites, and is also attaching the tracing header to the system nexus endpoint, resulting in a failed workflow task like this

BadScheduleNexusOperationAttributes: ScheduleNexusOperationCommandAttributes.NexusHeader must be empty when using __temporal_system endpoint

Minimal Reproduction

Create a file like this in tests/Temporalio.Tests/Worker/SystemNexusEndpointInterceptorTests.cs, which contains a simple test case that schedules a workflow side signal with start operation while using a client with the TracingInterceptor.

Run the test.

Verify failure.

using OpenTelemetry;
using OpenTelemetry.Trace;
using Temporalio.Client.Interceptors;
using Temporalio.Extensions.OpenTelemetry;

namespace Temporalio.Tests.Worker;

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Temporalio.Api.Enums.V1;
using Temporalio.Client;
using Temporalio.Worker;
using Temporalio.Workflows;
using Xunit;
using Xunit.Abstractions;

[Collection("Environment")]
public class NexusSystemEndpointInterceptorTests : WorkflowEnvironmentTestBase
{
    public NexusSystemEndpointInterceptorTests(ITestOutputHelper output, WorkflowEnvironment env)
        : base(output, env)
    {
    }

    [Fact]
    public async Task SignalWithStartFromWorkflow_FailsWhenInterceptorAddsHeaderToSystemNexusOperation()
    {
        using var tracerProvider = Sdk.
            CreateTracerProviderBuilder().
            AddSource(TracingInterceptor.ClientSource.Name, TracingInterceptor.WorkflowsSource.Name, TracingInterceptor.ActivitiesSource.Name).
            AddOtlpExporter().
            Build();

        TemporalClientConnectOptions connectOptions = new()
        {
            TargetHost = Client.Connection.Options.TargetHost,
            Interceptors = new IClientInterceptor[] { new TracingInterceptor() },
        };

        TemporalClient client = await TemporalClient.ConnectAsync(connectOptions);

        var options = new TemporalWorkerOptions($"tq-{Guid.NewGuid()}")
            .AddWorkflow<SignalWithStartCallerWorkflow>()
            .AddWorkflow<SignalWithStartTargetWorkflow>();

        using var worker = new TemporalWorker(client, options);
        await worker.ExecuteAsync(async () =>
        {
            var handle = await client.StartWorkflowAsync(
                (SignalWithStartCallerWorkflow wf) => wf.RunAsync($"target-{Guid.NewGuid()}", worker.Options.TaskQueue!),
                new(id: $"caller-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));

            // The workflow task fails (and retries) rather than failing the workflow, so assert on the
            // task-failure event instead of waiting for a result that never arrives.
            await AssertMore.TaskFailureEventuallyAsync(handle, attrs =>
            {
                Assert.Equal(WorkflowTaskFailedCause.BadScheduleNexusOperationAttributes, attrs.Cause);
                Assert.Contains("__temporal_system", attrs.Failure.Message);
            });
        });
    }

    [Workflow]
    public class SignalWithStartTargetWorkflow
    {
        private readonly List<string> events = new();

        [WorkflowRun]
        public async Task<IReadOnlyCollection<string>> RunAsync(string value)
        {
            this.events.Add($"Started: {value}");
            await Workflow.WaitConditionAsync(() => this.events.Count >= 2);
            return this.events;
        }

        [WorkflowSignal]
        public Task SignalAsync(string value)
        {
            this.events.Add($"Signal: {value}");
            return Task.CompletedTask;
        }
    }

    [Workflow]
    public class SignalWithStartCallerWorkflow
    {
        [WorkflowRun]
        public async Task<string> RunAsync(string targetWorkflowId, string taskQueue)
        {
            var handle = await Workflow.SignalWithStartWorkflowAsync(
                (SignalWithStartTargetWorkflow wf) => wf.RunAsync("start-value"),
                wf => wf.SignalAsync("signal-one"),
                new(targetWorkflowId, taskQueue) { IdConflictPolicy = WorkflowIdConflictPolicy.UseExisting });
            return handle.Id;
        }
    }
}

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]
• Temporal Version: .NET SDK 1.17.0, Temporal CLI 1.7.2, Temporal CLI v1.7.2-standalone-nexus-operations
• Are you using Docker or Kubernetes or building Temporal from source? v1.7.2-standalone-nexus-operations

Additional context

Users still want to have the ability to intercept these operations, just as they can intercept scheduling a child workflow or signaling an external workflow today, so probably system nexus endpoints need to skip interceptors at the ScheduleNExusOperationAsync call site, but add new interceptor methods for things like SignalWithStartWorkflowAsync need to be added, with a special code path if it's a system nexus endpoint