Unable to see structured logs in pubished app

[efonsecab]

I'm not being able to see the structured logs in the published app, they however show correctly locally.

Same is happening for traces and metrics.

This is the dashboard version

AppHost project

ServiceDefaults project

Log Streams are not showing any error, so no idea what may be causing it.

[mitchdenny]

If you publishing the aspire-starter template does it work for you? Just trying to narrow down whether we have a fundamental bug here or it is just something specific to your code.

[mitchdenny]

@efonsecab I am moving this issue over to discussions because I would be surprised if this is a bug given how often we hit this code path. If after some discussion it does end up being a bug we'll open an issue with the specifics of the bug.

[efonsecab]

@efonsecab I am moving this issue over to discussions because I would be surprised if this is a bug given how often we hit this code path. If after some discussion it does end up being a bug we'll open an issue with the specifics of the bug.

I think it has to do with my code since I recently upgraded some packages, though I still make sure to keep Aspire in 8.0.1, the main issue however is that the log streams are not showing any errors, and as far as I know the dashboard does not have information logs for itself, so I have no idea how to debug the issue other than guessing packages versions combinations.

[MrDeej]

Same problem for my. Logging/tracing/metrics works in localhost and my test environment. But not in my production envionment. And I dont find anything that explains why. I have no enviroment specific coding outside of the starter kit.

[kevcrooks]

This has started happening for me too. If I start a new Aspire project, the structured logs appear, but I have a project that has already been published and the structured logs have stopped.

I'm still getting console logs output, but nothing in structured logs or traces. Even running azd up from an older commit (that I believe used to work) doesn't fix the issue. Maybe a configuration problem in one of the containers?

Maybe I have to delete the entire resource group and re-publish from a new profile? But any suggestions to diagnose or fix when this happens to an existing Aspire application would be good.

[kevcrooks]

Just to confirm this, I created a new azd environment, and ran azd up - this created a new resource group in Azure with the dashboard, and the dashboard works perfectly fine.

However if I run azd up to the previous environment, the dashboard does not show the structured logs or traces.

So I think either the Aspire dashboard is being corrupted over time, or there ends up being a problem inside the resource group that cannot be reverted.

@mitchdenny any thoughts on this? I have both environments deployed so I can get more details, if you can let me know what to check?

[efonsecab]

I tried to reproduce it in a new app, no luck yet, however, comparing the Container Apps Environment logs I do see this, not sure if it is useful as a hint.

"ScaledObject doesn\u0027t have correct triggers specification"

This message appears only for the "Worker Service" projects

[mitchdenny]

@kvenkatrajan this might be an issue to raise the the ACA team. It seems like the problem is going to either be in the dashboard or in the resource server that ACA provides for the ACA-deployed dashboard to talk to.

[MrDeej]

Earlier I had logs in my test and dev environment. After I recreated the resource groups from scratch. Just had issue with production, but now all environments are without logs. I have not done anything, and the dev environment is just a clean Aspire starter kit solution with nothing.

I have a lot of logs in console

And the version differs from localhost:

Any tips on how to debug would be greatly appreciated.

[kvenkatrajan]

@mitchdenny sent it to ACA dev for investigation.

[snehapar9]

@MrDeej thanks for bringing this to our attention. This is a known issue, and we are working on its long-term fix. You were not seeing OTLP data because there was a mismatch in the OTLP Api key due to a race condition.
Can you please confirm you're seeing structured logs/metrics/traces now?

[efonsecab]

@snehapar9 is there any existent GitHub issue for this one?
Not sure about @MrDeej , but I do keep having the issue, tried re-deploying did not work, should I get latest daily build?

[MrDeej]

Hi @snehapar9 , thanks for response. No, I do not have any logs/metric/traces in any of my environments.

[efonsecab]

Today, out of nowhere, the structured logs started working, more or less, they began showing up for some of the projects in my solution, yet no all.

Resources

I did not change anything regarding configuration I just modified some existent pages and redeploy with azd up.

[snehapar9]

@efonsecab do all the apps deployed to your environment show up in the drop down? Do all these apps emit structured logs?

[efonsecab]

@efonsecab do all the apps deployed to your environment show up in the drop down? Do all these apps emit structured logs?

They used too a couple of months ago, but at some point they all stopped showing up, and it was until a couple of days ago that some of them started to show up, but not all.

[snehapar9]

@efonsecab can you check if traces/structured logs show up if you hit the app endpoint?

[efonsecab]

@efonsecab can you check if traces/structured logs show up if you hit the app endpoint?

The don't, it looks like this, I'm expecting the one named fairplaytube to show up, but it does not.

[MrDeej]

My test environment dashboard is working again after re-deploy, but production did not work after re-deploy and still says "No structured logs found"

[MrDeej]

Hello. Any news here? I am still missing my logs in my production environment.

[mitchdenny]

Had a chat with our Dashboard folks and partners in ACA and it looks like a fix is already rolling out but hasn't reached your region yet.

[snehapar9]

@MrDeej the fix is being rolled out now. Has not been deployed to Norway East yet.
It should be deployed in all regions by the end of this week.

[efonsecab]

Mine stopped showing up once again out of nowhere, apps are published in East US 2 Azure Region.

[MrDeej]

Today, all my dashboards are showing this one:

[efonsecab]

I can once again see everything in the Dashboard for the published apps.

[snehapar9]

@MrDeej Are you still seeing the 404 error?

[MrDeej]

Yes.

[snehapar9]

@MrDeej your container app may be scaled to 0. Can you please let me know if min replicas and max replicas are greater than 0?
Does the container app exist still?

[MrDeej]

I dont know wich container represents the dashboard, and I have 3 in my resource group.

These 3 containers look like this:

[snehapar9]

@MrDeej apologies for the delayed response. I looked at your dev, test and prod environments. The Aspire feature was disabled, and the underlying resources were deleted. Can you please re-enable it from the portal (in your container app environment page) and let me know if you can access the dashboard?

[Sebosek]

Hi, I have the same problem but with the local environment. In my case, I run just the Aspire Dashboard in the container next to the OpenTelemetry Collector with my Hello World App.

My docker-compose file:

version: '3.8'

services:
  boardgamesclub.backend:
    container_name: helloworld
    image: helloworld.backend
    build:
      context: ./backend/HelloWorld.Backend/HelloWorld.Backend
      dockerfile: Dockerfile
    environment:
      - ASPNETCORE_URLS=http://*:5000/
    ports:
      - "5000:5000"
    depends_on:
      - otel-collector
  
  otel-collector:
    container_name: otel-collector
    image: otel/opentelemetry-collector-contrib:latest
    command: [ "--config", "/etc/otel-collector-config.yaml" ]
    ports:
      - "4317:4317" # For gRPC
      - "8888:8888" # OpenTelemetry health check endpoint
    volumes:
      - ./backend/otel-collector-config.yml:/etc/otel-collector-config.yaml
  
  aspire:
    container_name: aspire
    image: mcr.microsoft.com/dotnet/aspire-dashboard:8.1.0
    ports:
      - "18888:18888" # For UI
      - "18889:18889"  # For OTLP protocol
    depends_on:
      - otel-collector

OpenTelemetry Conllector config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

  otlp:
    endpoint: "aspire:18889"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]

C# instrumentation

    public static IHostApplicationBuilder AddShared(this WebApplicationBuilder builder, IConfiguration configuration)
    {
        var serilog = new LoggerConfiguration();
        serilog.ReadFrom.Configuration(configuration);

        builder.Services.AddEndpointsApiExplorer();
        builder.Services.AddSwaggerGen();
        builder.Services.AddSerilog(serilog.CreateLogger());
        builder.Services.AddOptions<CorsConfig>()
            .Bind(configuration.GetSection(CorsConfig.KEY))
            .ValidateDataAnnotations()
            .ValidateOnStart();
        builder.Services.AddOptions<TelemetryConfig>()
            .Bind(configuration.GetSection(TelemetryConfig.KEY))
            .ValidateDataAnnotations()
            .ValidateOnStart();

        builder.Services.AddCors(setup =>
            setup.AddDefaultPolicy(policy =>
                policy.AddHostConfiguration(configuration)));

        var telemetry = configuration.GetSection(TelemetryConfig.KEY).Get<TelemetryConfig>()!;
        var serviceName = configuration.GetValue("ServiceName", defaultValue: "HelloWorld")!;
        var serviceVersion = typeof(Program).Assembly.GetName().Version?.ToString() ?? "dev";
        var serviceInstanceId = Environment.MachineName;

        builder.Logging
            .ClearProviders()
            .AddSerilog()
            .AddOpenTelemetry(logger =>
            {
                logger.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(
                    serviceName: serviceName,
                    serviceVersion: serviceVersion,
                    serviceInstanceId: serviceInstanceId));
                logger.IncludeScopes = true;
                logger.AddOtlpExporter(options =>
                {
                    options.Protocol = telemetry.Logging.Protocol;
                    options.Endpoint = telemetry.Logging.Endpoint;
                });
            });

        builder.Services
            .AddOpenTelemetry()
            .ConfigureResource(resource =>
                resource.AddService(
                    serviceName: serviceName,
                    serviceVersion: serviceVersion,
                    serviceInstanceId: serviceInstanceId))
            .WithTracing(tracing =>
            {
                tracing.AddAspNetCoreInstrumentation();
                tracing.AddHttpClientInstrumentation();
                tracing.AddSqlClientInstrumentation();
                tracing.AddOtlpExporter(options =>
                {
                    options.Protocol = telemetry.Tracing.Protocol;
                    options.Endpoint = telemetry.Tracing.Endpoint;
                });
            })
            .WithMetrics(metrics =>
            {
                metrics
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddMeter("Microsoft.AspNetCore.Hosting")
                    .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
                    .AddOtlpExporter(options =>
                    {
                        options.Protocol = telemetry.Metrics.Protocol;
                        options.Endpoint = telemetry.Metrics.Endpoint;
                    });
            });

        return builder;
    }

[davidfowl]

I don't think the problems are similar. The issues with the dashboard in azure containers apps are likely not the same as the ones you run into while running it locally.

[inlineHamed]

I have the same issue: on my local Windows machine, structured logs and metrics are available, but on my Debian 12 server, I have neither. I also tested the aspire-starter app on the server, but it does not show structured logs. Additionally, I tested it on my local WSL Debian environment, and there were no structured logs there either. so I think the issue is related to Debian/Linux.

version: 9.2.1

[davidfowl]

Did you setup https?

[inlineHamed]

That was the issue for me. thanks 🙏👍
it resolved by dotnet dev-certs https --trust