fix!: check if span has the attributes method to avoid internal error

[xuan-cao-swi]

Description

In rare cases, the span may be a no-op span because it is a non-recording span. If the span attempts to access its attributes, it will raise an error. A check has been added to prevent this internal error.

[robertlaurin]

Probably can just check if the span is recording, right?

Suggested change

	if span.respond_to?(:attributes) && !span.attributes.key?(OpenTelemetry::SemanticConventions::Trace::HTTP_STATUS_CODE)
	if span.recording? && !span.attributes.key?(OpenTelemetry::SemanticConventions::Trace::HTTP_STATUS_CODE)

[robertlaurin]

How is this span attribute getting set anyways? Instrumentation should really not be making any decisions based of the contents of the spans attributes.

[xuan-cao-swi]

aws lambda instrumentation attribute is set through otel_attributes(event, context) from in_span; and then it will add the attribute HTTP_STATUS_CODE.

Instrumentation should really not be making any decisions based of the contents of the spans attributes.

This part only sets attributes like HTTP_STATUS_CODE as part of span metadata.

The issue is that if the span is not recording, it won't have the attributes method, which can raise an error. This error could overshadow the user's original Lambda function error, causing confusion.

[spongenee]

thank you for this PR @xuan-cao-swi! This error causes lambda to fail when using the otel lambda layer detailed here. In our case, the request contains traceparent header that says no sampling and the lambda fails and api gateway returns 502.

[kaylareopelle]

Hi @spongenee, thanks for reaching out! Have you given this branch a try? Does it resolve the problem for you?

[spongenee]

Hi @spongenee, thanks for reaching out! Have you given this branch a try? Does it resolve the problem for you?

I would love to give a try but don't know how. I am currently using the ruby lambda layer, can I build a layer with this branch?

[kaylareopelle]

Hi @spongenee, thanks for reaching out! Have you given this branch a try? Does it resolve the problem for you?

I would love to give a try but don't know how. I am currently using the ruby lambda layer, can I build a layer with this branch?

Yes! You can build a layer with this branch. If you clone the opentelemetry-lambda repository, you can update the Gemfile to install the gem explicitly with a link to this branch.

For example, below the gem 'opentelemetry-instrumentation-all' line, you could write:

gem 'opentelemetry-instrumentation-aws_lambda', github: 'xuan-cao-swi/opentelemetry-ruby-contrib', glob: 'instrumentation/aws_lambda/*.gemspec', branch: 'lambda-inst-fix'

After that, bundle install and then cd out to the ruby/src directory and run ./build.sh. Docker needs to be running for the build script to work.

I'm not sure if it was just a problem with my machine, but I also needed to update lines 58-60 of the Dockerfile to include -rf after the rm commands:

# rm gem cache
RUN rm -rf /root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/cache/* \
    && rm -rf /root/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/cache/* \
    && rm -rf /root/.rbenv/versions/3.4.0/lib/ruby/gems/3.4.0/cache/*

Once the script finishes, you should have a new build directory appear under ruby/src with a zip file (opentelemetry-ruby-layer.zip). you can install.

If that's all more work than you're up for, don't worry about it 🙂

[spongenee]

thank you very much for the detailed instructions @kaylareopelle! I am able to follow the steps but I get the following error when triggering the function with the layer

"Could not find 'opentelemetry-instrumentation-aws_lambda' (~> 0.3.0) among 537 total gem(s)\nChecked in 'GEM_PATH=/var/task/vendor/bundle/ruby/3.2.0:/opt/ruby/gems/3.2.0:/var/runtime:/var/runtime/ruby/3.2.0' at: /opt/ruby/gems/3.2.0/specifications/opentelemetry-instrumentation-all-0.80.0.gemspec, execute `gem env` for more information"

I think it makes sense because the following line in the gemfile probably checks the version of 'opentelemetry-instrumentation-aws_lambda' but we are installing from branch

gem 'opentelemetry-instrumentation-all', '~> 0.80.0'

[xuan-cao-swi]

Hi @spongenee , one way (very hacky way) is that you can edit this file directly, and then pack everything into zip file. With the zip file, you can upload to layer.

After download the otel-lambda, you can navigate to the path cd ruby/src, and then run sam build -u -t template.yml (change the ruby version in template.yaml and otel/layer/Makefile if needed). You will have the .aws-sam folder. That folder is basically your layer content, and you can modify code there (e.g. instrumentation-aws_lambda). Finally, you can run ./zip_ruby_layer.sh to get the zip file opentelemetry-ruby-layer.zip.