LoggingInstrumentor and AWS Lambda Python RIC are duplicating logs #2417
Description
I'm not sure if this fits under Bug or Feature Request.
Describe your environment
AWS Lambda function with Otel Python lambda layer, Python 3.8 in x86_64.
Steps to reproduce
I have an AWS Lambda function equipped with the OpenTelemetry Python lambda layer 0.5.0 and have set the env var OTEL_PYTHON_LOG_CORRELATION: true. The function calls logger.getLogger(__name__) and logger.warning(<msg>), without adding any handlers. Invocation results in duplicate logs, for example:
2024-04-12 23:44:40,808 WARNING [root] [lambda_function.py:32] [trace_id=XXXX span_id=XXXX resource.service.name=lambda_function trace_sampled=XX] - My log inside lambda
[WARNING] 2024-04-12T23:44:40.808Z XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX My log inside lambda
The first log is what I expect for OTEL_PYTHON_LOG_CORRELATION (I've just X'd out the context IDs). The second log has a different format where the long ID is the AWS function invocation request ID. This comes from the aws-lambda-python-runtime-interface-client: this formatting by _setup_logging and always with a Lambda-specific handler. The AWS logging does not seem to be configurable.
What is the expected behavior?
Ideally: it would be great if there was a single set of logs that had the best of both kinds, something like:
2024-04-12 23:44:40,808 WARNING [root] [lambda_function.py:32] [request_id=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX] [trace_id=XXXX span_id=XXXX resource.service.name=lambda_function trace_sampled=XX] - My log inside lambda
What is the actual behavior?
Duplicate logs for the same messages, in 2 different formats.
Additional context
This might relate to how LoggingInstrumentor does a single logging.basicConfig call as others have already pointed out:
- Logging AutoInstrumentation does not support capturing INFO logs opentelemetry-python#3473
- OTEL_PYTHON_LOG_CORRELATION is mentioned in doc, but does not seem to work. #2346
Maybe the explicit setting of handler formats in this PR could add span context fields to the lambda runtime-set logging handler, by adaptor or filter: #1939