Systemd journal mapping#4995
Conversation
|
|
||
| | Field | Type | Description | Maps to Unified Model Field | | ||
| | ----- | ---- | ----------- | --------------------------- | | ||
| | `_SOURCE_REALTIME_TIMESTAMP` | uint64 | The earliest trusted timestamp of the message, if known, as CLOCK_REALTIME in microseconds since the Unix epoch. Optional. | Timestamp | |
There was a problem hiding this comment.
Since the PR description is sparse and there's no corresponding issue, I'd like to know a bit more about this. Specifically, whether this mapping is embodied in any code / component like the collector's journaldreceiver?
cc journald receiver code owners @belimawr, @namco1992
There was a problem hiding this comment.
Sorry I should have added some context
Currently this mapping is not implemented in any open source project, as far as I know.
I have a custom lua script that does it for fluent-bit which I would like toconvert to a C written processor and submit to fluentbit.
This PR is intended to come to define what a mapping from jorunald should look like before submitting code.
There was a problem hiding this comment.
@bachp https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver implements journald to Otel mapping. We cannot have the mapping description here to be different than what journaldreceiver implements. The implementation in Otel Collector should match what Otel spec says.
There are 2 possible ways forward:
- We document here what journaldreceiver implements.
- We adjust journaldreceiver implementation to match what is documented here.
There was a problem hiding this comment.
I looked at the implenetation in journaldreceiver. And as far as I understand it doesn't do any mapping except for __REALTIME_TIMESTAMP -> Timestamp.
All oder fields are just put into a Map inside the Body.
e.g.
LogRecord #2
ObservedTimestamp: 2026-04-01 09:01:45.855525618 +0000 UTC Modified Files
Timestamp: 2026-04-01 09:01:45.710521 +0000 UTC specification/logs/data-model-a +1 -2
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({
"MESSAGE":"OpenTelemetry journald demo test message",
"PRIORITY":"6",
"SYSLOG_FACILITY":"1",
"SYSLOG_IDENTIFIER":"pascal",
"SYSLOG_TIMESTAMP":"Apr 1 11:01:45 ",
"_HOSTNAME":"hostname",
[...]
})
So based on this I update the recommendatio with the following:
- Don't map
ObservedTimestamp - Align the
TimeStampsource with the implementation using__REALTIME_TIMESTAMPas fallback
The other extended mapping would then need to be implemented in journaldreceiver once agreed. But this would be a breaking change there.
There was a problem hiding this comment.
@bachp take a look at open-telemetry/opentelemetry-collector-contrib#46500 there is already an effort to map more fields to attributes consistent with otel.
There was a problem hiding this comment.
My advise is to speak with journaldreceiver codeowners to understand why they only did timestamp mapping and whether they are open to implementing mapping for additional fields that you are proposing in this PR.
There was a problem hiding this comment.
The proposal is now aligned with open-telemetry/opentelemetry-collector-contrib#46500
Only exception is the mapping of _COMM as discussed in open-telemetry/opentelemetry-collector-contrib#46500 (comment)
There was a problem hiding this comment.
Let's wait for open-telemetry/opentelemetry-collector-contrib#46500 to be merged.
There was a problem hiding this comment.
The journaldreceiver PR (open-telemetry/opentelemetry-collector-contrib#46500) looks good. In terms of ordering, I would rather merge this PR first as it is a sort of extension of the spec and then the implementation, given that both are aligned.
However this is not a strong option.
bachp
force-pushed
the
systemd-journal-mapping
branch
from
07e3550 to
18d9b0a
Compare
bachp
force-pushed
the
systemd-journal-mapping
branch
from
18d9b0a to
f35464d
Compare
|
Related discussion open-telemetry/opentelemetry-collector-contrib#7298 |
bachp
force-pushed
the
systemd-journal-mapping
branch
from
f35464d to
58b181f
Compare
There was a problem hiding this comment.
Blocking until open-telemetry/opentelemetry-collector-contrib#46500 is merged.
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
bachp
force-pushed
the
systemd-journal-mapping
branch
from
58b181f to
5a2e298
Compare
| <td>string</td> | ||
| <td>Not well defined. May be used as a meta field for protocol operation purposes or may be part of event source identity.</td> | ||
| <td>`Attributes["syslog.procid"]`</td> | ||
| <td>`Attributes["syslog.pid"]`</td> |
There was a problem hiding this comment.
@belimawr @thompson-tomo I update the syslog section to reflect the new field naming for syslog.procid => syslog.pid and syslog.msgid => syslog.msg.id as discussed in open-telemetry/opentelemetry-collector-contrib#46500 (comment) and open-telemetry/opentelemetry-collector-contrib#46500 (comment)
|
This PR was marked stale. It will be closed in 14 days without additional activity. |
Its still active, waiting for feedback |
|
This PR was marked stale. It will be closed in 14 days without additional activity. |
|
Still waiting for feedback |
7 participants
Changes
Add example how to map Journald fields to OpenTelemetry to the Log Data Model Appendix.
CHANGELOG.mdfile updated for non-trivial changes[chore]in the PR title to skip the changelog check