[cloudflare_logpush] ingest pipeline improments#3
Open
brijesh-elastic wants to merge 14 commits into
Open
Conversation
…tions Sort all fields.yml entries alphabetically across all data streams for better maintainability. Add support for new fields across device_posture, gateway_dns, gateway_http, gateway_network, http_request, network_analytics, network_session, workers_trace, firewall_event, and email_security_alerts data streams. Fix swapped field descriptions for firewall_event (origin.ray.id/origin.response.status) and http_request (cache.status/cache.response.status). Align header naming between pipeline and fields.yml (singular header to plural headers for RequestHeaders/ResponseHeaders in http_request).
Correct the Painless script to reference ctx.json.Timestamp (PascalCase) instead of ctx.json.timestamp, matching the actual field name from the Cloudflare API and the guard condition.
Fix the grok guard condition that used an incorrect path (ctx.json?.cloudflare_logpush) instead of (ctx.cloudflare_logpush) and a tautological || operator instead of &&. Also correct the remove processor to reference action instead of event_action. Update test data to use a valid disconnect timestamp.
Correct the split processor condition to reference ctx.json.TCPSackBlocks consistently instead of mixing TCPSACKBlocks and TCPSackBlocks casing.
Correct the rename condition to use ctx.json?.Interface (PascalCase) matching the actual Cloudflare API field name instead of lowercase.
Replace rename processors with convert processors (type: string) for fields documented as integers or arrays of integers but mapped as keyword type in fields.yml. Affected fields: gateway_dns (CNAMECategoryIDs, EDEErrors, InitialCategoryIDs, MatchedIndicatorFeedIDs, ResolvedIPCategoryIDs), gateway_http (ApplicationIDs), gateway_network (ApplicationIDs, CategoryIDs).
Remove ignore_failure: true from the first JSON processor in all data stream pipelines that had it. Parsing failures should surface as errors rather than silently producing partial documents.
Replace grok processors with equivalent dissect processors in firewall_event (protocol parsing), http_request (protocol and TLS parsing), and spectrum_event (TLS parsing). Dissect is faster than grok for simple delimiter-based patterns.
removal script, standardize error.message format, and add processor tags Consolidate multiple timestamp-to-Unix-millis script processors into a single, efficient script across all data streams. Update test input log files to include varied timestamp format test cases. Add missing convert processors to ensure fields declared as ip, long, boolean, or double in fields.yml are correctly typed at ingest time. Implement the latest null removal script (handleMap/handleList) across all data streams for consistent cleanup of null and empty values. Update error.message values in on_failure blocks to use the full standardized format including processor type, tag, pipeline, and message. Add a tag key to every processor across all data streams for easier debugging and error tracing. Align http_request header naming between pipeline and fields.yml (singular header to plural headers for RequestHeaders/ResponseHeaders).
dns_firewall data streams Add dynamic mapping for dns.response_code and dns.question.type to follow IANA keyword representations in dns and dns_firewall data streams. Also applies the same pipeline improvements as the previous commit (timestamp consolidation, convert processors, null removal script, standardized error.message, and processor tags) to these two data streams.
improvements for network_analytics Add ignore_failure: true to the community_id processor in the network_analytics pipeline to resolve build failures. Also applies the same pipeline improvements (timestamp consolidation, convert processors, null removal script, standardized error.message, and processor tags) to this data stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally