Skip to content

Fix block scalar chomping edge case for EOF without newline - #2454

Open
jonasfj wants to merge 2 commits into
dart-lang:mainfrom
jonasfj:fix-L24T-1
Open

Fix block scalar chomping edge case for EOF without newline#2454
jonasfj wants to merge 2 commits into
dart-lang:mainfrom
jonasfj:fix-L24T-1

Conversation

@jonasfj

@jonasfj jonasfj commented Jul 6, 2026

Copy link
Copy Markdown
Member

The YAML 1.2.2 specification (Section 8.1.1.2: Block Chomping Indicator) defines the rules for handling final line breaks in block scalars. The production rule [165] b-chomped-last(t) dictates that when a block scalar terminates at the end of a file (<end-of-input>), the EOF itself acts identically to a physical line break (b-as-line-feed). Therefore, under the default "clip" chomping behavior, this implicit final line break must be retained as part of the string content.

Detailed specification references

The YAML 1.2.2 specification says about this in Section 8.1.1.2: Block Chomping Indicator.

The specification explicitly addresses how the final line break of a block scalar is handled, even if the file abruptly ends.

The Chomping Indicators

The spec defines the three possible chomping behaviors (production [164]):

[164]
c-chomping-indicator(STRIP) ::= '-'
c-chomping-indicator(KEEP)  ::= '+'
c-chomping-indicator(CLIP)  ::= ""

If you write |-, it's STRIP. If you write |+, it's KEEP. If you just write | (an empty string for the indicator), it defaults to CLIP.

The Final Line Break Rule

The spec explains how the very last character of the block scalar is evaluated (production [165]):

"The interpretation of the final line break of a block scalar is controlled by the chomping indicator specified in the block scalar header."

[165]
b-chomped-last(STRIP) ::= b-non-content  | <end-of-input>
b-chomped-last(CLIP)  ::= b-as-line-feed | <end-of-input>
b-chomped-last(KEEP)  ::= b-as-line-feed | <end-of-input>

Explanation of Rule [165]

The grammar for the default CLIP chomping is:
b-chomped-last(CLIP) ::= b-as-line-feed | <end-of-input>

This means that to successfully complete a block scalar under CLIP chomping, the parser must encounter EITHER:

  1. b-as-line-feed (a physical newline character like \n).
  2. <end-of-input> (the physical end of the file/stream).

Because they are both valid, parallel branches of the exact same production rule (b-chomped-last(CLIP)), the parser is mathematically required to treat <end-of-input> as functionally identical to a physical line feed for the purpose of concluding the string.

Implications for users

If a user writes:

foo: |
  text

And they omit the final trailing newline, the parser hits <end-of-input>. Because <end-of-input> fulfills the b-chomped-last(CLIP) requirement in the exact same way a physical line feed would, the parser must inject/retain the \n in the final parsed JSON (resulting in "text\n"), pretending the line feed was there.


Fix

  • Block scalar chomping at EOF: Fixed a bug where block scalars ending at EOF without a physical trailing newline failed to include the implicit trailing newline mandated by the YAML specification.
  • Implementation: leadingBreak now explicitly defaults to \n when the scanner hits EOF, ensuring the spec-mandated implicit final newline is preserved under default clip chomping rules.
  • Test Suite: Removes L24T-1 from _expectedFailures as it now passes.

Implications

  • Block scalars at the very end of files without a physical trailing newline will now correctly include their implicit final newline character.
  • Validation: Tested against 1,051,993 YAML files from pub.dev. Exactly 310 cases parsed differently (correctly gaining the previously missing implicit newline) with 0 new parsing failures.

View diffs for the 310 changed YAML files.

@jonasfj
jonasfj requested a review from a team as a code owner July 6, 2026 11:31
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Package publishing

If you have publishing permissions, you can use the links below to publish the changes after merging this PR.

Package Version Status Publish tag (post-merge)
package:api_summary 0.1.0-wip WIP (no publish necessary)
package:bazel_worker 1.1.5 already published at pub.dev
package:benchmark_harness 2.4.0 already published at pub.dev
package:boolean_selector 2.1.2 already published at pub.dev
package:browser_launcher 1.2.0-wip WIP (no publish necessary)
package:cli_config 0.2.1-wip WIP (no publish necessary)
package:cli_util 0.5.1 already published at pub.dev
package:clock 1.1.3-wip WIP (no publish necessary)
package:code_builder 4.12.0-wip WIP (no publish necessary)
package:coverage 1.15.1 already published at pub.dev
package:csslib 1.0.2 already published at pub.dev
package:extension_discovery 2.1.0 already published at pub.dev
package:file 7.0.2-wip WIP (no publish necessary)
package:file_testing 3.1.0-wip WIP (no publish necessary)
package:glob 2.1.3 already published at pub.dev
package:graphs 2.4.0-wip WIP (no publish necessary)
package:html 0.15.7-wip WIP (no publish necessary)
package:io 1.1.0-wip WIP (no publish necessary)
package:json_rpc_2 4.1.0 already published at pub.dev
package:markdown 7.4.0 ready to publish markdown-v7.4.0
package:mime 2.1.0-wip WIP (no publish necessary)
package:oauth2 2.0.5 already published at pub.dev
package:package_config 3.0.0 already published at pub.dev
package:pool 1.5.3-wip WIP (no publish necessary)
package:process 5.0.5 (error) pubspec version (5.0.5) and changelog (5.0.6-wip) don't agree
package:pub_semver 2.2.0 already published at pub.dev
package:pubspec_parse 1.6.0-wip WIP (no publish necessary)
package:source_map_stack_trace 2.1.3-wip WIP (no publish necessary)
package:source_maps 0.10.14-wip WIP (no publish necessary)
package:source_span 1.10.2 already published at pub.dev
package:sse 4.2.1-wip WIP (no publish necessary)
package:stack_trace 1.12.2-wip (error) pubspec version (1.12.2-wip) and changelog (1.12.2-dev) don't agree
package:stream_channel 2.1.4 already published at pub.dev
package:stream_transform 2.1.2-wip WIP (no publish necessary)
package:string_scanner 1.4.2-wip WIP (no publish necessary)
package:term_glyph 1.2.3-wip WIP (no publish necessary)
package:test_reflective_loader 0.6.0 ready to publish test_reflective_loader-v0.6.0
package:timing 1.0.2 already published at pub.dev
package:unified_analytics 8.0.16-wip WIP (no publish necessary)
package:watcher 1.2.2-wip WIP (no publish necessary)
package:yaml 3.1.4-wip WIP (no publish necessary)
package:yaml_edit 2.2.4 already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

PR Health

Unused Dependencies ✔️
Package Status
yaml ✔️ All dependencies utilized correctly.

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
yaml None 3.1.3 3.1.4-wip 3.1.4-wip ✔️

This check can be disabled by tagging the PR with skip-breaking-check.

Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Coverage ✔️
File Coverage
pkgs/yaml/lib/src/scanner.dart 💚 97 % ⬆️ 0 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

License Headers ✔️
// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

This check can be disabled by tagging the PR with skip-license-check.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

@jonasfj

jonasfj commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

From reading the changed YAML files on pub.dev, I'd suggest we have:

  • build.yaml, ffigen.yaml where preamble gets an extra newline
  • many .github/ files which we wouldn't be primary readers of anyway
  • a few test files that were written in YAML, which depending on how they are used could
    cause problems.
  • Some scripts and similar written in YAML where script: "dart pub get && dart test\n" gets what is hopefully a harmless newline.

@jonasfj
jonasfj force-pushed the fix-L24T-1 branch 2 times, most recently from ba51b74 to 5a3c6e6 Compare July 6, 2026 11:52
@jonasfj

jonasfj commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Added skip-leaking-check because there are no public API changes.

@@ -1190,7 +1190,11 @@ class Scanner {
// libyaml always reads a line here, but this breaks on block scalars at
// the end of the document that end without newlines. See example 8.1:
// http://yaml.org/spec/1.2/spec.html#id2793888.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does comment still apply?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants