Add `pkl:syntax` module by stackoverflow · Pull Request #1569 · apple/pkl

stackoverflow · 2026-04-30T15:27:10Z

This new stdlib module allows for full manipulation of Pkl source code in Pkl: parsing text to a Pkl CST and formatting a Pkl CST back to text.

odenix · 2026-04-30T17:54:01Z

Have you considered offering the equivalent Java library? Wrapping that library instead of implementing pkl:syntax in Pkl would also improve (especially cold/warm) performance of pkl:syntax.

HT154 · 2026-04-30T18:39:30Z

The major thing missing from this PR is the ability to whole-cloth construct and then format SyntaxNode instances without laboriously creating the underlying Nodes, especially given how common and important code generation is for the Pkl ecosystem. I'm not opposed to offering a Java API (pklpoet?) for this as long as the SyntaxNode classes remain instantiable (not external).

stackoverflow · 2026-05-04T08:10:15Z

The Java library is already Node. We still need the classes to be implemented in Pkl because we don't want to offer a generic API to users. As Jen mentioned, we don't want the classes to be external.

odenix · 2026-05-22T23:21:37Z

As Jen mentioned, we don't want the classes to be external.

As far as I know, external members of non-external classes can still be implemented in Java.

If this stdlib module is implemented in Pkl, it should not be cached (as it is now), as this is very likely to cause multithreading issues.

stackoverflow · 2026-05-27T15:31:12Z

As Jen mentioned, we don't want the classes to be external.

As far as I know, external members of non-external classes can still be implemented in Java.

I don't think there would be much difference performance wise in implementing it in java. All the parsing and formatting is already in java. The Pkl classes are just a sugar layer.

If this stdlib module is implemented in Pkl, it should not be cached (as it is now), as this is very likely to cause multithreading issues.

What do you mean by cached?

bioball

Some initial comments

bioball · 2026-07-14T20:53:47Z

+    | "module_declaration"
+    | "module_definition"


This should probably be called "module_header" and "module_declaration", respectively.

Where, module_header is all of:

module Foo amends "bar"

and module_declaration is specifically the line that starts with module, e.g.

module Foo

As I mentioned some comments above, this is done to keep the Java and Pkl nodes in sync. Diverging them is possible but not simple.

Can we just change this in the Java parser too?

bioball · 2026-07-28T21:51:55Z

+  /// [span] is carried through unchanged unless set explicitly.
+  /// [parent] is populated on the returned tree for nodes originating from [Parser.parseModule];
+  /// nodes constructed from scratch retain their given `parent`.
+  external function walk(visit: (Node) -> Pair<Node, Boolean>?): Node


What's the difference between (node) -> null and (node) -> Pair(node, true)?

Also, maybe call this method "transform", and call the argument "operator".

Pair(node, true) means "I want to visit/process this node, but not its children". null means I don't want to visit/process this node, nor its children.
It's true you can just pass the node unchanged to the pair, but null makes this intent more explicit. Also, most calls to walk only want to process a single or couple things and don't bother about the rest. Accepting null makes this simpler and more performant (no need to allocate a pair for every node visited, just check for null).

Also, maybe call this method "transform", and call the argument "operator".

Done.

bioball · 2026-07-28T23:21:33Z

+  grammarVersion: "V1" | "V2" = "V2"
+
+  /// Render [node] as Pkl source code.
+  external function render(node: Node): String


How would you render nodes from the typed syntax API? Do users call render(syntaxNode.builtNode)?

I wonder if we should have:

external function render(node: Node | SyntaxNode): String

Where the implementation calls builtNode if given a SyntaxNode. And, if so, I wonder if we even need builtNode as a user-facing API? The underlying implementation would do all the building under the hood.

I think builtNode retains utility in cases where you want to transform a parsed module (incl. round-tripping affixes) and add/replace something built from a SyntaxNode.

For the "transform a parsed module" scenario, you probably should be transforming the raw node. If you go into a typed node and back via builtNode, you'll end up with all sorts of changes to your code that you didn't intend; re-organized members, missing comments, etc.

Sorry, that wasn't super clear. I don't mean parse -> generic -> typed -> some transformation -> generic -> render, I mean parse -> generic -> walk -> modify a (generic) object body and append a child that was constructed from a typed node -> render. For example: modifying a PklProject file to add a dependency, where the new entry was created as a SyntaxNode.

Do users call render(syntaxNode.builtNode)

Yes. It's a lossy operation. To perfectly keep all affixes you have to use GenericNode.

HT154 · 2026-07-31T21:56:36Z

+  /// The integer literal (e.g. `42`, `"0xFF"`).
+  value: Int | String


I was going to suggest this

Suggested change

/// The integer literal (e.g. `42`, `"0xFF"`).

value: Int | String

/// The integer literal (e.g. `42`, `"0xFF"`).

value: Int | String(toIntOrNull() != null)

But then I realized we don't actually have any in-language API for parsing the non-decimal int formats supported in Pkl code!

This would work if we pick up #1808

HT154 · 2026-07-31T22:54:38Z

+class SingleLineStringLiteralExprNode extends ExprNode {
+  /// The string parts (chars, escapes, interpolations).
+  parts: List<StringPartNode>
+}
+
+/// A multi-line string literal expression.
+///
+/// Use [StringNewlineNode] entries in [parts] to separate lines.
+class MultiLineStringLiteralExprNode extends ExprNode {
+  /// The string parts (chars, escapes, newlines, interpolations).
+  parts: List<StringPartNode>
+}


These are missing support for custom string delimiters, which should also influence how escapes/newlines/interpolations are rendered

stackoverflow force-pushed the pkl-syntax branch from 7227c15 to 89c02b4 Compare May 27, 2026 15:44

stackoverflow marked this pull request as ready for review May 27, 2026 15:59

HT154 reviewed May 29, 2026

View reviewed changes

stackoverflow force-pushed the pkl-syntax branch from 56827fa to 3d3ae43 Compare July 13, 2026 15:13

HT154 reviewed Jul 14, 2026

View reviewed changes

Comment thread stdlib/syntax.pkl Outdated

Comment thread stdlib/syntax.pkl Outdated

Comment thread stdlib/syntax.pkl Outdated

bioball requested changes Jul 14, 2026

View reviewed changes

stackoverflow force-pushed the pkl-syntax branch from d277baa to c6ba6c2 Compare July 20, 2026 16:40

bioball reviewed Jul 28, 2026

View reviewed changes

stackoverflow added 16 commits July 30, 2026 17:57

Add syntax package

75d540f

Add support for full round-trip of Pkl code

1e29ca9

Add ast builders

b196781

Fix build

dcf48d8

Join SyntaxNodes and Builders into one; add walker, visitor, and fold

e189aa8

Changed tests from examples to facts

5458321

Move static identifiers to Identifier.java

c6b2449

Change toNode to a property

fef2425

Refactor functions

1ab39ed

Refactor list joins

cc99159

Move parsing methods to Parser class

f47be50

Move node building to Java

acbbaad

Move builtNode to Java

83344ed

Remove unused methods

7e17090

Improve doc comment node

f2e6542

Make parenthesized type non-nullable

81d3f5a

stackoverflow added 2 commits July 30, 2026 17:57

Move top level functions to Node

83c2c62

Fix some review remarks

fd4d48a

stackoverflow force-pushed the pkl-syntax branch from 4f7a99c to fd4d48a Compare July 31, 2026 12:27

stackoverflow added 4 commits July 31, 2026 15:40

Add display url to spans and source locations

2456208

Remove span from SyntaxNode

66071d3

Make ObjectMemberNode a sum type

be92f1d

Explode object members

aad10d1

HT154 requested changes Jul 31, 2026

View reviewed changes

stackoverflow added 22 commits August 3, 2026 11:22

Refine modifier list types

8b9473c

Change some properties to keyword for consistency

b08054d

Fix some remarks

4d91dba

Change binary operator to have their own classes

b756406

Fix bug in generic parser and syntax node

39c13ad

Add tests for quoted identifiers

6e4a52b

Remove toNode

a9b7fac

Rename Node to GenricNode and SyntaxNode to Node

52e9204

Change List to Listing for Node properties

d8704a5

Add members property to ModuleNode

c6bf5cd

Add members property to ClassBodyNode

31e951b

Rename StringConstantTypeNode to StringLiteralTypeNode

817614b

Reuse extra storage for building nodes

b610c43

Remove render function from GenericNode

dde5152

Add name property to identifiers

af30e87

Add constraint to expr bodies

79af24c

Change variance to modifiers

058dba7

Add members to ObjectBodyNode

3bcca77

Fix review remarks

119b667

Add helper to build strings

d64c48a

Improve string parts

bae41f4

Rename walk to transform

7f67e51

		/// The integer literal (e.g. `42`, `"0xFF"`).
		value: Int \| String

Uh oh!

Conversation

stackoverflow commented Apr 30, 2026

Uh oh!

odenix commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HT154 commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stackoverflow commented May 4, 2026

Uh oh!

odenix commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stackoverflow commented May 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bioball left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HT154 Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

odenix commented Apr 30, 2026 •

edited

Loading

HT154 commented Apr 30, 2026 •

edited

Loading

odenix commented May 22, 2026 •

edited

Loading

bioball Aug 1, 2026 •

edited

Loading

HT154 Aug 1, 2026 •

edited

Loading