Addressable comments (a very small DOM parts subset)

[noamr]

One of the use cases that keeps coming up for DOM parts is being able to have a reference to a part of the DOM that is not directly tied to elements.

See WICG/declarative-partial-updates#6 and whatwg/dom#736.

The way frameworks often do this today is by creating comment nodes and keeping reference to them.
What if we implemented exactly that?

Allow some sort of string syntax inside a comment that makes it efficiently addressable from a DOM API, but it's a comment node for all other purposes (e.g. it's not stylable, doesn't affect nth-child, doesn't appear in querySelector etc).

That comment is there only for the purpose of being addressable.

Some use examples::

• we can have <template contentmethod> patches able to stream between a pair of these.
• a JS API can keep it as a stable reference into a table and replace multiple rows

The syntax inside the comment can be similar to a data URL, e.g. marker:key=value;my-placeholder-x, but I'm sure people here are more opionated than myself about this part.

Then there can be a JS API that efficiently addresses it, e.g. documentOrShadowRoot.getMarkerCommentsByName("my-placeholder-x").

This way patching can address this comment or a range between two comments, server/client flows can update the same DOM range without requiring them to be between elements, and streaming can maintain a stable place in the DOM to stream into.

This is supposedly a lot smaller in scope than the existing DOM parts proposal, and DOM parts can perhaps be built in userland on top of this?

[mfreed7]

I love this idea! It does indeed seem like all of DOM Parts can be built (in userspace, if desired) on top of this primitive. And it expands the use cases beyond DOM Parts also, as you point out.

[justinfagnani]

I'm having trouble following this idea. What does it offer template libraries over today's comments?

[noamr]

I'm having trouble following this idea. What does it offer template libraries over today's comments?

I realized some things were missing from the OP. It allows you to address a range in the DOM without requiring it to be element-bound. Hope the updates make more sense.

[justinfagnani]

I see the edit to add a query API.

Honestly, ParentNode.getComments() would work just about as well for us. We can easily filter out comments that don't match our comment format. There are usually very few non-markers comments in templates anyway.

But to analyze this API against DOM Parts (or really Template Instantiation), being only able to address child node ranges isn't all that useful because we have to visit and insert markers for attributes, inside rat text elements like <script> and <style> and ideally recurse into nested <template>s.

There two big places we'd like to optimize with something like parts:

1. Attaching parts to the template. Right now we have to insert markers of the appropriate type (comments where we can, text in unquoted attribute values, comment data, and raw text elements), create the <template>, then walk the template content to find the markers. Because this is only comments it doesn't eliminate the tree walk. We would still have to have a walk for attributes, raw text, comments with bindings in them, and <template>s.

   You can see this code here

2. Finding markers in cloned template instances. This requires a simpler tree walk because we're recorded the preorder DFS index of all the nodes that have attached parts. It could maybe be useful here if we insert these markers instead of recording the node indices. It's not much code that we would save (example here) - instead of iterating on a TreeWalker we'd iterate through the comments returns, so maybe a couple of lines - but maybe it'd be faster. The biggest hope we have for a boost is that we don't create JS wrappers for nodes we don't care about.

So I don't see that this, or NodeGroup, or any DOM Parts alternative ideas that only supports child ranges is that useful of a substitute for DOM Parts or Template Instantiation.

If this ended up being faster than a TreeWalker, and that was because of fewer Node wrappers, I think it'd be very powerful to explore a new TreeWalker API that supports more declarative node filtering. A filter object with regexes that can run on tag names, attribute names and values, child text, and comment data, would let you pull out just the comment nodes you care about and more.

[noamr]

I see the edit to add a query API.

Honestly, ParentNode.getComments() would work just about as well for us. We can easily filter out comments that don't match our comment format. There are usually very few non-markers comments in templates anyway.

I get that for this use case. The proposal here is more general and not tied to templates.

But to analyze this API against DOM Parts (or really Template Instantiation), being only able to address child node ranges isn't all that useful because we have to visit and insert markers for attributes, inside rat text elements like <script> and <style> and ideally recurse into nested <template>s.

DOM parts allows you inserting markers into <script> and <style>?
I mean we can also develop marking comments in CSS/JS as well but I'm not convinced it's necessary.

There two big places we'd like to optimize with something like parts:

1. Attaching parts to the template. Right now we have to insert markers of the appropriate type (comments where we can, text in unquoted attribute values, comment data, and raw text elements), create the <template>, then walk the template content to find the markers. Because this is only comments it doesn't eliminate the tree walk. We would still have to have a walk for attributes, raw text, comments with bindings in them, and <template>s.

The idea is that you can most all of this efficiently with addressable comments, without having to walk the tree.
You can apply your own semantics to comments, like "the node range between those markers means something" or "the parent element of this comment means something" or whatever.

You can see this code here
2. Finding markers in cloned template instances. This requires a simpler tree walk because we're recorded the preorder DFS index of all the nodes that have attached parts. It could maybe be useful here if we insert these markers instead of recording the node indices. It's not much code that we would save (example here) - instead of iterating on a TreeWalker we'd iterate through the comments returns, so maybe a couple of lines - but maybe it'd be faster. The biggest hope we have for a boost is that we don't create JS wrappers for nodes we don't care about.

Yea but getMarkerCommentsByName is recursive and cached ahead of time by the parser, so this might be much more optimized?

So I don't see that this, or NodeGroup, or any DOM Parts alternative ideas that only supports child ranges is that useful of a substitute for DOM Parts or Template Instantiation.

I wouldn't assume that it only supports child ranges. Child ranges are just a main use I had in mind.

[justinfagnani]

1. Attaching parts to the template. Right now we have to insert markers of the appropriate type (comments where we can, text in unquoted attribute values, comment data, and raw text elements), create the <template>, then walk the template content to find the markers. Because this is only comments it doesn't eliminate the tree walk. We would still have to have a walk for attributes, raw text, comments with bindings in them, and <template>s.

The idea is that you can most all of this efficiently with addressable comments, without having to walk the tree.

I don't think comments only will eliminate this tree walk. In this walk we look for things like attribute names with a suffix that means they're bound, and we search <script> and <style> and comments for marker text. We can't replace all of this with comments because unquoted attributes and comment text can't contain >.

I wouldn't assume that it only supports child ranges. Child ranges are just a main use I had in mind.

Do you have any ideas for supporting attributes and ranges in raw text and comment text?

We are going from a list of strings like*:

['<input .value=', '><div class="foo ', ' ', '">', '</div><style>', '</style>']

then insert markers between the strings that represent the parts, then create a <template>, then clone, etc...

The HTML we currently generate from the strings looks like this:

<input .value$lit$=lit$x$><div class$lit$="foo lit$x$ lit$x$"><!--lit$x$--></div><style><!--lit$x$--></style>

We place the markers where the string breaks are, and we do just enough tokenization to know whether we can place a comment or just a sentinel string, and to suffix bound attribute names, and a very, very tight code size and perf budget with which to do that tokenization.

So if this were going to be useful for us on that tree walk, we would have to be able to use comments everywhere. Otherwise what we really want is Part objects that can be attached to just about anywhere in a template (ie attributes, etc.) and cloned with the template.

* This would the the strings for a template like:

html`<input .value=${v}><div class="foo ${c1} ${c2}">${content}</div><style>${styles}</style>`

[noamr]

1. Attaching parts to the template. Right now we have to insert markers of the appropriate type (comments where we can, text in unquoted attribute values, comment data, and raw text elements), create the <template>, then walk the template content to find the markers. Because this is only comments it doesn't eliminate the tree walk. We would still have to have a walk for attributes, raw text, comments with bindings in them, and <template>s.

The idea is that you can most all of this efficiently with addressable comments, without having to walk the tree.

I don't think comments only will eliminate this tree walk. In this walk we look for things like attribute names with a suffix that means they're bound, and we search <script> and <style> and comments for marker text. We can't replace all of this with comments because unquoted attributes and comment text can't contain >.

I wouldn't assume that it only supports child ranges. Child ranges are just a main use I had in mind.

Do you have any ideas for supporting attributes and ranges in raw text and comment text?

We are going from a list of strings like*:

['<input .value=', '>

', '

<style>', '</style>']
then insert markers between the strings that represent the parts, then create a <template>, then clone, etc...

The HTML we currently generate from the strings looks like this:

<input .value$lit$=lit$x$><div class$lit$="foo lit$x$ lit$x$"><style></style>
We place the markers where the string breaks are, and we do just enough tokenization to know whether we can place a comment or just a sentinel string, and to suffix bound attribute names, and a very, very tight code size and perf budget with which to do that tokenization.

So if this were going to be useful for us on that tree walk, we would have to be able to use comments everywhere. Otherwise what we really want is Part objects that can be attached to just about anywhere in a template (ie attributes, etc.) and cloned with the template.

• This would the the strings for a template like:

html<input .value=${v}><div class="foo ${c1} ${c2}">${content}</div><style>${styles}</style>

These are interesting but they all match what addressable elements can already do. pretty efficiently e.g.

<input marker="$lit1">
<div marker="$lit2" class="foo current_c1 current_c2">current_content</div>
<style marker=$lit3>current_styles</style>

    render(({v, c1, c2, content, styles}) => {
        const lit1 = app_root.querySelector("[marker=$lit1]");
        lit1.value = v;
        const lit2 = app_root.querySelector("[marker=$lit2]");
        lit2.setAttribute("class", `foo ${c1} ${c2}`);
        lit2.replaceChildren(content);
        const lit3 = app_root.querySelector("[marker=$lit3]");
        lit3.replaceChildren(styles);
    });

The thing that's hard to address is ranges...
Sure it's nice to also have attribute markers but do they actually make anything efficient? Seems like a thin user-space layer can do the attribute templating using element markers.

Either way, there might be a use case for attribute markers that I don't see yet but perhaps they are somewhat orthogonal to comment markers?

[FrameMuse]

Honestly, I wanted to propose something like Comment Attributes as well. I went through this little discussion and now I see the problems too.

I see that the problem that is being solved with Comments is Hydration, which requires markers somewhere in a HTML string and they must be indexed.
I refer to Hydration as a process where you evaluate anything you previously left in HTML with a potential to be addressed.

There should be CommentWalker, which should efficiently help to filter out desired comments.

Or something like CommentRegistry can be introduced, so that we can track and access comments from a WeakSet or maybe even Map by giving a pattern that should be used to group comments under a certain format.

class CommentRegistry {
  constructor(predicate: (commentName: string) => string | null)

  readonly set: WeakSet<Comment> // Includes all filtered comments.
  readonly map: Map<string, WeakSet<Comment>> // Groups all filtered comments by given group key.
}

const commentRegistry = new CommentRegistry(commentName => {
  if (commentName === "$lit1") return "lit1"
  if (commentName === "$lit2") return "lit2"
})

document.body.append(new Comment("$lit1"))

for (comment of commentRegistry.map.get("lit1") ?? []) {
  doSomething(comment)
}

document.body.innerHTML = someNewHTML

for (comment of commentRegistry.map.get("lit2") ?? []) {
  doSomething(comment)
}

It would do indexing at the moment when Comment Node is parsed or added to the document (after commentRegistry is initialized to avoid unnecessary overhead), which could be a better strategy than "at the moment query", but maybe I'm missing something.

---

As for DOM parts, if narrowing to the least primitive, I think it should be either Custom Comments (like Custom Elements), so that they can be tracked or anything other that do node connection tracking.

Because if an element or a comment is detached, you're cooked, you can't do anything about, but if you have tracking, you can replace a "gone element" with a Comment Node to replace.

So it's not that we're missing Comment Attributes - we can implement it on our own, but it's the problem that we lack control of DOM itself - we can't "customize" the tree.

That's why we propose something that can be put into the DOM, being a point of connection to elements and invisible to basically anything except the owner. This is just a pursuit for more control over DOM tree behavior. IMHO.

[noamr]

btw seems like whatwg/dom#736 was going that way, but I was having a hard time following that thread.

[sorvell]

I want to mostly echo #1116 (comment). It's a nice advancement if we can have a performance win by avoiding the tree walk. Just to be clear, it has to be faster than something like:

function collectComments(root: Node = document): Comment[] {
  const walker = document.createTreeWalker(root, NodeFilter.SHOW_COMMENT);
  const comments: Comment[] = [];
  let node: Comment | null;
  while (node = walker.nextNode() as Comment | null) comments.push(node);
  return comments;
}

Other than that...

• I think it's possible easily to extract any attribute/event/property info into a comment that's say the first child of an element.
• Supporting ranges in text only elements (mostly style) would be really great as well, and putting the comment after the entire element makes parsing the input trickier.

[noamr]

I want to mostly echo #1116 (comment). It's a nice advancement if we can have a performance win by avoiding the tree walk. Just to be clear, it has to be faster than something like:

function collectComments(root: Node = document): Comment[] {
const walker = document.createTreeWalker(root, NodeFilter.SHOW_COMMENT);
const comments: Comment[] = [];
let node: Comment | null;
while (node = walker.nextNode() as Comment | null) comments.push(node);
return comments;
}

Exactly, the idea is to do the above but more efficiently, and perhaps support some use cases out of the box such as selecting the range between two comments, or patching using <template contentmethod> (whatwg/html#11542).

Other than that...

• I think it's possible easily to extract any attribute/event/property info into a comment that's say the first child of an element.

I don't exactly understand.

• Supporting ranges in text only elements (mostly style) would be really great as well, and putting the comment after the entire element makes parsing the input trickier.

Is there a need to replace partial ranges inside a <style>, or the whole style? If it's the latter, why not address the style element itself? If the former, perhaps the caller could manage the templating?
I mean we could add something to the CSS parser that contributes to these ranges but it might make this exponentially more complicated so I would like to understand the use case better before baking it in.

[sorvell]

I think it's possible easily to extract any attribute/event/property info into a comment that's say the first child of an element.

I don't exactly understand.

A templating library typically wants to be able to support updating properties, attributes, and event handlers on elements. You'd want to encode info about that into a comment somewhere. That's all I meant.

Is there a need to replace partial ranges inside a <style>

It's a nice to have, but it's a pretty bad practice performance wise. I'd be willing to just say it's not supported.

[EisenbergEffect]

Apologies for the upcoming long post...

In the event that it helps this discussion, I thought I would show what my template compiler takes as input and what its code emitters (roughly) produce as output. I'll demonstrate with a basic todo app template, which most folks should be familiar with, and which provides the opportunity to show a variety of templating constructs.

Here's the input template:

<template>
  <form @submit="this.addTodo($event)">
    <input type="text" ref="description" placeholder="description">
    <button type="submit">Add Todo</button>
  </form>

  <ul>
    {#for todo of this.todos}
      <li>
        <input type="checkbox" ?checked="todo.done" @change="todo.done = !todo.done" title="done">
        <span class="{todo.done && 'done'}">{todo.description}</span>
        <button type="button" @click="this.removeTodo(todo)">X</button>
      </li>
    {/}
    {#empty}
      No todos...
    {/}
  </ul>
</template>

The important thing to note is that there are a variety of aspects of the DOM that the runtime needs to affect. It needs to:

• Add DOM events (@submit, @click, and @change)
• Capture element references (ref)
• Mark locations in the DOM to potentially insert/update HTML fragments. (#for and #empty)
• Mark locations in the DOM to potentially insert/update text. ({...})
• Synchronize boolean attributes with state (?checked="...")
• Synchronize content attributes with state (class="{...}")
• Synchronize IDL attributes (properties) with state (not shown in this example, but imagine :value or .value similar to ?checked)

Given this template, the compiler will produce a model that is passed to two different emitters, each of which produces an optimized output for its target context:

• Browser Emitter - Generates code that enables the template to be instantiated and data-bound entirely in the browser.
• Server Emitter - Generates HTML that can be streamed over HTTP and non-destructively hydrated in the browser.

Let's look at the output of the browser emitter first.

const itemTemplate = __template(
	`<li><input type="checkbox" title="done"><span><!----></span><button type="button">X</button></li>`,
	r => {
		const n = {};
		n.B = r.firstChild.firstChild;
		n.C = n.description.nextSibling;
		n.D = __toText(n.C.firstChild);
		n.E = n.C.nextSibling;
		return n;
	},
	function(v, n) {
		__boolAttrBehavior(v, n.B, () => this.todo.done, "checked");
		__listener(v, n.B, () => this.todo.done = !this.todo.done, "change");
		__propBehavior(v, n.C, () => __class(this.todo.done && 'done'), "className");
		__contentBehavior(v, n.D, () => this.todo.description);
		__listener(v, n.E, () => this.$root.removeTodo(this.todo), "click");
	}
);

function createItemScope(p, o, k) {
	const s = Object.create(null);
	s.$root = p;
	s.todo = o[k];
	return s;
}

const emptyTemplate = __template(
	`
        No todos...
      `
);

const mainTemplate = __template(
		`<form><input type="text" placeholder="description" id="description"><button type="submit">Add Todo</button></form><ul><!----></ul>`,
		r => {
			const n = {};
			n.A = r.firstChild;
			n.description = n.A.firstChild;
			n.B = n.A.nextSibling.firstChild;
			return n;
		},
		function(v, n) {
			__listener(v, n.A, $event => this.addTodo($event), "submit");
			__forOfBehavior(v, n.B, () => this.todos, itemTemplate, this, createItemScope, void 0, emptyTemplate);
		}
	);

This may be a lot to absorb, but the key insight is that the runtime template has three distinct parameters:

1. The Raw HTML - This is the HTML that will be used to create a template instance at runtime.
2. The Node Locator - This is compile-time optimized code used to locate the DOM nodes that need behavior applied.
3. The Behavior Binder - This code applies behaviors to the previously located nodes. (e.g. adds listeners, sets up state sync, etc.)

Notice that in some cases the raw HTML has comments. These are used to mark locations where blocks or text need to be inserted or updated. Sometimes they are also used to provide a "stable" first child for a view (the compiler determines whether this is needed). But comments are not the best way to mark all nodes that need behavior. They are not without a cost, and in larger apps with many templates, the number of comments does have an effect on performance. As such, anything that is easily locatable directly, without a comment marker, is located that way, so that the minimal number of additional nodes is needed.

Comments play a critical role in today's browser rendering though. And they typically also have metadata associated with them so that behavior can be attached. But they aren't the optimal way to handle everything.

Now, let's look at the server render version of this same template. It's different because it has different needs and there are more efficient ways to produce this HTML in a request/response context.

NOTE: I'm showing debug output here because it's easier to read. Non-debug code combines yields as much as possible.

function* itemTemplate(ctx, registry) {
	yield `<!--${ctx.viewId}-->`;
	yield `<li>`;
		yield `<input data-r="description" data-v="${ctx.viewId}" type="checkbox" title="done" ${this.todo.done ? "checked" : ""}>`;
		yield `<span data-r="C" data-v="${ctx.viewId}" class="${__class(this.todo.done && 'done')}">`;
			yield `<!---->${__escapeText(this.todo.description)}<i-m data-r="D" data-v="${ctx.viewId}"></i-m>`;
		yield `</span>`;
		yield `<button data-r="E" data-v="${ctx.viewId}" type="button">`;
			yield `X`;
		yield `</button>`;
	yield `</li>`;
}

function* emptyTemplate(ctx, registry) {
	yield `<!--${ctx.viewId}-->`;
	yield `
        No todos...
      `;
}

function createItemScope(p, o, k) {
	const s = Object.create(null);
	s.$root = p;
	s.todo = o[k];
	return s;
}

function* mainTemplate(ctx, registry) {
	yield `<form data-r="A" data-v="${ctx.viewId}">`;
		yield `<input data-r="description" data-v="${ctx.viewId}" type="text" placeholder="description" id="description"${ctx.renderRefAttributeHTML("description")}>`;
		yield `<button type="submit">`;
			yield `Add Todo`;
		yield `</button>`;
	yield `</form>`;
	yield `<ul>`;
		const $viewId = ctx.viewId;
		const $var2 = this.todos;
		if (!$var2 || $var2.length < 1) {
			yield* ctx.renderComposition("B", emptyTemplate, this, registry);
		} else {
			for (let $var1 = 0; $var1 < $var2.length; ++$var1) {
				ctx.pushViewId();
				const $viewTargetId = ctx.viewId;
				
				yield* ctx.renderTemplate(itemTemplate, createItemScope(this, $var2, $var1), registry);
				
				ctx.popViewId();
				if ($var1 === $var2.length - 1) {
					yield `<i-m data-v="${$viewId}" data-r="B" data-vt="${$viewTargetId}"></i-m>`;
				} else {
					yield `<i-m data-vt="${$viewTargetId}"></i-m>`;
				}
			}
		}
	yield `</ul>`;
}

Ok, this is obviously quite a bit different. We don't need to split out raw html from node location and data-binding. These templates are just functions that emit bits of HTML into the response stream.

But there's again some important details because we need adequate metadata in order to be able to non-destructively hydrate (add the same behaviors without destroying nodes or unnecessarily writing to the DOM) in the browser. We use a combination of things to accomplish this:

• data-v - This attribute marks the view that a particular behavior-laden element is part of. We need this because, as you can see with the todo list rendering, we need to render the same template multiple times. And sometimes, we need to do this with a sequence of sibling nodes. We cannot assume that there is a singular parent for a view. So, this attribute marks which view instance each behavior-laden element is part of.
• data-r - This attribute indicates the node reference name. These are collected onto a lookup object so that the browser behavior binder can apply the behaviors.
• <i-m ...> - This element is a general marker, easily locatable via a query selector and serves as a relative anchor for locating other things.
• data-vt - Used to mark the end of a view. The runtime can traverse backwards from here to find the view start and gather nodes.

And yes, there are comments as well. These may be used to mark the start of a block of text so that the browser doesn't collapse nodes that are needed for rendering. They also mark a view's start.

So, again comments are important, but they aren't the only thing that plays a role in hydration in this example.

Regarding this proposal, I don't know that it's sufficient to replace all location-related code shown here. It may help, but it's less likely if I need to create a tree walker to find the comments. Particularly in the browser render scenario, I've already got an optimized path for node location.

Would this new approach be able to provide similar performance to my compiled output? Would it introduce complication into the non-destructive hydration code and would there be enough benefit? Perhaps I could remove the <i-m> elements and use comments for those, but is it worth it?

I don't know the answers to these questions. But I wanted to share something concrete from a real framework so folks can understand at least one way that these scenarios are being approached today.

[trusktr]

current_content

<style marker=$lit3>current_styles</style>

render(({v, c1, c2, content, styles}) => {
    const lit1 = app_root.querySelector("[marker=$lit1]");
    lit1.value = v;
    const lit2 = app_root.querySelector("[marker=$lit2]");
    lit2.setAttribute("class", `foo ${c1} ${c2}`);
    lit2.replaceChildren(content);
    const lit3 = app_root.querySelector("[marker=$lit3]");
    lit3.replaceChildren(styles);
});

There is something missing in this example. From your code snippet, there's no indication of how the marker attributes track what part of the DOM should be updated. marker="$lit1" alone isn't enough information.

What @sorvell mentioned via tracking that info in a first-child comment, is one way to solve it.

How might you do it with your example?

Basically template library authors want to easily get all the interpolation points no matter where they are.