feat: add internal spec for debug-info

[djolertrk]

This introduces the compiler-side model for carrying Solidity semantic debug info through the Yul pipeline, including AST-ID based reattachment across Yul reparse.
Current scope is intentionally narrow and we will add more entries into it once we are okay with the proposal on a high-level.

See docs/internals/ethdebug_internal_metadata.rst for more detailed explanation of the design.

[djolertrk]

cc @nikola-matic @clonker

[nikola-matic]

Can you fix the build (the comment I left below should do it), and then can you drop a solc-bench in here.

[nikola-matic]

Suggested change

	std::vector<SemanticDebugVariable> variableDefinitions;
	std::vector<SemanticDebugVariable> variableDefinitions = {};

Since you're using named initializers, this should fix the build.

[djolertrk]

@nikola-matic I fixed the build and here is the solc-bench output:

Benchmark          Metric         ir-ethdebug           ir                   Δ%        winner
-----------------  -------------  --------------------  -------------------  --------  ------
prb-math-4.1.1     cpu_time       12.1321s ± 0.4704s    10.0406s ± 0.3165s   +20.83%   ir
                   creation_size  592,270 ± 0           592,270 ± 0          0.0%      tie
                   peak_rss       2044080 ± 96536 MiB   1295600 ± 52003 MiB  +57.77%   ir
                   runtime_size   589,640 ± 0           589,640 ± 0          0.0%      tie
                   wall_time      12.8027s ± 0.6464s    10.4773s ± 0.4362s   +22.2%    ir
                   ethdebug_size  44.80 MiB ± 0.00 MiB  n/a                  N/A       n/a

forge-std-1.16.1   cpu_time       11.8554s ± 0.2979s    10.2449s ± 0.1482s   +15.72%   ir
                   creation_size  725,280 ± 0           725,280 ± 0          0.0%      tie
                   peak_rss       1554816 ± 70351 MiB   796144 ± 9474 MiB    +95.29%   ir
                   runtime_size   698,851 ± 0           698,851 ± 0          0.0%      tie
                   wall_time      12.1417s ± 0.3452s    10.4575s ± 0.1829s   +16.1%    ir
                   ethdebug_size  44.94 MiB ± 0.00 MiB  n/a                  N/A       n/a

morpho-blue-1.0.0  cpu_time       16.3965s ± 0.7634s    14.7222s ± 1.0004s   +11.37%   ~noise
                   creation_size  1,261,421 ± 0         1,261,421 ± 0        0.0%      tie
                   peak_rss       1824912 ± 50816 MiB   921792 ± 34683 MiB   +97.97%   ir
                   runtime_size   1,251,246 ± 0         1,251,246 ± 0        0.0%      tie
                   wall_time      16.9356s ± 0.9110s    15.0786s ± 1.1852s   +12.32%   ~noise
                   ethdebug_size  57.38 MiB ± 0.00 MiB  n/a                  N/A       n/a

solmate-6          cpu_time       6.9523s ± 0.1573s     6.0532s ± 0.3605s    +14.85%   ir
                   creation_size  540,758 ± 0           540,758 ± 0          0.0%      tie
                   peak_rss       919888 ± 33125 MiB    379440 ± 18556 MiB   +142.43%  ir
                   runtime_size   523,577 ± 0           523,577 ± 0          0.0%      tie
                   wall_time      7.1106s ± 0.1848s     6.2070s ± 0.4427s    +14.56%   ir
                   ethdebug_size  30.92 MiB ± 0.00 MiB  n/a                  N/A       n/a

Thank you!

[nikola-matic]

This should assert that _bytes > 0 as per the spec.

[nikola-matic]

I'm really not a fan of dumping all of this into CompilerStack, especially when these are just free static functions; can you please introduce another class to wrap this (in libsolidity/interface), much like ABI and Natspec. The call sites should remain more or less identical.

[djolertrk]

Hmm, lets do it, I agree!

[nikola-matic]

We can skip this if ethdebug isn't requested, no?

[djolertrk]

Right, done.

[nikola-matic]

In general, if you're using Solidity or Yul as an input to test case, said test case (and thus the suite) should really be an isoltest.

[djolertrk]

yeah, I dropped this suite and moved the coverage to the Ethdebug isoltests

[cameel]

Here's a bunch of remarks from the first reading through the spec. I'm not done yet so I'll likely have more later.

[cameel]

I think this needs more details on how it's meant to be used by the codegen:

• Which Yul AST nodes can have this semantic info? Just blocks or also other types? When the scope is e.g. a function definition or a conditional, is the info in that node or in the node of its block?
• This only notes which Solidity variables are declared in the current scope. How do I know which specific Yul variable maps to which Solidity variable?
• Why is location the initial location? Can location change?

[djolertrk]

Added sections for all three:

• "Scope Attachment": any Yul node can carry it, but the producer attaches to the node that owns the scope so function/modifier records go on the FunctionDefinition itself, not its body block; a source scope lowered to a block goes on that Block; a conditional only gets a payload if it actually introduces a source scope.
• The pointer expression is the mapping: for stack variables its free variables are the exact IRVariable::stackSlots() names, so declaration AST ID gives you the source identity and the Yul names give you the generated one.
• Dropped "initial" - dataLocation is the location valid at the program point of the containing context, and yes, it can change (move, split, get optimized out). There's a "Location Changes" section now with the rules, including the conservative invalidation to OptimizedOut when a stack pointer's Yul names disappear.

[cameel]

Does it really make sense to make OptimizedOut a part of the type description though? I'm actually a bit confused whether the type here is meant to be the original type in the source language or concrete type at the EVM level. Almost all of the fields seem consistent with the former, but then presence of this value and the fact that you went with my suggestion in #16780 (comment) points towards the latter.

Looking at it now, it seems more like an aspect of the pointer to me. Why would the original type info change at all during optimization?

[cameel]

Are we sure that the AST ID is unique enough for this? What about e.g. duplicated or specialized function definitions at Yul level that trace back to the same function at Solidity level? You won't get these in the codegen, but the optimizer could introduce them, so we still need to account for that possibility.

[djolertrk]

Hmm good point (the current spec did not cover optimisations). It's not unique enough once cloning is possible. Added an "Identity and Optimizer Cloning" section: the AST ID is an origin identity, and the full model needs a second, per-instance discriminator (scopeInstanceID) so cloned/specialized Yul functions can carry different locations while sharing declaration/type info. wdyt? @cameel

[cameel]

Ah, wait. Forgot that we're talking about disambiguating scopes, not variables. I still can't get used to the fact that debug data is per-scope, not per-variable. Disregard my comment above :)

In that case yes, scopeInstanceID does seem unique enough, but:

• Where is it stored? I don't see it in any of the structures the spec mentions.
• What does it look like and how do you ensure it's stable? Yul AST does not have IDs so it can't be that.

[cameel]

A few more that the spec should account for:

• File-level constant variables.
  • Note that they can have multiple names due to import aliases (e.g import {CONST as DIFFERENT_CONST} from "a.sol";)
• Variables defined in the try/catch clauses.
• Variables without definition like this, super or msg.

[djolertrk]

Added to the "Complete Variable Model" section

[nikola-matic]

Not a fan of duplicated the enum tables here; you could alter the enumToString/enumFromString templated functions a bit, and have something like this instead:

using PointerClass = SemanticDebugPointer::Class;
constexpr std::pair<PointerClass, std::string_view> pointerClassNames[]{
	{PointerClass::Region, "region"},
	{PointerClass::Group, "group"},
	{PointerClass::List, "list"},
	{PointerClass::Conditional, "conditional"},
	{PointerClass::Scope, "scope"},
	{PointerClass::TemplateReference, "templateReference"},
	{PointerClass::Unknown, "unknown"}
};

std::string pointerClassToString(SemanticDebugPointer::Class _class)
{
	return enumToString(_class, pointerClassNames);
}
SemanticDebugPointer::Class pointerClassFromString(std::string const& _class, std::string const& _path)
{
	return enumFromString(_class, pointerClassNames, _path);
}

Not only would this reduce the line count, but also the risk of drift.

[djolertrk]

Yeah, that does make sense. Thanks @nikola-matic!

[cameel]

Sorry for the delay. Here's another batch of remarks. The spec is much better now, containing a lot of detail that was missing before. I think there are still some holes in it though.

[cameel]

What do we use for user-defined value types (UDVTs)? The ethdebug spec does not have them - is it an intentional omission or are they meant to map to some of the other elementary types it does have?

[cameel]

Another thing to consider is a long address type (32 bytes rather than 20). The EVM needs to eventually migrate to those. It's not clear when and how it will look like, but we promised to introduce them (and matching contract and external function types since these contain addresses) in the near future to ease the potential future migration.

This will also need to be addressed in the spec I guess. The list of types fixed and based on Classic Solidity as it is now, not allowing for easy extension.

[cameel]

We also have an array slice type. Not sure if ethdebug can express that.

Though we only use that for calldata and its internal representation is the same as for ordinary calldata arrays (i.e. both offset and length are stored on stack) so it probably won't be an obstacle to proper implementation, but might be a problem later, in Core Solidity.

[djolertrk]

What do we use for user-defined value types (UDVTs)? The ethdebug spec does not have them - is it an intentional omission or are they meant to map to some of the other elementary types it does have?

no no, it is there, but was not added into the doc. we will use alias. it is now in the document.

Another thing to consider is a long address type (32 bytes rather than 20). The EVM needs to eventually migrate to those. It's not clear when and how it will look like, but we promised to introduce them (and matching contract and external function types since these contain addresses) in the near future to ease the potential future migration.

This will also need to be addressed in the spec I guess. The list of types fixed and based on Classic Solidity as it is now, not allowing for easy extension.

hmm, thank you! addressed: width is data, not a kind. the address/contract carry it in bytes (20), so a 32-byte address is the same kind with a different value, so nothing structural changes.

on extension generally: the format will be versioned and and renders will reject unknown versions, so new kinds are a version bump that will fail loudly. both now in the doc.

We also have an array slice type. Not sure if ethdebug can express that.

Though we only use that for calldata and its internal representation is the same as for ordinary calldata arrays (i.e. both offset and length are stored on stack) so it probably won't be an obstacle to proper implementation, but might be a problem later, in Core Solidity.

hmm, yes, I see. I have added a slice kind, even though it is not in the format, so it only exists in the sidecar. in the public output slices are emitted as plain dynamic arrays, which matches their representation anyway (offset + length on stack), so nothing is lost today. if Core Solidity later needs the distinction visible to debuggers, the internal side already carries it and we can propose a slice kind upstream in ethdebug/format at that point.

[cameel]

Where and how are the templates stored in the sidecar? This says we can have a reference but not how it looks like (do templates have IDs? names?) and where it points to.

[cameel]

Also, how does this relate to the ethdebug.resources.pointers output that we emit during analysis?

[djolertrk]

templates are not stored twice, they are the entries of the ethdebug.resources.pointers output, keyed by the pointerID recorded on the variable's data location. a TemplateReference just names one of those keys (plus yields bindings for its results): the sidecar carries the reference, the resource output carries the template. this is now added to doc, thanks!

[cameel]

This could use more detail. Is data literally langutil::DebugData? That already contains the AST ID, so is that duplicated?

Also, why a list of pairs and not a dictionary/map? Is that because AST IDs on their own are not unique? Why not a two-level map with the scope ID as the second level key? Given that there are multiple variables per scope it's not really a proper flat table anyway.

[djolertrk]

hmm yes, I agree.

data is SemanticDebugData, not langutil::DebugData, so no AST ID inside, no duplication.

on uniqueness you were right. the entries are now keyed by (astId, instance), so your two-level suggestion, with instance 0 implicit, so un-cloned output is unchanged and cloned code later isn't a format break. and we are using array because the key is an integer pair and it keeps output deterministic.

[cameel]

It seems to me that with this approach we'll not make much use of the pointer capabilities and pretty much always use the stack $read expression. It makes things easy in the codegen, because we do emit local variables for everything, but it will make it harder to preserve that debug info in the optimizer.

Let's take a concrete example. Say, we have a uint storage variable and the codegen produces something like this:

let _1 := read_from_storage_split_offset_0_t_uint256(0x00)
let _2 := foo(_1)

Based on your description, I imagine the pointer expression will look like this:

{
    "location": "storage",
    "slot": {
        "$read" {
            "location": "stack",
            "slot": "_1"
        }
    }
}

Later some optimizer step reduces the code to this:

let _2 := foo(read_from_storage_split_offset_0_t_uint256(0x00))

How will the debug info be transformed to survive this step? The codegen knows that read_from_storage_split_offset_0_t_uint256() is just an sload() with some cleanup but to the optimizer it's an opaque function call it cannot make assumptions about.

[cameel]

I guess you could simply dump the whole expression in there:

{
    "location": "storage",
    "slot": {
        "$read" {
            "location": "stack",
            "slot": "read_from_storage_split_offset_0_t_uint256(0x00)"
        }
    }
}

It's not the proper ethdebug format but still just the sidecar, so it only needs to be carried through Yul. When we get to Yul->EVM transform the result will end up in some stack slot we'll be able to point at.

A new problem here is that for this to work we will have to allow arbitrary Yul in these expressions, which means that every optimizer step will have to transform them along with the input code. And they may come from the user so they could be large and very complex in the worst case.

---

A simpler approach would be to resolve the region already in the Yul codegen:

{
    "location": "storage",
    "slot": "0"
}

That's much easier to propagate. The problem here is that we lose the information that _1 carries this information. We only know that the uint variable is used in the current scope and where to find its value. But is that even important to the debugger if the value of the Solidity variable can be calculated anyway?

Another problem here: we can use this for storage or memory, but not for Solidity locals. When these get expanded at Yul level into an expression, the only thing we can point at is that whole expression. Which brings us back to carrying arbitrary Yul.

---

One way to side-step this would be to forgo scopes and attach variable information directly to the Yul node that represents the Solidity value. Then it can no longer be arbitrary and we don't have to carry it in the sidecar at all. The pointer can be the simpler fully resolved version or it can be omitted and only produced when we emit ethdebug info (at which point the expression already has a stack slot assigned to it).

[djolertrk]

It seems to me that with this approach we'll not make much use of the pointer capabilities and pretty much always use the stack $read expression. It makes things easy in the codegen, because we do emit local variables for everything, but it will make it harder to preserve that debug info in the optimizer.

Let's take a concrete example. Say, we have a uint storage variable and the codegen produces something like this:

let _1 := read_from_storage_split_offset_0_t_uint256(0x00)
let _2 := foo(_1)

Based on your description, I imagine the pointer expression will look like this:

{
    "location": "storage",
    "slot": {
        "$read" {
            "location": "stack",
            "slot": "_1"
        }
    }
}

Later some optimizer step reduces the code to this:

let _2 := foo(read_from_storage_split_offset_0_t_uint256(0x00))

How will the debug info be transformed to survive this step? The codegen knows that read_from_storage_split_offset_0_t_uint256() is just an sload() with some cleanup but to the optimizer it's an opaque function call it cannot make assumptions about.

you're right that this didn't survive. it's reworked: a storage variable's pointer is now resolved at codegen to plain slot/offset {"location": "storage", "slot": "0"}, so with no Yul names in it, so your example folds freely and the pointer is untouched. the stack $read` is no longer the default shape and the doc now specifies the split.

I guess you could simply dump the whole expression in there:

{
    "location": "storage",
    "slot": {
        "$read" {
            "location": "stack",
            "slot": "read_from_storage_split_offset_0_t_uint256(0x00)"
        }
    }
}

It's not the proper ethdebug format but still just the sidecar, so it only needs to be carried through Yul. When we get to Yul->EVM transform the result will end up in some stack slot we'll be able to point at.

A new problem here is that for this to work we will have to allow arbitrary Yul in these expressions, which means that every optimizer step will have to transform them along with the input code. And they may come from the user so they could be large and very complex in the worst case.

A simpler approach would be to resolve the region already in the Yul codegen:

{
    "location": "storage",
    "slot": "0"
}

That's much easier to propagate. The problem here is that we lose the information that _1 carries this information. We only know that the uint variable is used in the current scope and where to find its value. But is that even important to the debugger if the value of the Solidity variable can be calculated anyway?

Another problem here: we can use this for storage or memory, but not for Solidity locals. When these get expanded at Yul level into an expression, the only thing we can point at is that whole expression. Which brings us back to carrying arbitrary Yul.

One way to side-step this would be to forgo scopes and attach variable information directly to the Yul node that represents the Solidity value. Then it can no longer be arbitrary and we don't have to carry it in the sidecar at all. The pointer can be the simpler fully resolved version or it can be omitted and only produced when we emit ethdebug info (at which point the expression already has a stack slot assigned to it).

hmm, yeah, I went with a combination of your last two suggestions, and ruled out arbitrary Yul for exactly the reason you give. it would make the sidecar a second program every pass has to rewrite.

Storage/transient/code: resolved at codegen, your simpler form. On your question: I don't think the debugger needs to know _1 carries the value since it's computable from the resolved pointer, and nothing else depends on which local held it. That trade is now stated in the doc.
Locals: your node-attachment idea. identity rides on the DebugData of the producing node, and the pointer is completed at emission once slots exist. I've split YulLocal from Variable in the expression kinds so emission drops an unresolved pointer instead of publishing a Yul name as a slot; and for values that move mid-scope there's now locationUpdates (it is rebinding at the node where it takes effect). the emission time resolution for locals is part of the next milestones, so no need to change schema for that.

[cameel]

How does this relate to the ethdebug.resources.types that we emit during analysis? We do not really have to pass type info through Yul, do we?

The main difference I can see is that after you followed my suggestion in #16780 (comment), we are storing the actual location for constants/immutables at creation/deployed time. That's only possible if we have separate types for these. I now see that you can express this distinction with pointers though so maybe it was unnecessary. Unnecessary for the implementation I mean - I do think it should be addressed in the spec by making it more generic and less tied to Solidity.

Also is the type info duplicated in each variable? ethdebugType does not say "pointer" so I assume it's a copy, but given a very limited number of types that would unnecessarily produce much larger JSON.

[djolertrk]

ethdebugType is exactly what gets registered in ethdebug.resources.types under typeId so public output has one descriptor per type. It does need to cross Yul: with two-stage compilation (and other Yul frontends) the second stage has no AST to look types up in, right?

On constants/immutables, yes, agreed, and after your earlier suggestion that's how it works now: EVM-level location kinds plus pointers, nothing Solidity-specific.

On duplication: fixed.

[cameel]

Where is typeID defined in the ethdebug spec? I don't see it in the types schema. I vaguely remember the native type identifier being there somewhere, but I can't find it now. Is it actually a part of the spec?

[cameel]

It does need to cross Yul: with two-stage compilation (and other Yul frontends) the second stage has no AST to look types up in, right?

I mean, the information has to be made available to Yul, but not necessarily in a custom format inside the sidecar. Yul compilation could just as well consume ethdebug.resources.types directly, which would make the sidecar format simpler. Instead of defining SemanticDebugType, which is a rehash for the type schema, we could say that types is the type schema.

[cameel]

It changes "pointer location", right? Spilling should not really change the location of the data.

[djolertrk]

hmmm, right. the pointer's location changes, the data doesn't move. reworded, and the Remap rule now also notes that renaming can't reach statically addressed pointers since those hold no Yul names.

[cameel]

We should either enumerate those here or refer to the ethdebug spec for full list. At least if we do support all of them, which is not a given - we may not need the full expressivity of the spec for the sidecar.

[cameel]

BTW, it would generally be good to have more links to the relevant parts of the ethdebug spec across the document.

[djolertrk]

done. it's now a table enumerating exactly the kinds the producer emits, linked to the ethdebug pointer schema, with a note that the reader accepts the rest of the spec's kinds without the producer emitting them.
And yeah, I also added links to the program/type/pointer schemas throughout the document, so I hope those do not become stale/hard to maintain.

[cameel]

Did you mean to move this field to SemanticDebugVariable? I see it was added there, but not removed from there so now we store location twice.

[cameel]

Ah wait, this one is a string. Is the idea that this is literally what is specified on the variable in Solidity source (memory, storage, calldata, etc.)? Perhaps also immutable and constant?

I'm still not sure how it is relevant to the compiler. It might be relevant to the debugger, but then it should be a part of the type schema, not the sidecar.