feat: into_package() for ProtocolLib and StandardLib + CI Artifact

[lima-limon-inc]

Closes 0xMiden/midenup#200

This PR is part of an ongoing effort to migrate serialization towards the .masp Package format exclusively (see 0xMiden/midenup#191 (comment)).

A new into_package() method for both ProtocolLib and StandardLib was added which directly returns the corresponding Package.
This new method is then used in generate_package.rs script, which publishs the masp artifact onto the Github release page.

Notes:

• into_package() was also added for StandardLib for consistency's sake, it wasn't strictly necessary.
• This is a follow up of feat: add action to upload pre-compiled artifacts miden-vm#3029

[mmagician]

Overall looks good, but there are two items I'd still like double checked:

• the CI should follow whatever patterns 0xMiden/miden-vm#3029 establishes, it looks like there are a lot of useful comments about the upload-artifcats job there which will be relevant in protocol repo, too
• verify whether only ProtocolLib or StandardsLib needs to be uploaded as per my comments

[mmagician]

I wonder if there are any meaningful tests we could add for into_package()

[bitwalker]

I think before this gets merged, we should switch to building the protocol and standards packages as Miden projects, i.e. via miden-project.toml. You can still obtain a Library from the resulting package for legacy uses, but it produces a proper package rather than trying to create one from a Library, which IMO we should not do.

That way this function goes away, and packaging uses the standard workflow and tooling provided by the assembler, and we're guaranteed that all of the package metadata is correct.

[mmagician]

we should switch to building the protocol and standards packages as Miden projects, i.e. via miden-project.toml

Would this need to happen here in the protocol repo? I haven't followed up on the packages/projects work recently

[lima-limon-inc]

Would this need to happen here in the protocol repo? I haven't followed up on the packages/projects work recently

I believe so. I believe that would imply switching to a masm miden-project here on the protocol repo.

[bitwalker]

@mmagician Yes, the miden-project.toml would be defined for each logical workspace/project in this repo for which we want to produce a Miden package (we modeled the transaction kernel in the examples under miden-project in the VM repo, but the actual definition would look slightly different here). If you have specific questions about that, I'd be glad to hop on a call to discuss how to approach the project structure , or whatever is most convenient. I can take a stab at defining it too, but you guys know best how you intend to structure packages for things like standards.

[mmagician]

@bitwalker regarding your suggestion to switch to using miden-project.toml to build packages, IIUC we have two options:

Option A - each dependency package with its own manifest:

• crates/miden-protocol/asm/miden-project.toml would be a workspace with four members:
  • shared_utils,
  • kernels/transaction
  • protocol
  • asm/shared_modules/ is either pre-linked as currently, or maybe refactored to promote to a 4th workspace member.
• crates/miden-standards/asm/miden-project.toml would be a workspace with a single standards member that depends on protocol above
• build.rs in both crates would be rewritten to call Assembler::link_package
  instead of assembling kernel/library imperatively. I think this is a positive change and will result in less boilerplate code for building.

Option B - only the userspace libraries (protocol & standards) get a manifest:

• crates/miden-protocol/asm/protocol/miden-project.toml and
  crates/miden-standards/asm/standards/miden-project.toml, both single-
  package manifests
• Kernel, shared_utils, etc. stay assembled via build.rs as today and only the final protocol/standards libraries are produced via link_package.

LMK if this understanding is correct. AFAIU, for option B - if feasible - is simpler and would be a good start already.
But maybe long-term, we could still implement A.
Am I making sense? (as I said I'm only catching up on the package/project concepts)

[mmagician]

It looks like the last commit regressed the naming and we have the miden- prefix again