Figure out a better story around optional proc_macro

[dtolnay]

Registering @alexcrichton's ideas from #137 (comment) so they don't get lost:

---

We've got the following state of affairs:

• This report saying we don't compile by default on alpine
• This report saying the same
• This comment indicating that some musl builds have proc_macro
• This report saying that if proc_macro2 doesn't link to proc_macro then the dependencies of proc_macro2 aren't ready for that.

I think it's sort of a bummer that this crate doesn't work by default on the standard musl target, and I think it'd be great to fix this. Here's an idea, though, of how we could get this working:

1. Somehow in this crate's build script we deterine whether or not to link to proc_macro
2. This crate also indicates links = "proc-macro2-04x"
3. This crate's build script now prints out whether or not it links to proc_macro
4. Crates like syn have a build script.
5. syn reads the env var DEP_PROC_MACRO2_04X, learning about whether proc_macro was linked.
6. syn's build script then prints out something like cargo:rustc-args=--cfg proc_macro_works and is compiled appropriately

This is sort of heinously complex unfortunately though. It also requires this crate's build script to figure out whether to depend on proc_macro or not (which I'm not sure how to do)

[dtolnay]

I would put this effort into rust-lang/rust#49219 instead. In a perfect world:

• libproc_macro would be just as well behaved build-wise as libcore,
• available on all the same targets as liballoc,
• proc macros run out-of-process using ipc with the compiler rather than dynamic linking,
• proc-macro2 is superseded by syn/quote using proc_macro directly.

[alexcrichton]

To pave a way to not use proc-macro2 (which does seem laudable!) we'd have to add token stream parsing and such to rust-lang/rust#49219 as well for when it's not linked to the compiler, which it doesn't currently contain. I do largely agree, though, that rust-lang/rust#49219 is a better solution than what I mentioned.

[dtolnay]

^ @eddyb any issues with providing token stream parsing in proc_macro to make it possible to use standalone from a build script or main.rs?

[mystor]

IIRC there was some resistance to supporting libproc_macro when not linked into the compiler, due to not wanting to link a portion of libsyntax statically into all created binaries.

Perhaps we could factor out the pieces of libsyntax which are needed to get the minimal pieces working & share them?

Unless we can do that, I think we're going to continue to need proc-macro2.

[dtolnay]

@mystor If a binary crate depends on proc_macro, wouldn't only the parts of libsyntax that are used by proc_macro end up in the result? How would we do better by refactoring libsyntax? This seems like the intended behavior.

[mystor]

@mystor If a binary crate depends on proc_macro, wouldn't only the parts of libsyntax that are used by proc_macro end up in the result? How would we do better by refactoring libsyntax? This seems like the intended behavior.

Currently libsyntax is a dylib, so depending on it would create a dynamic dependency, rather than a static one. We'd have to be able to link libsyntax both statically & dynamically IIRC.

[eddyb]

@dtolnay It's not just parsing, @alexcrichton was summarizing. It's an RPC system with compiler objects being controlled from the proc macro client.

However, with that PR, anyone (on nightly) can use the proc_macro::bridge forever-unstable API to use their own proc-macro2-based (or any other) compatible implementation of the public proc_macro API as the server in a proc_macro session.

So if you really wanted, you could put a copy of proc-macro2's internals in proc_macro and have that as the default "loopback" server.

---

Beware of putting the public proc_macro API objects into any global state, btw, as the client<->server bridge will only maintain validity of objects within one session. You can nest sessions, but if you created an object outside of a proc macro being invoked by the compiler (e.g. by life-before-main hacks), you can't use it while the proc macro entry point is running (but you could afterwards).

The whole system is hardened pretty well, no client will ever see the same object handle being used more than once, so cross-session object use will deterministically result in a panic inside the server, which will be propagated to the client - but it is a potential footgun.

---

^{Also, yes, I still need to finish that PR...}

[mystor]

So if you really wanted, you could put a copy of proc-macro2's internals in proc_macro and have that as the default "loopback" server.

I think that would be a potential solution here to eliminate proc-macro2. It would be nice to have some mechanism for proc_macro-based code to run as a standalone library without dynamically linking to or using RPC to talk to the compiler in any way. From my POV, the nicest solution there would be to have a fallback "loopback" server built into proc_macro which is used when there is no compiler session, and doesn't work against real rustc objects.

This would have to be done within rustc itself, as we wouldn't want to require an perma-unstable API to do this. I don't think the way that proc-macro2 implements it is a solution we'd want to upstream, but if we want to do that I can try to find some time to write up a cleaner & more maintainable loopback implementation.

Also, yes, I still need to finish that PR...

❤️

[eddyb]

This would have to be done within rustc itself,

It'd be done entirely within libproc_macro (which only depends on libstd, in my PR).
(I guess you meant Rust in general, not rustc, the program, specifically)

[mystor]

@eddyb yup, that's more accurate. Should've said something like "within rust-lang/rust" or "not crates.io" :-)

[ifreund]

Any update on this? I was rather disappointed to find that many of the programs I use fail to compile on my new musl based system due to this crate.

[kevinmehall]

@ifreund proc-macros work on musl / Alpine if you set RUSTFLAGS=-C target-feature=-crt-static. rust-lang/rust#69519 makes this happen for proc-macro crates by default, so it may "just work" on 1.44 (currently beta).

[ifreund]

@kevinmehall thats great to hear thanks!