Add KVP Tracer and OpenTel

[peytonr18]

Draft PR to discuss OpenTelemetry Tracer and next steps for making tracer async, refactoring , and instrumenting the rest of the repo

[jeremycline]

I think we can shuffle things around a bit and configure the initial layers slightly differently, but after that I'd advocate for merging this and adding additional instrumentation and tracing layers to export to KVP in follow-up PRs.

[jeremycline]

The tracing subscriber lets you register multiple layers, and each layer can be configured with its own filters and formatters and so on. It takes a little digging in the tracing_subscriber docs and I forever forget how to do it properly, but here's an example that would let us avoid needing to duplicate the file descriptor:

Suggested change

	// Create a `tracing` subscriber
	let subscriber = Registry::default()
	.with(otel_layer);
	// Create a tracing layer for logging to stdout
	let log_file = Arc::new(File::create("example.log").unwrap());
	let stderr_layer = tracing_subscriber::fmt::layer()
	.with_writer(std::io::stderr.and(log_file))
	.with_filter(tracing_subscriber::EnvFilter::from_env("AZURE_INIT_LOG"));
	// Create a `tracing` subscriber
	let subscriber = Registry::default()
	.with(stderr_layer)
	.with(otel_layer);

This adds a second layer which writes to stderr as well as a file, and applies a filter you can define (requires env-filter feature on the tracing-subscriber crate) in the AZURE_INIT_LOG environment variable. We don't necessarily need to use EnvFilter, but I tend to start with it because it's easy to get started with. I tend to abandon it if I start using a configuration file for the application.

If you run with this diff, you'll get logs written to stderr and "example.log" for that layer, and then separately to "spans.log". I don't think we want this literal suggestion, but I think this is the general approach we want to take when setting things up.

We can also apply a global filter that applies to all layers instead (or in addition to) filters on each layer.

[jeremycline]

I would recommend that we move the code within this block into the binary in the azure-init crate. Each application tends to want to configure its logging in its own way - different file names, different environment variables for configuration, different filtering rules, etc.

What would be good in a library is any custom Layers we end up making, if any. These can be consumed (and configured) by applications in their tracing-subscriber setup.

[jeremycline]

The tracing-subscriber setup should probably live here, then we add a root span and instrument the provision call in that span, something like:

    let subscriber = Registry::default().with(layers);
    tracing::subscriber::set_global_default(subscriber).expect("...")
    let root_span = tracing::info_span!("root");
    match provision().instrument(root_span).await {

[peytonr18]

Closed, kept this open for documentation's sake but final implementation was merged in with #133.