Integrate 3dtoolkit into desert scene

[anderm]

[bengreenier]

I made a ton of progress on this today see https://github.com/bengreenier/unity-drones-multiplayer/tree/3dtoolkit-desert

[morsh]

Guys, looks like a lot of work, when you guys have come back online, let's have a sync over those things :)
@bengreenier, @anderm

[bengreenier]

Sure! Since syncs might be tricky I'll try to keep a better log of what i'm doing here. 😸

So far, the goal is to enable the desert scene to stream, and support multi-peers. This uses our pending 3dtoolkit multi-peer work. This requires some changes to the mental model, as the server and the desert experiences effectively merge. This is because when we're streaming, the server is running the core experience on behalf of all users, and is just streaming video and accepting input from a bunch of "dumb" clients. per #4 we're going to modify one of our existing clients to support MR and input from an xbox controller - so that will be our "dumb" client.

At this point, I have the core 3dtoolkit plugin behaving correctly in the desert scene, and my focus will now shift to ensuring the streaming video works as expected (not always as trivial as it sounds, unfortunately). Then I'll be making some changes to enable managing the lifecycle of all the drones in the desert as 3dtoolkit connections get created/removed. I'll try to keep all my work prefab managed, so we can avoid scene merges and make it easier to enable this scenario going forward.

[bengreenier]

f9ba598 supports streaming! it needs a flip fix (the content is upside down in a mono feed) but it's functional and mouse controls enable movement for a single client.

[bengreenier]

note: this required me to change your project default resolution to our supported streaming resolution (currently there isn't flexibility here).

[bengreenier]

bengreenier-archive@9213e07 flips the render, so we get the following client rendering now:

[bengreenier]

Tried to modify webrtc to run at 120fps (or really, any arbitrary fps) but am stuck there. It's quite complicated to update all the webrtc plumbing. I believe I modified all the transmission framerates to run at 120, but i am only receiving ~50fps still. I'm investigating receiving framerate logic to see if that's also a blocker, but i can't really verify that my transmission is running at 120 without a receiver so i'm unable to verify where the issue really lies.

This will be very tricky to unblock, I'm not confident i'll have this working by Wednesday.

[tylergibson]

webrtc currently caps out at 60fps. There are a ton of places, especially in the video jitter engine that aren't really plumbed to handle greater than 60fps transmission. If we really need >60fps, then we would be better off using a custom data channel and sending video frames through it, bypassing the video engine entirely. You also need to ensure Unity has vsync disabled and the host system has vsync disabled.

We also need to make sure in our DirectX samples that we disable vsync on present

[bengreenier]

thanks @tylergibson, that makes sense. The work mentioned above was to investigate the cost of modifying those places in webrtc that do cap out at 60fps. As you correctly assumed, and we determined, that work is non-trivial ☹️