Render terminal texture with direct Vello path#107
Open
gold-silver-copper wants to merge 17 commits into
Open
Conversation
Owner
|
Can you also update the rendering pipeline docs? I think it will help with understanding this PR better as well. |
Contributor
Author
|
will do |
5ff72be to
aa1392e
Compare
Cell metrics were floored to whole physical pixels at render scale 1.0 (the forced default scale-factor override), so font-size steps whose advance delta was under one pixel changed only the cell height and zoom collapsed into a vertical-only stretch. - update to parley_ratatui 0.3.0 and opt into fractional cell quantization so both axes scale on every font-size step - stop overriding the display scale factor by default; windowed rendering now measures and draws at native scale, and the config scale_factor remains available as an explicit override - interpret the configured font size as points (1pt = 4/3px), so the default 18 renders at 18pt
The direct render path stored the Vello GpuRenderer as a non-send render-world resource, which pinned render_terminal_frame to a specific thread. During shutdown the pipelined render app runs one final update with the main thread already gone from the rendezvous, so the Render schedule parked forever waiting for the non-send system and the main thread deadlocked in World::clear_all waiting for the render app to return. vello::Renderer is Send but not Sync (a RefCell inside its WgpuEngine), so wrap it in SyncCell to qualify as a regular resource with exclusive access.
Vello writes sRGB-encoded, display-ready bytes into its Rgba8Unorm storage target (verified by readback probe), but Bevy sampled that texture as linear and re-encoded at the swapchain, washing out colors relative to the old Rgba8UnormSrgb readback path. Sample through a separate Rgba8UnormSrgb view (via view_formats and the image's texture_view_descriptor) while Vello keeps rendering through a plain Rgba8Unorm storage view. Frames that arrive before the GPU image is prepared, or while its size lags a resize, were silently recycled, leaving the texture stale or uninitialized. Retain such frames and retry on the next render frame (newer published frames supersede retained ones during extraction), and zero-fill the texture so it can never present uninitialized memory.
- Make TerminalRuntime and TerminalSurface regular Send resources (SyncCell around the PTY receiver and master) so systems using them are no longer pinned to the main thread. - Split the redraw god-system into an ordered TerminalRedrawSet pipeline (widget render, material sync, deferred model load) gated by a shared frame-dirty flag. - Render the terminal Vello scene inside the RenderGraph schedule's Begin set instead of RenderSystems::Prepare, ordering its submission with the frame's GPU work. - Run the winit loop with WinitSettings::continuous() in both focus states and drop the Instant-based redraw throttle, so PTY output and animations are never paced by timers. Dirty-flag coalescing stays: each update drains all pending PTY chunks into one scene build.
Owner
|
Getting this while trying to run: |
…hanges Rebuild the terminal texture on terminal-state changes or blink ticks (4Hz) rather than every frame, and only take `get_mut` on the terminal image and plane materials when the size, texture handle, or cull mode actually changes, so Bevy does not re-extract and re-upload unchanged assets. Gate the presentation, inline-object, Mobius, plane-warp, and cursor-sync systems behind run conditions so they only run when their state changes or for the relevant presentation mode.
0.3.1 snaps cell background fills to the pixel grid (fixing base-colour bleed between cells with fractional cell sizes) and renders block elements such as the half block as exact geometric fills instead of font glyphs, which only filled the em box and left gaps in the taller cell. Document why the terminal texture keeps linear filtering: it is mapped onto transformable/scaled 3D geometry, so point sampling would pixelate it; the seam fixes are now made at authoring time in parley_ratatui.
…esent texture
The terminal texture was a single Rgba8Unorm storage texture that also
exposed an Rgba8UnormSrgb view for sampling. wgpu rejects an sRGB view of
a storage texture, so on stricter backends bind-group creation failed
("TextureView with '' label is invalid") and the app quit.
Split it into two textures: Vello rasterizes into a plain Rgba8Unorm
storage render texture (no sRGB view), and each frame that is copied into
a separate Rgba8Unorm present texture that carries the Rgba8UnormSrgb view
the plane material and sprite sample. The present texture has no storage
usage, so the sRGB view is valid on every backend, and the copy is a
same-format texel copy. Colors are unchanged; the invalid combination is
gone by construction.
Inspired by linebender/bevy_vello, which keeps sRGB off the storage
texture.
Contributor
Author
|
The error you got should be fixed, was a Vulkan vs Metal wgpu issue. Could you try rerunning to test, i dont have a decent linux machine here for it |
The render/present texture split added an eighth parameter to update_direct_terminal_frame, tripping clippy::too_many_arguments (8/7) and failing `cargo clippy --all-targets -- -D warnings`. Group the two handles into a TerminalImages struct, which also reads better at the call site. No behavior change.
Both are only called internally by resize_to_fit; they don't need to be part of the crate's public surface.
Contributor
Author
|
PR is ready for review here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces Ratty's CPU-readback terminal rendering with a direct Vello→Bevy GPU texture path:
parley_ratatuirecords a Vello scene, the scene is handed to Bevy's render world through a double-buffered exchange, and Vello rasterizes it straight into a Bevy-owned GPU texture with no CPU readback. On top of that core change it reworks the scale/zoom model to render at native display scale, makes the terminal systemsSendso they run continuously and in parallel, gates per-frame work behind dirty/state checks, and fixes a series of color, scale, shutdown, and GPU-validation bugs.Updates
parley_ratatuito0.3.1and Bevy to0.19.0-rc.2.Fixes #97.
1. New direct Vello→Bevy render path (replaces CPU readback)
The old path rendered the terminal on a private offscreen
wgpudevice, read the pixels back to the CPU, and uploaded them into a BevyImageevery frame. That whole bridge is gone.src/direct_render.rs(new file) —DirectTerminalRenderPlugin:DirectTerminalSceneExchange— anArc-backedResourceshared between the main world and theRenderApp. Its inner state is two single-slot, mutex-guarded mailboxes:pending: Mutex<Option<DirectTerminalFrame>>(latest main→render frame, newest wins) andrecycled: Mutex<Option<Scene>>(one spare VelloSceneflowing render→main).publish_framereplaces the pending slot and recycles any displaced frame's scene;take_recycled_scene/recycle_scene(which callsscene.reset()and keeps at most one) implement double-buffered scene ownership so neither side reallocates a scene per frame.update_direct_terminal_frame(main side) — does the double-buffer dance: take a recycled scene, swap it into theTerminalRenderer, build the scene from the Ratatui buffer (build_scene_with_elapsed), swap the spare back, andpublish_framewith both image handles, size, andbase_color.extract_terminal_frame(ExtractSchedule) — moves the pending frame into the render world'sExtractedDirectTerminalFrame, recycling any frame it supersedes.render_terminal_frame(RenderGraph, setRenderGraphSystems::Begin) — lazily builds the VelloGpuRenderer, rasterizes into the render texture, and copies into the present texture. Placed inBeginso Vello's submission is ordered before the camera passes that sample the terminal texture.DirectTerminalRenderState { renderer: SyncCell<Option<GpuRenderer>> }.vello::RendererisSendbut notSync, so a plain resource is impossible; the previous non-send resource pinnedrender_terminal_frameto a thread and deadlocked the pipelined render app's final update during shutdown (World::clear_allwaited forever).SyncCellmakes it a normalSendresource.src/terminal.rs— deletes the entireOffscreenGpustruct (its ownwgpu::Device/Queue,GpuRenderer,TextureTarget,TextureReadback, CPUrgba: Vec<u8>), thepollster::block_oninit, and therender_to_rgba8…→image.data.copy_from_slicereadback.TerminalSurfacebecomes a regular#[derive(Resource)].Cargo.toml— removespollster(no CPU readback to block on).README.md— rewrites the pipeline section: the terminal image is now "fully GPU-resident; the only data crossing from the main world to the render world each frame is the recorded scene, not pixels."src/lib.rs— addsmod direct_render;.2. Color correctness & GPU-validation safety (sRGB / storage split)
Vello writes sRGB-encoded, display-ready bytes into its
Rgba8Unormstorage target (confirmed by a readback probe). Two problems had to be solved, and the final design uses two textures:new_terminal_render_image— the texture Vello rasterizes into:Rgba8Unorm,STORAGE_BINDING | COPY_SRC | COPY_DST, no sRGB view. Vello binds it as a compute storage target, which requires a plain (non-sRGB) view.new_terminal_image— the present texture materials/sprite sample:Rgba8Unorm,TEXTURE_BINDING | COPY_DST(no storage), withview_formats = [Rgba8UnormSrgb]and atexture_view_descriptorso it is sampled through an sRGB view that decodes the bytes on read (rather than treating them as linear and re-encoding at the swapchain, which washed out colors).render_terminal_framerasterizes into the render texture, thencopy_texture_to_texture(a plain same-formatRgba8Unorm→Rgba8Unormtexel copy) into the present texture each frame.Why two textures: earlier the present texture was the storage texture with an sRGB view bolted on. wgpu rejects an sRGB view of a storage texture, so on stricter backends bind-group creation failed (
TextureView with '' label is invalid) and the app quit. Splitting them keeps the sRGB view on a non-storage texture (Bevy's standard render-target pattern, valid on every backend); the storage texture has no sRGB view. Inspired bylinebender/bevy_vello, which keeps sRGB off the storage texture.Frame retry & zero-fill — frames that arrive before the GPU image is prepared, or while its size lags a resize, are now retained and retried on the next render frame (a newer published frame supersedes a retained one during extraction) instead of being silently dropped, and both textures are zero-filled so a frame sampled before the first copy reads transparent black rather than uninitialized memory.
src/rendering.rs—sync_plane_textureonly takesmaterials.get_mut(...)when thebase_color_texturehandle actually differs (checked via an immutableget), becauseget_mutmarks the material modified and forces a GPU re-prepare.3. Scale & zoom model (fixes #97)
Previously cell metrics were floored to whole physical pixels at a forced scale factor of
1.0, so a font-size step whose advance delta was under one pixel changed only the cell height — zoom collapsed into a vertical-only stretch.WindowConfig.scale_factor(src/config.rs) changes fromf32(default1.0) toOption<f32>(defaultNone= "use the display's scale factor").src/main.rs's newwindow_resolution()helper applieswith_scale_factor_override(...)only when the config sets it; otherwise the window renders at the display's real scale factor.config/ratty.tomlcomments outscale_factorwith documentation.render_scale_for_window()(src/terminal.rs) chooses its base scale factor asscale_factorwhen a Bevy override is set, elsebase_scale_factor()— "a scale-factor override already defines the physical/logical ratio; applying the backend factor on top of it over-sizes fullscreen terminal textures."build_terminal_rendererbuilds theparley_ratatuirenderer withCellQuantization::Fractionalandnew_scaled(..., render_scale), so both axes grow on every font-size step. A regression test (font_size_steps_scale_cells_on_both_axes) assertscell_width/cell_heightstrictly increase across sizes 8..=24 at scales 1.0 and 2.0.FontConfig.sizeis now interpreted as points; the renderer appliesPT_TO_PX = 96.0/72.0before handing the size to Parley (so the default18renders at 18pt).config/ratty.tomldocuments this.TerminalLayout(new) bundlescols,rows,texture_size: UVec2(physical px),logical_size: Vec2(Bevy world units), andrender_scale;pty_pixels()clamps to portable-pty'su16API.resize_to_fit(logical_size, render_scale)recomputes the grid fromrenderer.logical_metrics(render_scale)(logical metrics keep the column/row count stable across HiDPI scales) and returns a layout that the resize/render/input systems all consume.set_render_scalerebuilds the renderer;char_dimensionsnow returns fractionalVec2logical metrics instead of a ceiledUVec2;snapped_translation()snaps the sprite to the physical pixel grid (the "align zoom resize layout" fix).4. Scene & presentation
src/scene/mod.rs:setup_scene— createsnew_terminal_render_image(storage,terminal.render_image_handle) andnew_terminal_image(present,terminal.image_handle); the legacycreate_terminal_imageis kept only for the 3D "back" debug image.setup_scenebecomes aSetupSceneParamssystem param that reads the realPrimaryWindowandTerminalRuntime, computeslayout = terminal.resize_to_fit(window_size, render_scale_for_window(window)), resizes the PTY tolayout.pty_pixels(), and derives all image sizes, theTerminalViewport, the spritecustom_size/snapped translation, and both plane transform scales fromlayoutinstead ofapp_config.window.{width,height}.TerminalSprite(2D, samples the present texture),TerminalPlane+TerminalPlaneBack(3D meshes,StandardMaterial { unlit: true, alpha_mode: Blend }; front samples the present texture, back samples the debug image). Two cameras:Camera2d(order: 0) andTerminalPlaneCamera(Camera3d,order: 1, orthographic,clear_color: Noneby default).apply_terminal_presentationonly mutates GPU state when it changes — front-materialcull_mode(Nonefor Mobius's double-sided ribbon, elseSome(Face::Back)) is only taken viaget_mutwhen it differs; the 2D camera'sis_active = !is_3dand the 3D camera'sclear_color(the 3D camera stays active everywhere — it renders the cursor model/RGP objects even in 2D — so "whichever camera renders first owns the screen clear") are only assigned when changed.sync_terminal_layout(new) re-applies aTerminalLayoutto live entities (viewport, sprite size/translation, plane scales) on resize, backed by newTerminalSpriteLayoutQuery/TerminalPlaneLayoutQuery/TerminalPlaneBackLayoutQueryaliases.src/config.rs— addsTERMINAL_RENDER_TEXTURE_LABELfor Vello's storage texture and clarifiesTERMINAL_TEXTURE_LABELas the sampled present texture.5. Scheduling, parallelism & performance
Sendresources.src/runtime.rs:TerminalRuntimebecomes#[derive(Resource)]by wrapping its!SyncPTY handles inSyncCell(rx: SyncCell<Receiver<Vec<u8>>>,master: SyncCell<Option<Box<dyn MasterPty + Send>>>) and exposingtry_recv(&mut self). WithTerminalRuntimeandTerminalSurfacebothSend,src/main.rsinserts them withinsert_resource(wasinsert_non_send_resource) and every consuming system insrc/systems.rsflipsNonSend*→Res*, so Bevy can schedule them off the main thread / in parallel.pump_pty_outputnow callsruntime.try_recv().src/main.rsreplaces the per-focus reactive throttle (UpdateMode::reactive_low_power(FOCUSED_UPDATE_INTERVAL)) withWinitSettings::continuous()— Bevy's default switches unfocused windows to reactive mode, which would delay background PTY output. TheInstant-based redraw throttle inTerminalRedrawStateis removed.TerminalRedrawSetpipeline. The oldredraw_soft_terminalgod-system (which redrew, uploaded, refreshed the back texture, synced materials, and drove cursor-model loading) is split into three.chain()-ed systems in a newTerminalRedrawSet:render_terminal_widget→sync_terminal_materials→finish_terminal_model_load, ordered.after()input/resize/PTY systems.render_terminal_widgetcomputes a sharedTerminalFrameDirtyfromneeds_redraw || blink_ticked || !model_loaded, whereblink_tickeduses aLocal<u64>bucket overBLINK_TICK_SECS = 0.25(rapid blink half-period; slow blink is a multiple). Texture content only changes with terminal state or blink phase (warp/camera motion is mesh-/camera-side), so an idle terminal re-rasterizes only ~4×/s instead of every frame, and the downstream pipeline systems early-return on clean frames.sync_imagelikewise only takesget_muton each image when its size changes.src/plugin.rs) so continuous mode doesn't burn work:apply_terminal_presentationruns only when presentation/plane-view/Mobius stateis_changed();apply_inline_objectsonly when presentation changed or inline-object entities wereAdded;sync_rgp_objectsonly when RGP objects exist;animate_mobius_transitiononly in Mobius mode or during a transition;animate_terminal_plane_warponly when notFlat2d;sync_asset_to_terminal_cursoronly when the cursor model is visible.src/systems.rshandle_window_resizeis simplified to callterminal.resize_to_fit(...)+sync_terminal_layout(...)instead of inlining grid/viewport math, and the redraw pipeline (not the resize system) uploads the resized image.6. Input handling under the new layout
src/keyboard.rs— font-zoom bindings now resize through the window-based layout model: instead of computing the grid from the viewport and integer char dimensions, it callsterminal.resize_to_fit(window_size, render_scale_for_window(window)), resizes the PTY fromlayout.pty_pixels(), and propagates geometry withsync_terminal_layout(...). This keeps zoom proportional under fractional cell metrics. Params switchNonSendMut→ResMutand gain the window + layout queries.src/mouse.rs— mouse-to-cell mapping is corrected for a viewport that is smaller than and centered in the window (letterbox/pillarbox under the new scale model).position_to_cellgains awindow_sizeparameter, subtracts the half-margin ((window_size - viewport.size) * 0.5), returnsNonewhen the cursor is outside the terminal area, and then clamps within the viewport — previously it clamped raw window coordinates, assuming the viewport filled the window from the origin. The wheelPixelbranch drops a now-unneeded.max(1) as f32sincechar_dimensions()returns fractionalVec2.7. Dependency & Bevy 0.19 migration
Cargo.toml—bevy0.18.1→0.19.0-rc.2; adds thebevy_scenefeature;parley_ratatui0.2.1→0.3.1; removespollster.parley_ratatui0.3.1 also fixes terminal-cell rendering upstream: it snaps cell background fills to the pixel grid (no base-color bleed between cells with fractional cell sizes) and renders block elements such as the half block (▀) as exact geometric fills instead of font glyphs (which only filled the em box and left gaps in the taller line-height cell). The present texture keeps linear filtering because it is mapped onto transformable/scaled 3D geometry where point sampling would pixelate it; the seam fixes are made at authoring time inparley_ratatui, not by the sampler.Scene→WorldAsset/SceneRoot→WorldAssetRoot:src/inline.rs(Handle<Scene>→Handle<WorldAsset>) andsrc/model.rs(SceneRoot→WorldAssetRoot) are mechanical migrations.insert_non_send_resource/init_non_send_resource→insert_non_send/init_non_send(AppWindowIcon,TerminalClipboard);StandardMaterial.shadows_enabled→shadow_maps_enabled;apply_plane_warptakesOption<AssetMut<'_, Mesh>>.Bug fixes in this PR (by commit)
TextureView … is invalidcrash on strict backends (sRGB view on a storage texture).