From 57c1430615fc95b4e2720857b19b61ea31920876 Mon Sep 17 00:00:00 2001 From: Marc Rousavy Date: Mon, 6 May 2024 13:07:52 +0200 Subject: [PATCH] docs: Update frame buffer size for VisionCamera example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Just a tiny detail, it doesn't really matter much but I thought I wanted to clarify this a bit. There's two formats in video processing; YUV and RGB. For 4k buffers, let's calculate the size of one frame: - YUV = Y(3840 × 2160) + UV(3840 × 2160 / 2) = 12 MB - RGB = (3840 × 2160 × 4) = 33 MB RGB is always BGRA (4 bytes), and YUV is a bi-planar format with a Y plane of 1 byte per pixel, and UV plane of half the size of the image. While VisionCamera implements optimizations to trim the buffers and uses YUV or even compressed YUV whenever possible, almost 90% of the times people need to use RGB because the ML models just work in RGB. So the exact number would be 33.177.600 bytes for a Frame, which is 1.990.656.000 bytes per second (or ~2 GB per second) of data flowing through the Frame Processor. Thank to JSI it does not matter how big the data is, because we only pass references without making any copies - this is the part that should be highlighted here. --- docs/the-new-architecture/landing-page.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/the-new-architecture/landing-page.md b/docs/the-new-architecture/landing-page.md index 6ae1cd3097b..f79fd796a5c 100644 --- a/docs/the-new-architecture/landing-page.md +++ b/docs/the-new-architecture/landing-page.md @@ -189,7 +189,7 @@ You'll notice that with the frequent updates in a transition, React renders fewe The New Architecture removes the [asynchronous bridge](https://reactnative.dev/blog/2018/06/14/state-of-react-native-2018#architecture) between JavaScript and native and replaces it with JavaScript Interface (JSI). JSI is an interface that allows JavaScript to hold a reference to a C++ object and vice-versa. With a memory reference, you can directly invoke methods without serialization costs. -JSI enables [VisionCamera](https://github.com/mrousavy/react-native-vision-camera), a popular camera library for React Native, to process frames in real time. Typical frame buffers are 10 MB, which amounts to roughly 1 GB of data per second, depending on the frame rate. In comparison with the serialization costs of the bridge, JSI handles that amount of interfacing data with ease. JSI can expose other complex instance-based types such as databases, images, audio samples, etc. +JSI enables [VisionCamera](https://github.com/mrousavy/react-native-vision-camera), a popular camera library for React Native, to process frames in real time. Typical frame buffers are ~30 MB, which amounts to roughly 2 GB of data per second, depending on the frame rate. In comparison with the serialization costs of the bridge, JSI handles that amount of interfacing data with ease. JSI can expose other complex instance-based types such as databases, images, audio samples, etc. JSI adoption in the New Architecture removes this class of serialization work from all native-JavaScript interop. This includes initializing and re-rendering native core components like `View` and `Text`. You can read more about our [investigation in rendering performance](https://github.com/reactwg/react-native-new-architecture/discussions/123) in the New Architecture and the improved benchmarks we measured.