fix: avoid OSS multipart limit in Parquet writes

[jerry-024]

Purpose

Paimon Rust previously created OpenDAL writers without an explicit chunk size. Parquet writes use OpenDAL FuturesAsyncWriter, which buffers 256 KiB at a time. Because the minimum OSS multipart part size is 100 KiB, OpenDAL forwarded each 256 KiB buffer as a separate part instead of coalescing it.

OSS allows at most 10,000 parts per multipart upload, which limited a single Parquet file to about 2.44 GiB before the upload exceeded the part-count limit.

Brief change log

• Create the OpenDAL writer with writer_with.
• Set the writer chunk size to 8 MiB so OpenDAL coalesces Parquet writes into larger multipart parts.
• Increase the resulting multipart size ceiling from about 2.44 GiB to about 78.125 GiB.

Tests

• cargo fmt --all -- --check
• The targeted FileIO write/read test was blocked locally by the existing paimon-vindex-core use of the unstable stdarch_neon_f16 feature on aarch64 Rust 1.92.0; CI will run the repository test suite.

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required.

[JingsongLi]

+1