Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
2e218a0
Batch builder API
gbrgr May 11, 2026
83a8eb5
Fix tests, format
gbrgr May 11, 2026
48f7e09
Fix date test: access columns via tbl.col[1] not tbl[1].col
gbrgr May 11, 2026
f554297
Fix date/timestamp test: compare Arrow wrapper .x values directly
gbrgr May 11, 2026
6477aa1
batch_builder: extract append_primitive!/append_transform! macros, me…
gbrgr May 11, 2026
d0e3e56
.
gbrgr May 11, 2026
a953ae9
.
gbrgr May 11, 2026
a4d9192
prefetching
gbrgr May 12, 2026
407b813
Format
gbrgr May 12, 2026
8db1a0b
Use spawn_blocking instead of encode pool
gbrgr May 12, 2026
dfdb437
Format
gbrgr May 12, 2026
2556bc9
Revert "Format"
gbrgr May 12, 2026
c4d0d2f
Revert "Use spawn_blocking instead of encode pool"
gbrgr May 12, 2026
5d89e00
writer: per-writer queues + work-stealing encode pool
rgankema May 20, 2026
e75ad28
writer: add LZ4_RAW compression codec
rgankema May 20, 2026
dd04750
gitignore: ignore .vscode/settings.json
rgankema May 20, 2026
67723b6
writer: run encode workers as async tokio tasks instead of OS threads
rgankema May 22, 2026
e250805
writer: default encode worker count to 2 * available_parallelism
rgankema May 22, 2026
171ff87
writer: collapse Batch/Builder API to RowChunk + append! + flush!
rgankema May 22, 2026
8ded709
writer: defer FFI prep into append!; reuse scratch in writer
rgankema May 26, 2026
d3a4784
writer: reusable RowChunk + empty!; build ColumnSlice inline at push!…
rgankema May 26, 2026
dfeca64
Fix segfault
rgankema May 27, 2026
867c7bb
writer: source-aligned validity semantics; gather through sel
rgankema May 27, 2026
03121ad
writer: accept any contiguous AbstractVector{Int64} as sel
rgankema May 27, 2026
e1512b0
doc cleanup
rgankema May 27, 2026
ebb5248
Merge remote-tracking branch 'origin/main' into rg/batch-builder
rgankema May 27, 2026
15219d4
ci: satisfy rustfmt + clippy
rgankema May 28, 2026
a435adb
Merge remote-tracking branch 'origin/main' into rg/batch-builder
rgankema May 28, 2026
85a23b8
Bump version to 0.8.2; align writer exception codes with deferred-err…
rgankema May 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ iceberg_rust_ffi/integration_test
**/*.dylib
**/.claude
**/.DS_Store
LocalPreferences.toml
LocalPreferences.toml
.vscode/settings.json
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name = "RustyIceberg"
uuid = "390bdf5b-b624-43dc-a846-0ef7a3405804"
version = "0.8.1"
version = "0.8.2"

[deps]
Arrow = "69666777-d1a9-59fb-9406-91d4454c9d45"
Expand Down
2 changes: 1 addition & 1 deletion iceberg_rust_ffi/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion iceberg_rust_ffi/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "iceberg_rust_ffi"
version = "0.8.1"
version = "0.8.2"
edition = "2021"

[lib]
Expand Down
8 changes: 6 additions & 2 deletions iceberg_rust_ffi/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,13 @@ mod transaction;
// Writer module
mod writer;

// Column-based writer module (zero-copy from Julia)
// Shared FFI structs/constants for the column-based write path
mod writer_columns;

// Incremental RecordBatch builder: per-slice copy into owned buffers, finalize to RecordBatch.
// Embedded inside `IcebergDataFileWriter`; not directly exposed across the FFI.
mod record_batch_builder;

// Profiling stats for the file-parallel pipeline
mod pipeline_stats;

Expand Down Expand Up @@ -80,7 +84,7 @@ pub use transaction::{IcebergDataFiles, IcebergTransaction, IcebergTransactionRe
pub use writer::{
IcebergDataFileWriter, IcebergDataFileWriterResponse, IcebergWriterCloseResponse,
};
pub use writer_columns::ColumnDescriptor;
pub use writer_columns::ColumnSlice;

// We use `jl_adopt_thread` to ensure Rust can call into Julia when notifying
// the Base.Event that is waiting for the Rust result.
Expand Down
Loading
Loading