Package and access data on Amazon S3 in a squash manner
-
Updated
Aug 6, 2023 - Rust
#
small-files
Here are 8 public repositories matching this topic...
A file uploader specialized for uploading many small files onto HDFS
-
Updated
Apr 18, 2018 - Java
PySpark script to aggregate small parquet files in a prefix into larger files. Designed to be run on AWS Glue
-
Updated
May 13, 2026 - Python
Command Line Hangman/Word Guessing game.
-
Updated
Sep 4, 2016 - C
Read-only Delta Lake table health scanner with safe fix plans for Databricks, Python, and PySpark.
python spark optimize pyspark data-engineering vacuum databricks small-files delta-lake table-maintenance
-
Updated
May 12, 2026 - Python
Read-only "small-file tax" analyzer for Delta Lake. Reads only the _delta_log to reconstruct the per-partition file-size distribution, then prices the wasted compute and S3 requests in dollars using real Spark internals (openCostInBytes, maxPartitionBytes) and emits a compaction plan with ROI and payback plus the OPTIMIZE SQL. Never runs OPTIMIZE.
python cli spark s3 optimize data-engineering zero-dependency roi cost-optimization small-files finops compaction delta-lake lakehouse data-observability delta-transaction-log
-
Updated
Jul 8, 2026 - Python
File merge action to merge files in HDFS or local filesystem
-
Updated
Apr 12, 2022 - Java
Improve this page
Add a description, image, and links to the small-files topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the small-files topic, visit your repo's landing page and select "manage topics."