Data Processing Engines Benchmark

Experiment to publish in a paper for Performance Evaluation of Computer Systems Master degree class

Overview

The project generates synthetic financial transactions and compares Pandas and Apache Spark on identical workloads.
All datasets and result artifacts live inside src/.datalake, making it easy to inspect or reuse the data produced during each run.

Repository layout

src/
├── config.py                # Central settings (seeds, data lake path, scenario sizes)
├── data_generation.py       # NumPy/Faker-based synthetic data builder
├── experiments/
│   ├── pandas_runner.py     # Pandas workflow (load, filter, group by, join)
│   └── spark_runner.py      # Spark workflow with cache/count actions
├── utils/
│   └── system.py            # Process-level metrics helpers
└── main.py                  # Orchestrates dataset prep, runs experiments, saves CSV results

Support files:

• docker-compose.yaml – spins up a local Spark master, two workers and a Jupyter notebook container.
• docs/experiment*.md|csv – notes and sample outputs for past experiment runs.
• Makefile – helper targets to download/extract Spark locally if you do not want Docker.

Running locally (Pandas only or external Spark)

uv run python src/main.py

The script will:

1. Generate each dataset declared in config.CENARIOS if it is missing from .datalake.
2. Execute the Pandas and Spark workflows sequentially.
3. Persist the consolidated metrics to .datalake/resultados_experimento.csv.

To point the script to an existing Spark cluster, set:

export SPARK_MASTER="spark://<host>:7077"
export SPARK_DRIVER_HOST="<reachable-hostname>"

Running with Docker Compose (local Spark cluster)

docker compose up -d          # starts master, workers, and Jupyter notebook
docker compose logs -f spark-driver

Mounts map ./src into every container at /home/src, so the data lake is shared automatically.
From the apache-spark container or attached Jupyter notebook, run python /home/src/main.py to execute the benchmark inside the cluster.

Outputs

• Synthetic datasets: src/.datalake/dataset_<SCENARIO>_{transactions|users}.csv
• Experiment summary: src/.datalake/experiment_results.csv

You can duplicate or archive the entire .datalake directory to keep artifacts for different experiment sessions.