Pavel Tisner

.github/workflows/ci.yml

@@ -15,7 +15,7 @@ name: CI
 
 on:
   push:
-    branches: ["TODO-replace-with-main"]
+    branches: ["main"]
   pull_request:
 
 jobs:
@@ -29,10 +29,10 @@ jobs:
       - name: Install dependencies
         run: pip install -r requirements.txt
       - name: Lint
-        run: echo "TODO implement this step"
+        run: ruff check src tests
       - name: Format
-        run: echo "TODO implement this step"
+        run: ruff format --check src tests
       - name: Test
-        run: echo "TODO implement this step"
+        run: pytest -q
       - name: Build image
-        run: echo "TODO implement this step"
+        run: docker build -t pavel-tisner-pipeline:${{ github.sha }} .

AI_ASSIST.md

@@ -6,20 +6,69 @@
 
 ## The prompt I gave
 
-<!-- Paste the exact prompt you gave to an LLM (ChatGPT, Claude, Copilot, etc.). -->
+My pull request checks were failing even though my own CI workflow had passed. GitHub only showed that the HYF autograder ran `bash test.sh` and exited with code 1. I asked ChatGPT to help me understand what was failing and how to debug the autograder output.
 
-TODO: paste your prompt here.
+I also shared the output from running the grader locally, including:
+
+```bash
+
+bash .hyf/test.sh > test-output.txt 2>&1
+echo $?
+cat test-output.txt
+```
+and then the trace output from:
+
+```
+bash -x .hyf/test.sh
+```
 
 ## The code or suggestion it returned
 
-<!-- Paste the code or key suggestion the LLM returned. -->
+The assistant suggested that the visible GitHub error was not enough, because it only showed that test.sh exited with code 1. It suggested running the autograder locally with shell trace mode:
+
+```
+bash -x .hyf/test.sh
+```
+
+The assistant then helped me read the trace. The important part of the trace showed that the grader was checking all Python files inside src/, not only the main pipeline file. It found several old print() calls:
+
+```
+src/ingest_files.py: print(...)
+src/ingest_api.py: print(...)
+src/validate.py: print(...)
+```
+
+The assistant also pointed out that the grader was searching for the text NotImplementedError in src/pipeline.py. In my case, this text was not executable code anymore. It was leftover starter-template text inside the docstring, but the static grader still detected it.
+
+The assistant suggested checking the project with:
 
-```python
-# TODO: paste the AI-generated code here
 ```
+grep -R "NotImplementedError\|print(" src tests
+```
+
+and then removing the leftover print() calls and the leftover NotImplementedError text.
 
 ## What I changed after reviewing it
 
-<!-- Describe what you accepted, rejected, or modified, and why. -->
+After reviewing the trace, I removed the old debug print() calls from the helper modules:
+
+```
+src/ingest_api.py
+src/ingest_files.py
+src/validate.py
+```
+
+These print() calls were left over from the Week 3 version of the pipeline and were only used for manual debugging. The Week 5 assignment requires logging instead of print() for pipeline status, and the autograder checks all files under src/.
+
+I also removed the leftover NotImplementedError text from the docstring in src/pipeline.py. It was not part of running code, but the static grader still searched for that string and marked it as a remaining starter stub.
+
+After making those changes, I reran:
+
+```
+ruff check src tests
+ruff format --check src tests
+pytest -q
+bash .hyf/test.sh
+```
 
-TODO: describe your review and any changes you made.
+The checks passed after the cleanup. This was useful because the problem was not in the pipeline logic or Dockerfile. The issue was leftover debug output and starter-template text that the autograder detected during static analysis.

Dockerfile

@@ -9,16 +9,15 @@
 #
 # Replace each TODO comment with the correct Dockerfile instruction.
 
-# TODO: set the base image
-FROM TODO
+FROM python:3.11-slim
 
 WORKDIR /app
 
-# TODO: copy requirements.txt (before source — this keeps the install layer cached)
+COPY requirements.txt .
 
-# TODO: install dependencies
+RUN pip install --no-cache-dir -r requirements.txt
 
-# TODO: copy source code
+COPY src/ src/
+COPY data/ data/
 
-# TODO: set the command that runs when the container starts
-CMD ["TODO"]
+CMD ["python", "-m", "src.pipeline"]

data/weather_stations.csv

@@ -0,0 +1,11 @@
+station,timestamp,temperature_c,humidity_pct
+Copenhagen,2025-01-15T10:00,18.5,72
+,2025-01-15T11:00,20.1,65
+Aarhus,2025-01-15T12:00,N/A,58
+Odense,2025-01-15T13:00,15.2,150
+Copenhagen,2025-01-15T10:00,19.0,70
+Aalborg,,14.8,62
+Roskilde,2025-01-15T15:00,-95.0,45
+Esbjerg,2025-01-15T16:00,16.3,
+Herning,2025-01-15T17:00,17.1,55
+Kolding,2025-01-15T18:00,14.9,68

requirements.txt

@@ -1,13 +1,4 @@
-# Task 2: Pin every dependency your pipeline uses.
-#
-# Format: package==version
-# Example: requests==2.31.0
-#
-# Find the current version of any package:
-#   pip show <package>
-#
-# Always include pytest and ruff:
-# pytest==
-# ruff==
-#
-# Add your pinned dependencies below:
+pydantic==2.13.4
+pytest==9.0.3
+requests==2.34.2
+ruff==0.15.15

src/database.py

@@ -0,0 +1,135 @@
+# Step 5 — Task 5: Database Storage
+# create_tables()  — run once at startup to set up raw_weather and weather_readings.
+# insert_raw()     — store every record before validation so nothing is lost.
+# upsert_readings()— insert valid records; ON CONFLICT updates instead of duplicating.
+# count_readings() — query the final row count for the pipeline summary.
+import sqlite3
+from pathlib import Path
+
+from src.models import WeatherReading
+
+DB_PATH = Path("weather.db")
+
+
+def get_connection() -> sqlite3.Connection:
+    conn = sqlite3.connect(DB_PATH)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+
+def create_tables(conn: sqlite3.Connection) -> None:
+    """Create raw_weather and weather_readings tables if they do not exist.
+
+    raw_weather columns: id, station, timestamp, temperature_c, humidity_pct, source, ingested_at
+    weather_readings columns: id, station, timestamp, temperature_c, humidity_pct
+        + UNIQUE(station, timestamp) constraint for upserts
+    """
+
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS raw_weather (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            station TEXT,
+            timestamp TEXT,
+            temperature_c TEXT,
+            humidity_pct TEXT,
+            source TEXT NOT NULL,
+            ingested_at TEXT DEFAULT CURRENT_TIMESTAMP
+        )
+        """
+    )
+
+    conn.execute(
+        """
+        CREATE TABLE IF NOT EXISTS weather_readings (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            station TEXT NOT NULL,
+            timestamp TEXT NOT NULL,
+            temperature_c REAL NOT NULL,
+            humidity_pct INTEGER NOT NULL,
+            UNIQUE(station, timestamp)
+        )
+        """
+    )
+
+    conn.commit()
+
+
+def insert_raw(conn: sqlite3.Connection, records: list[dict], source: str) -> None:
+    """Insert raw records (before validation) into raw_weather.
+
+    Use parameterized queries with placeholder syntax; do not build SQL via string formatting.
+    """
+
+    rows = [
+        (
+            record.get("station"),
+            record.get("timestamp"),
+            record.get("temperature_c"),
+            record.get("humidity_pct"),
+            source,
+        )
+        for record in records
+    ]
+
+    conn.executemany(
+        """
+        INSERT INTO raw_weather (
+            station,
+            timestamp,
+            temperature_c,
+            humidity_pct,
+            source
+        )
+        VALUES (?, ?, ?, ?, ?)
+        """,
+        rows,
+    )
+
+    conn.commit()
+
+
+def upsert_readings(conn: sqlite3.Connection, readings: list[WeatherReading]) -> None:
+    """Upsert valid WeatherReading objects into weather_readings.
+
+    Use the upsert pattern to handle duplicate (station, timestamp) pairs.
+    Use parameterized queries.
+    """
+
+    rows = [
+        (
+            reading.station,
+            reading.timestamp,
+            reading.temperature_c,
+            reading.humidity_pct,
+        )
+        for reading in readings
+    ]
+
+    conn.executemany(
+        """
+        INSERT INTO weather_readings (
+            station,
+            timestamp,
+            temperature_c,
+            humidity_pct
+        )
+        VALUES (?, ?, ?, ?)
+        ON CONFLICT(station, timestamp)
+        DO UPDATE SET
+            temperature_c = excluded.temperature_c,
+            humidity_pct = excluded.humidity_pct
+        """,
+        rows,
+    )
+
+    conn.commit()
+
+
+def count_readings(conn: sqlite3.Connection) -> int:
+    """Return the total number of rows in weather_readings."""
+
+    cursor = conn.execute("SELECT COUNT(*) FROM weather_readings")
+    result = cursor.fetchone()
+
+    return result[0]