diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 9b31b94..bbf0e6d 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -6,20 +6,45 @@ ## The prompt I gave - - -TODO: paste your prompt here. +My Python pipeline runs locally and inside Docker on my Apple Silicon Mac. I pushed the image to Azure Container Registry, but Azure Container Apps Job fails with: "no child with platform linux/amd64 in index". What does this mean and how should I fix it? ## The code or suggestion it returned - +ChatGPT explained that the issue was not in my Python code. The problem was the Docker image platform. Because I built the image on an Apple Silicon Mac, the image was built for: + +linux/arm64 + +But Azure Container Apps expected: + +linux/amd64 + +The suggested fix was to rebuild and push the image with an explicit platform: + +docker buildx build \ + --platform linux/amd64 \ + -t hyfregistry.azurecr.io/week6-pavel-tisner-pipeline:latest \ + --push . -```text -TODO: paste the AI output here. -``` +It also suggested checking the job logs with the container name: + +az containerapp job logs show \ + --name job-week6-pavel-tisner \ + --resource-group rg-hyf-data \ + --container job-week6-pavel-tisner ## What I changed after reviewing it - +I accepted the explanation because it matched the error message exactly. The pipeline and Dockerfile had already worked locally, so the most likely issue was the image architecture, not the application code. + +I rebuilt and pushed the image using docker buildx build --platform linux/amd64 --push. After that, the Azure Container Apps Job was created successfully. + +I verified the result by checking the job logs. The deployed container ran the pipeline successfully: + +starting pipeline +uploaded blob raw/weather/2026-06-15.json +wrote 3 rows to postgres +pipeline complete (today=2026-06-15) + +I also checked Azure Blob Storage and PostgreSQL separately. The blob raw/weather/2026-06-15.json existed in the raw container, and the PostgreSQL table in my schema dev_pavel_tisner contained 3 rows. I reran the pipeline and confirmed that the row count stayed 3, so the ON CONFLICT DO UPDATE upsert worked correctly. -TODO: describe your review here. +I did not change the Python pipeline because of this AI suggestion. The fix was only in the Docker build and push step: I had to build the image for the platform Azure Container Apps expected. \ No newline at end of file diff --git a/Dockerfile b/Dockerfile index ee3c978..f14adfc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,11 +11,10 @@ FROM python:3.11-slim WORKDIR /app -# TODO Task 4: copy requirements.txt (must appear before any COPY src command) +COPY requirements.txt . -# TODO Task 4: install dependencies with pip +RUN pip install --no-cache-dir -r requirements.txt -# TODO Task 4: copy the src/ folder +COPY src/ src/ -# TODO Task 4: set the CMD to run the pipeline (python -m src.pipeline) -CMD ["python", "-c", "raise SystemExit('Dockerfile not finished: Task 4 still pending')"] +CMD ["python", "-m", "src.pipeline"] diff --git a/README.md b/README.md index fa94600..b413357 100644 --- a/README.md +++ b/README.md @@ -53,23 +53,18 @@ az login --use-device-code --tenant 07a14c4e-d88c-42f7-83b3-13af7e57ff3d ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt -cp .env.example .env # fill in real connection strings, never commit them +cp .env.example .env # fill in real connection strings, never commit them; POSTGRES_URL must include sslmode=require set -a && source .env && set +a python -m src.pipeline ``` -## Verifying your deployment (Task 5) +## Verification -After deploying the Container App Job and triggering a run, capture proof: +The Azure Container App Job `job-week6-pavel-tisner` was deployed and triggered successfully. -1. Open the Azure portal, find your Container App Job, open the **Execution - history** blade. -2. Screenshot the most recent successful run. -3. Save the screenshot to `docs/`. -4. Replace this whole section with one called `## Verification` and embed - your screenshot using a Markdown image link. The grader looks for the - `## Verification` heading and a `![alt](docs/your-file.png)` reference - pointing at the image you committed. +The latest execution completed the pipeline and wrote raw JSON to Azure Blob Storage and 3 rows to Azure PostgreSQL. + +![Execution history](docs/execution_history.png) ## Check your score locally diff --git a/docs/execution_history.png b/docs/execution_history.png new file mode 100644 index 0000000..d1cd3f5 Binary files /dev/null and b/docs/execution_history.png differ diff --git a/docs/execution_history.txt b/docs/execution_history.txt new file mode 100644 index 0000000..6dd7bb1 --- /dev/null +++ b/docs/execution_history.txt @@ -0,0 +1,68 @@ +Local pipeline execution + +Command: +python -m src.pipeline + +Result: +INFO starting pipeline +INFO uploaded blob raw/weather/2026-06-15.json +INFO wrote 3 rows to postgres +INFO pipeline complete (today=2026-06-15) + +Blob verification: +Command: +az storage blob list --connection-string "$AZURE_STORAGE_CONNECTION_STRING" --container-name raw --prefix raw/weather/ -o table + +Verified blob: +raw/weather/2026-06-15.json +Content Type: application/json + +Postgres verification: +Schema: +dev_pavel_tisner + +Table: +weather_readings + +Query result: +count: 3 +('amsterdam', 2026-06-15 06:00:00+00:00, 18.5, 72) +('utrecht', 2026-06-15 06:00:00+00:00, 17.8, 75) +('rotterdam', 2026-06-15 06:00:00+00:00, 19.1, 70) + +Rerun behavior: +The pipeline was run more than once on the same day. The row count stayed 3, confirming that ON CONFLICT DO UPDATE prevents duplicates. + +Docker local execution + +Build command: +docker build -t week6-pavel-tisner-pipeline:local . + +Run command: +docker run --rm \ + -e POSTGRES_URL="$POSTGRES_URL" \ + -e AZURE_STORAGE_CONNECTION_STRING="$AZURE_STORAGE_CONNECTION_STRING" \ + week6-pavel-tisner-pipeline:local + +Result: +INFO starting pipeline +INFO uploaded blob raw/weather/2026-06-15.json +INFO wrote 3 rows to postgres +INFO pipeline complete (today=2026-06-15) + +Azure Container Apps Job execution + +Job name: +job-week6-pavel-tisner + +Execution: +job-week6-pavel-tisner-79lqj89 + +Replica: +job-week6-pavel-tisner-79lqj89-7zmr8 + +Logs: +starting pipeline +uploaded blob raw/weather/2026-06-15.json +wrote 3 rows to postgres +pipeline complete (today=2026-06-15) diff --git a/requirements.txt b/requirements.txt index b5d18d7..a157c59 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,8 +6,5 @@ # The pipeline needs the Azure Blob SDK and a Postgres driver. Add them below # with explicit pins. -# TODO: pin azure-storage-blob (uncomment and add a version) -# azure-storage-blob== - -# TODO: pin psycopg2-binary (uncomment and add a version) -# psycopg2-binary== +azure-storage-blob==12.19.0 +psycopg2-binary==2.9.9 diff --git a/src/pipeline.py b/src/pipeline.py index ef1fc8a..352c5f3 100644 --- a/src/pipeline.py +++ b/src/pipeline.py @@ -13,12 +13,17 @@ - Container Job: Data Track/Week 6/week_6__5_container_apps_jobs.md """ +import json import logging import os +from contextlib import closing from datetime import date +import psycopg2 +from azure.storage.blob import BlobServiceClient logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") logger = logging.getLogger(__name__) +logging.getLogger("azure").setLevel(logging.WARNING) # TASK 3 hint: quiet the Azure SDK so its DEBUG output does not drown your own # pipeline logs. The right call lives in Chapter 5 (Viewing logs). @@ -37,9 +42,22 @@ def get_config() -> dict: Raise RuntimeError with a clear message if a required variable is missing. """ - raise NotImplementedError( - "Task 3: read POSTGRES_URL and AZURE_STORAGE_CONNECTION_STRING from os.environ" - ) + postgres_url = os.environ.get("POSTGRES_URL") + azure_storage_connection_string = os.environ.get("AZURE_STORAGE_CONNECTION_STRING") + + if not postgres_url: + raise RuntimeError("Missing required environment variable: POSTGRES_URL") + + if not azure_storage_connection_string: + raise RuntimeError( + "Missing required environment variable: AZURE_STORAGE_CONNECTION_STRING" + ) + + return { + "postgres_url": postgres_url, + "azure_storage_connection_string": azure_storage_connection_string, + "source_name": os.environ.get("SOURCE_NAME", "weather"), + } def fetch_records() -> list[dict]: @@ -49,7 +67,28 @@ def fetch_records() -> list[dict]: one dict with a stable key set (for example: station, timestamp, temperature_c, humidity_pct). """ - raise NotImplementedError("Task 3: return a list of at least one record") + today = date.today().isoformat() + + return [ + { + "station": "amsterdam", + "timestamp": f"{today}T06:00:00Z", + "temperature_c": 18.5, + "humidity_pct": 72, + }, + { + "station": "utrecht", + "timestamp": f"{today}T06:00:00Z", + "temperature_c": 17.8, + "humidity_pct": 75, + }, + { + "station": "rotterdam", + "timestamp": f"{today}T06:00:00Z", + "temperature_c": 19.1, + "humidity_pct": 70, + }, + ] def upload_raw_to_blob(records: list[dict], blob_conn_str: str, source: str) -> str: @@ -62,7 +101,23 @@ def upload_raw_to_blob(records: list[dict], blob_conn_str: str, source: str) -> teacher has pre-created it). Overwrite if the blob already exists so same-day reruns succeed. """ - raise NotImplementedError("Task 1 + Task 3: upload records to blob storage") + blob_name = f"raw/{source}/{date.today().isoformat()}.json" + + blob_service_client = BlobServiceClient.from_connection_string(blob_conn_str) + + container_client = blob_service_client.get_container_client("raw") + + raw_json = json.dumps(records, ensure_ascii=False, indent=2) + raw_bytes = raw_json.encode("utf-8") + + container_client.upload_blob( + name=blob_name, + data=raw_bytes, + overwrite=True, + content_type="application/json", + ) + + return blob_name def write_to_postgres(records: list[dict], postgres_url: str) -> int: @@ -78,7 +133,55 @@ def write_to_postgres(records: list[dict], postgres_url: str) -> int: See Chapter 4 for the connection-and-cursor pattern this is based on. """ - raise NotImplementedError("Task 2 + Task 3: insert rows into Azure Postgres") + with closing(psycopg2.connect(postgres_url)) as conn: + with conn: + with conn.cursor() as cur: + cur.execute(""" + CREATE SCHEMA IF NOT EXISTS dev_pavel_tisner + """) + + cur.execute(""" + SET search_path TO dev_pavel_tisner + """) + + cur.execute(""" + CREATE TABLE IF NOT EXISTS weather_readings ( + id SERIAL PRIMARY KEY, + station TEXT NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + temperature_c DOUBLE PRECISION NOT NULL, + humidity_pct INTEGER NOT NULL, + ingested_at TIMESTAMPTZ DEFAULT NOW(), + UNIQUE(station, timestamp) + ) + """) + + for record in records: + cur.execute( + """ + INSERT INTO weather_readings ( + station, + timestamp, + temperature_c, + humidity_pct + ) + + VALUES (%s, %s, %s, %s) + ON CONFLICT (station, timestamp) DO UPDATE SET + temperature_c = EXCLUDED.temperature_c, + humidity_pct = EXCLUDED.humidity_pct, + ingested_at = NOW() + """, + + ( + record["station"], + record["timestamp"], + record["temperature_c"], + record["humidity_pct"], + ), + ) + + return len(records) def run() -> None: