Mareh A.

AI_ASSIST.md

@@ -8,18 +8,75 @@
 
 <!-- Paste the exact prompt you gave an LLM (ChatGPT, Claude, Copilot, etc.). -->
 
-TODO: paste your prompt here.
+python pipeline.py
+POSTGRES_URL is not set.
+Retrieve it from Key Vault using the CLI, then export it before running:
+
+    export POSTGRES_URL="$(az keyvault secret show --vault-name kv-hyf-data --name postgres-url --query value -o tsv)"
+
+Traceback (most recent call last):
+  File "C:\Users\Beheerder\c55-data-week-6\src\pipeline.py", line 170, in <module>
+    run()
+  File "C:\Users\Beheerder\c55-data-week-6\src\pipeline.py", line 152, in run
+    config = get_config()
+             ^^^^^^^^^^^^
+  File "C:\Users\Beheerder\c55-data-week-6\src\pipeline.py", line 53, in get_config
+    raise RuntimeError("missing POSTGRES_URL")
+RuntimeError: missing POSTGRES_URL
+((.venv) ) 
+why i got this error i already add the url to .env
 
 ## The code or suggestion it returned
 
 <!-- Paste the suggestion verbatim — code, shell commands, or both. -->
 
-```text
-TODO: paste the AI output here.
+```You got this error because even though you saved your URL inside the `.env` file, your terminal hasn't actually read that file into its active memory yet.
+
+Python's `os.environ.get()` function doesn't know how to read a `.env` file automatically—it can only look at variables that are actively loaded into your terminal session's environment memory.
+
+Since you are using **Git Bash**, you need to run the specific commands that read your `.env` file and export those secrets into your terminal's memory.
+
+---
+
+### How to fix it (Load your variables)
+
+Make sure you are in your project folder, your `(.venv)` is active, and run this exact command line in your Git Bash terminal:
+
+```bash
+set -a && source .env && set +a
+
+```
+
+### What this just did:
+
+* **`source .env`** told Git Bash to open your `.env` file and read your lines.
+* **`set -a`** and **`set +a`** told your terminal to automatically export those lines into its active memory space so that any Python script run afterward can see them.
+
+---
+
+### Test it again!
+
+To make sure your terminal successfully loaded the variables before running your script, you can print them to the screen using `echo`:
+
+```bash
+echo $POSTGRES_URL
+
+```
+
+*(If you see your connection string print out in the terminal, it worked perfectly!)*
+
+Now, try running your pipeline module again:
+
+```bash
+python -m src.pipeline
+
+```
+
+It should pass right through the `get_config()` check and start executing your cloud pipeline steps!
 ```
 
 ## What I changed after reviewing it
 
 <!-- Describe what you accepted, rejected, or modified, and why. -->
 
-TODO: describe your review here.
+I got another error with the connection string. I discovered that I had entered the URLs for POSTGRES_URL and AZURE_STORAGE_CONNECTION_STRING in my .env file without quotation marks. The AI explained that using double quotation marks ("") is important because it ensures the strings are correctly passed to Python, preventing issues with special characters and protecting critical account details.

Dockerfile

@@ -12,10 +12,10 @@ FROM python:3.11-slim
 WORKDIR /app
 
 # TODO Task 4: copy requirements.txt (must appear before any COPY src command)
-
+COPY requirements.txt .
 # TODO Task 4: install dependencies with pip
-
+RUN pip install --no-cache-dir -r requirements.txt
 # TODO Task 4: copy the src/ folder
-
+COPY src/ src
 # TODO Task 4: set the CMD to run the pipeline (python -m src.pipeline)
-CMD ["python", "-c", "raise SystemExit('Dockerfile not finished: Task 4 still pending')"]
+CMD ["python", "-m", "src.pipeline"]

README.md

@@ -70,6 +70,12 @@ After deploying the Container App Job and triggering a run, capture proof:
    your screenshot using a Markdown image link. The grader looks for the
    `## Verification` heading and a `![alt](docs/your-file.png)` reference
    pointing at the image you committed.
+## Verification
+
+The pipeline has been successfully deployed as an Azure Container App Job and executed.
+Below is the proof of the successful run from the Azure Portal execution history:
+
+![Azure Job Execution Success](docs/execution_history.png)
 
 ## Check your score locally
 

docs/execution_history.txt

@@ -0,0 +1,3 @@
+Name                         StartTime                  Status
+---------------------------  -------------------------  ---------
+mareh-aboghanem-job-u6ikgff  2026-06-10T20:14:29+00:00  Succeeded

requirements.txt

@@ -11,3 +11,5 @@
 
 # TODO: pin psycopg2-binary (uncomment and add a version)
 # psycopg2-binary==
+azure-storage-blob==12.30.0
+psycopg2-binary==2.9.12

src/pipeline.py

@@ -5,7 +5,6 @@
 Database for PostgreSQL. When you finish the assignment it will run as a
 Container App Job triggered from the Azure Portal or the CLI.
 
-Replace every `raise NotImplementedError` below with a real implementation.
 
 Reference chapters:
 - Blob upload:      Data Track/Week 6/week_6__3_azure_blob_storage.md
@@ -16,13 +15,18 @@
 import logging
 import os
 from datetime import date
+import sys
+import json
+from contextlib import closing
+import psycopg2
+from azure.storage.blob import BlobServiceClient
 
 logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
 logger = logging.getLogger(__name__)
 
 # TASK 3 hint: quiet the Azure SDK so its DEBUG output does not drown your own
 # pipeline logs. The right call lives in Chapter 5 (Viewing logs).
-
+logging.getLogger("azure").setLevel(logging.WARNING)
 
 def get_config() -> dict:
     """Return configuration read from environment variables.
@@ -37,9 +41,29 @@ def get_config() -> dict:
 
     Raise RuntimeError with a clear message if a required variable is missing.
     """
-    raise NotImplementedError(
-        "Task 3: read POSTGRES_URL and AZURE_STORAGE_CONNECTION_STRING from os.environ"
-    )
+    conn_postgres=os.environ.get("POSTGRES_URL")
+    if not conn_postgres:
+        logging.info(
+            "POSTGRES_URL is not set.\n"
+            "Retrieve it from Key Vault using the CLI, then export it before running:\n\n"
+            "    export POSTGRES_URL=\"$(az keyvault secret show --vault-name kv-hyf-data --name postgres-url --query value -o tsv)\"\n",
+            file=sys.stderr,
+        )
+        raise RuntimeError("missing POSTGRES_URL")
+    conn=os.environ.get("AZURE_STORAGE_CONNECTION_STRING")
+    if not conn:
+        logging.info(
+            "AZURE_STORAGE_CONNECTION_STRING is not set.\n"
+            "Retrieve it from Key Vault using the CLI, then export it before running:\n\n"
+            "    export AZURE_STORAGE_CONNECTION_STRING=\"$(az keyvault secret show --vault-name kv-hyf-data --name storage-connection-string --query value -o tsv)\"\n",
+            file=sys.stderr,
+        )
+        raise RuntimeError("missing AZURE_STORAGE_CONNECTION_STRING")
+    return {
+        "postgres_url": conn_postgres,
+        "azure_storage_connection_string": conn,
+        "source_name": os.environ.get("SOURCE_NAME", "weather"),
+    }
 
 
 def fetch_records() -> list[dict]:
@@ -49,7 +73,13 @@ def fetch_records() -> list[dict]:
     one dict with a stable key set (for example: station, timestamp,
     temperature_c, humidity_pct).
     """
-    raise NotImplementedError("Task 3: return a list of at least one record")
+    mock_record = {
+        "station": "Amsterdam",
+        "timestamp": "2024-06-01T12:00:00Z",
+        "temperature_c": 20.5,
+        "humidity_pct": 60,
+    }
+    return [mock_record]
 
 
 def upload_raw_to_blob(records: list[dict], blob_conn_str: str, source: str) -> str:
@@ -62,7 +92,12 @@ def upload_raw_to_blob(records: list[dict], blob_conn_str: str, source: str) ->
     teacher has pre-created it). Overwrite if the blob already exists so
     same-day reruns succeed.
     """
-    raise NotImplementedError("Task 1 + Task 3: upload records to blob storage")
+    blob_service_client = BlobServiceClient.from_connection_string(blob_conn_str)
+    container_client = blob_service_client.get_container_client("raw")
+    blob_name = f"raw/{source}/{date.today().isoformat()}.json"
+    json_data = json.dumps(records)
+    container_client.upload_blob(blob_name, data=json_data, overwrite=True)
+    return blob_name
 
 
 def write_to_postgres(records: list[dict], postgres_url: str) -> int:
@@ -78,7 +113,39 @@ def write_to_postgres(records: list[dict], postgres_url: str) -> int:
 
     See Chapter 4 for the connection-and-cursor pattern this is based on.
     """
-    raise NotImplementedError("Task 2 + Task 3: insert rows into Azure Postgres")
+    # its already added sslmode=require to the connection string, so we can just use it as is
+    with closing(psycopg2.connect(postgres_url)) as conn:
+        # with psycopg2.connect(postgres_url)as conn:
+        with conn.cursor() as cur:
+            cur.execute("CREATE SCHEMA IF NOT EXISTS dev_mareh;")
+            cur.execute("SET search_path TO dev_mareh;")
+            cur.execute("""
+                CREATE TABLE IF NOT EXISTS weather_readings (
+                    station VARCHAR(50),
+                    timestamp TIMESTAMPTZ,
+                    temperature_c FLOAT,
+                    humidity_pct INT,
+                    PRIMARY KEY (station, timestamp)
+                );
+            """)
+            for each_record in records:
+                cur.execute(
+                    """
+                    INSERT INTO weather_readings (station, timestamp, temperature_c, humidity_pct)
+                    VALUES (%s, %s, %s, %s)
+                    ON CONFLICT (station, timestamp) DO UPDATE
+                    SET temperature_c = EXCLUDED.temperature_c,
+                        humidity_pct = EXCLUDED.humidity_pct;
+                """,
+                    (
+                        each_record["station"],
+                        each_record["timestamp"],
+                        each_record["temperature_c"],
+                        each_record["humidity_pct"],
+                    ),
+                )
+        conn.commit()
+    return len(records)
 
 
 def run() -> None: