Mohammed A

AI_ASSIST.md

@@ -1,21 +1,59 @@
 # AI Assistance Log
 
-Document one session where you used an LLM to help with a query or a design decision while completing Tasks 1-4. Replace every TODO.
+## The problem
 
-> ⚠️ Never paste real customer data or PII into an LLM. The NYC taxi dataset used here is public, so sample rows are safe to share.
+I was working on Task 4, question 1b, where I needed to count the number of trips per pickup borough. Borough names are stored in `vw_dim_zones`, while trip records are stored in `vw_fact_trips`, so I needed to join the fact view to the dimension view.
 
-## The problem
+My first version used an inner join:
 
-TODO: What were you trying to solve? Paste the relevant SQL or schema fragment.
+```sql
+SELECT
+    z.borough,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+INNER JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY z.borough;
+```
 
 ## The prompt
 
-TODO: What did you ask the AI? Include the context you provided.
+I am completing an analytics engineering SQL assignment using NYC taxi data in PostgreSQL. I created a fact view called `vw_fact_trips` and a dimension view called `vw_dim_zones`. For Task 4, question 1b, I need to count rows per pickup borough.
+My first idea was to use this query:
+
+```sql
+SELECT
+    z.borough,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+INNER JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY z.borough;
+```
+
+However, my validation queries showed that some trips have `NULL` `pickup_location_id`, and `pickup_location_id = 999` exists in the trips table but does not exist in the zones table. Is my query still a good choice for the borough count, or would it hide some data-quality issues? What would be a better way to write this query for analytics reporting, and why?
 
 ## The response
 
-TODO: What did it suggest? Did it work first try?
+The AI explained that my original `INNER JOIN` query would run, but it would hide trips where the pickup location does not match a row in `vw_dim_zones`. This includes trips with `NULL` `pickup_location_id` and trips with invalid pickup IDs such as `999`.
+
+The AI suggested changing the query to use a `LEFT JOIN` so that all rows from `vw_fact_trips` are kept in the result. It also introduced `COALESCE(z.borough, 'Unknown') AS borough`, which replaces a missing borough value with `Unknown` instead of leaving it as `NULL`.
+
+The suggested query was:
+
+```sql
+SELECT
+    COALESCE(z.borough, 'Unknown') AS borough,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+LEFT JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY COALESCE(z.borough, 'Unknown')
+ORDER BY trip_count DESC;
+```
 
 ## Reflection
 
-TODO: Did you understand *why* the suggestion worked, or did you accept it blindly?
+I understood why the suggestion worked after comparing it with my validation results. Before this, I was thinking only about joining the fact view to the dimension view, so an `INNER JOIN` seemed normal. However, the validation step showed that some pickup locations were missing or invalid.
+
+Using an `INNER JOIN` would silently remove those trips from the borough count. Using a `LEFT JOIN` keeps all fact rows, and `COALESCE(z.borough, 'Unknown')` makes unmatched rows visible and readable in the result. This is better for analytics reporting because it shows the data-quality issue instead of hiding it. I did not accept the suggestion blindly; I checked it against the data issues found in Task 1 and understood why it was a better choice for this query.

data_dictionary.md

@@ -1,17 +1,15 @@
 # Data Dictionary
 
-Document both views. State the grain in one sentence, identify the keys, and list the measures (the columns you can aggregate). Replace every TODO.
-
 ## vw_fact_trips
 
-- **Grain:** TODO (one sentence, e.g. "One row per ...")
-- **Primary key:** TODO
-- **Foreign keys:** TODO
-- **Measures:** TODO (columns you would SUM or AVG)
+- **Grain:** One row represents one NYC taxi trip after basic cleaning, where trips with negative `fare_amount` have been excluded.
+- **Primary key:** No declared primary key is included in this view. I checked the database constraints for `nyc_taxi.raw_trips`, and no primary key or unique constraint was returned. The view is at trip-event grain, but the selected columns do not provide a guaranteed unique trip identifier. A possible natural key could be a combination of attributes such as `vendor_id`, `pickup_datetime`, `dropoff_datetime`, `pickup_location_id`, and `dropoff_location_id`, but this should not be treated as guaranteed unique without further validation.
+- **Foreign keys:** `pickup_location_id` and `dropoff_location_id` reference `vw_dim_zones.location_id`.
+- **Measures:** `passenger_count`, `trip_distance`, `fare_amount`, `extra`, `mta_tax`, `tip_amount`, `tolls_amount`, `improvement_surcharge`, and `total_amount`.
 
 ## vw_dim_zones
 
-- **Grain:** TODO
-- **Primary key:** TODO
-- **Foreign keys:** TODO (or "none")
-- **Measures:** TODO (or "none, descriptive attributes only")
+- **Grain:** One row represents one NYC taxi zone/location.
+- **Primary key:** `location_id`.
+- **Foreign keys:** none.
+- **Measures:** none, descriptive attributes only. The descriptive columns are `borough`, `zone`, and `service_zone`.

schema_setup.sql

@@ -2,25 +2,46 @@
 -- CREATE OR REPLACE VIEW lets you re-run this script while you iterate.
 
 -- Dimension: one row per location_id. Treat location_id as the primary key.
--- TODO: complete the SELECT (location_id, zone, borough).
 CREATE OR REPLACE VIEW vw_dim_zones AS
 SELECT
-    -- TODO
+    location_id,
+    borough,
+    zone,
+    service_zone
 FROM nyc_taxi.raw_zones;
 
 -- Fact: one row per taxi trip.
---   - Exclude rows where fare_amount is less than 0.
---   - Cast pickup_datetime to TIMESTAMP.
---   - Keep the location IDs so the view can join to vw_dim_zones.
--- TODO: complete the SELECT and the WHERE.
+-- Negative fare_amount rows are excluded.
+-- pickup_datetime is explicitly cast as TIMESTAMP.
 CREATE OR REPLACE VIEW vw_fact_trips AS
 SELECT
-    -- TODO
+    vendor_id,
+    pickup_datetime::TIMESTAMP AS pickup_datetime,
+    dropoff_datetime::TIMESTAMP AS dropoff_datetime,
+    passenger_count,
+    trip_distance,
+    pickup_location_id,
+    dropoff_location_id,
+    fare_amount,
+    extra,
+    mta_tax,
+    tip_amount,
+    tolls_amount,
+    improvement_surcharge,
+    total_amount,
+    payment_type,
+    trip_type
 FROM nyc_taxi.raw_trips
--- TODO: WHERE fare_amount >= 0
-;
+WHERE fare_amount >= 0;
 
--- Join-readiness test (run after creating the views; it must run without error
--- and return a count close to the vw_fact_trips row count):
--- SELECT COUNT(*) FROM vw_fact_trips f
--- JOIN vw_dim_zones d ON f.pickup_location_id = d.location_id;
+
+-- Verification: fact view row count after removing negative fares.
+SELECT COUNT(*) AS fact_trip_count
+FROM vw_fact_trips;
+
+
+-- Verification: join-readiness test.
+SELECT COUNT(*) AS joined_trip_count
+FROM vw_fact_trips AS f
+INNER JOIN vw_dim_zones AS d
+    ON f.pickup_location_id = d.location_id;

validation_queries.sql

@@ -1,20 +1,68 @@
 -- Task 1: Data Quality Audit
--- Run every query against nyc_taxi.raw_trips / nyc_taxi.raw_zones in YOUR OWN schema (not public).
+-- Run every query against nyc_taxi.raw_trips / nyc_taxi.raw_zones in YOUR OWN schema (not public). -- noqa: LT05
 -- The shared pattern is a query that returns the bad rows (or a count).
 -- Zero rows back means the check passed.
 
--- 1. Duplicate check: are there rows with the same vendor_id, pickup_datetime, dropoff_datetime?
--- TODO: GROUP BY the three columns and keep only groups with HAVING COUNT(*) > 1.
+-- 1. Duplicate check
+SELECT
+    vendor_id,
+    pickup_datetime,
+    dropoff_datetime,
+    COUNT(*) AS duplicate_count
+FROM nyc_taxi.raw_trips
+GROUP BY
+    vendor_id,
+    pickup_datetime,
+    dropoff_datetime
+HAVING COUNT(*) > 1
+ORDER BY duplicate_count DESC;
 
+-- Finding:
+-- Duplicate records exist. Some combinations appear 2 times
 
--- 2. Null integrity: how many rows have a NULL pickup_location_id or dropoff_location_id?
--- TODO: count the NULLs (COUNT(*) FILTER (WHERE ... IS NULL) is handy for several columns at once).
+-- 2. Null integrity
+SELECT
+    COUNT(*) FILTER (
+        WHERE pickup_location_id IS NULL
+    ) AS null_pickup_location_id_count,
+    COUNT(*) FILTER (
+        WHERE dropoff_location_id IS NULL
+    ) AS null_dropoff_location_id_count,
+    COUNT(*) FILTER (
+        WHERE pickup_location_id IS NULL
+        OR dropoff_location_id IS NULL
+    ) AS rows_with_any_null_location_id
+FROM nyc_taxi.raw_trips;
 
+-- Finding:
+-- 5 rows have NULL pickup_location_id.
+-- 0 rows have NULL dropoff_location_id.
+-- In total, 5 rows have at least one missing location ID.
 
--- 3. Range validation: what are the min and max fare_amount? Are there negative values?
--- TODO: SELECT MIN(fare_amount), MAX(fare_amount), and a count of rows where fare_amount < 0.
 
+-- 3. Range validation
+SELECT
+    MIN(fare_amount) AS min_fare_amount,
+    MAX(fare_amount) AS max_fare_amount,
+    COUNT(*) FILTER (WHERE fare_amount < 0) AS negative_fare_count
+FROM nyc_taxi.raw_trips;
 
--- 4. Relationship check: which pickup_location_id values in nyc_taxi.raw_trips do NOT exist in nyc_taxi.raw_zones?
--- TODO: LEFT JOIN nyc_taxi.raw_zones ... WHERE z.location_id IS NULL  (or NOT EXISTS).
--- Do NOT use NOT IN: a single NULL in the subquery hides every orphan.
+-- Finding:
+-- fare_amount ranges from -70 to 1422.6.
+-- There are 182 rows with negative fare_amount.
+
+-- 4. Relationship check
+SELECT
+    t.pickup_location_id,
+    COUNT(*) AS trip_count
+FROM nyc_taxi.raw_trips AS t
+LEFT JOIN nyc_taxi.raw_zones AS z
+    ON t.pickup_location_id = z.location_id
+WHERE
+    t.pickup_location_id IS NOT NULL
+    AND z.location_id IS NULL
+GROUP BY t.pickup_location_id
+ORDER BY trip_count DESC;
+
+-- Finding:
+-- pickup_location_id 999 appears in 5 trips but does not exist in nyc_taxi.raw_zones. -- noqa: LT05

verification_results.sql

@@ -1,22 +1,97 @@
 -- Task 4: Verification Queries.
--- Query your views and label each query with the question it answers.
--- Borough and zone names live in vw_dim_zones, so join on pickup_location_id = location_id.
 
--- 1. Volume: how many total rows in vw_fact_trips? How many rows per borough?
---    What is the most common pickup/dropoff location combination?
--- TODO
--- (Take a screenshot of the per-borough counts and save it as assets/borough_count.png.)
+-- 1a. Volume: total rows in vw_fact_trips
+SELECT COUNT(*) AS total_trips
+FROM vw_fact_trips;
 
 
--- 2. Revenue: which pickup zone (name, not ID) generated the highest total fare_amount?
---    Which pickup zone collected the highest total fare_amount on any single day?
--- TODO
+-- 1b. Volume: rows per pickup borough
+-- Screenshot this result and save it as assets/borough_count.png.
+SELECT
+    COALESCE(z.borough, 'Unknown') AS borough,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+LEFT JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY COALESCE(z.borough, 'Unknown')
+ORDER BY trip_count DESC;
 
 
--- 3. Geospatial: total number of trips and average trip_distance for each borough.
--- TODO
+-- 1c. Volume: most common pickup/dropoff location combination
+SELECT
+    z_pickup.zone AS pickup_zone,
+    z_dropoff.zone AS dropoff_zone,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+INNER JOIN vw_dim_zones AS z_pickup
+    ON t.pickup_location_id = z_pickup.location_id
+INNER JOIN vw_dim_zones AS z_dropoff
+    ON t.dropoff_location_id = z_dropoff.location_id
+GROUP BY
+    z_pickup.zone,
+    z_dropoff.zone
+ORDER BY trip_count DESC
+LIMIT 1;
 
 
--- 4. Time patterns: which day of the week had the highest total tip_amount?
---    What hour of the day has the highest average tip?
--- TODO
+-- 2a. Revenue: pickup zone with highest total fare_amount
+SELECT
+    z.zone AS pickup_zone,
+    SUM(t.fare_amount) AS total_fare
+FROM vw_fact_trips AS t
+INNER JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY z.zone
+ORDER BY total_fare DESC
+LIMIT 1;
+
+
+-- 2b. Revenue: pickup zone with highest total fare_amount on any single day
+SELECT
+    z.zone AS pickup_zone,
+    DATE(t.pickup_datetime) AS trip_date,
+    SUM(t.fare_amount) AS total_fare
+FROM vw_fact_trips AS t
+INNER JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY
+    z.zone,
+    DATE(t.pickup_datetime)
+ORDER BY total_fare DESC
+LIMIT 1;
+
+
+-- 3. Geospatial: total number of trips and average trip_distance for each pickup borough -- noqa: LT05
+SELECT
+    COALESCE(z.borough, 'Unknown') AS borough,
+    COUNT(*) AS total_trips,
+    AVG(t.trip_distance) AS avg_trip_distance
+FROM vw_fact_trips AS t
+LEFT JOIN vw_dim_zones AS z
+    ON t.pickup_location_id = z.location_id
+GROUP BY COALESCE(z.borough, 'Unknown')
+ORDER BY total_trips DESC;
+
+
+-- 4a. Time patterns: day of the week with the highest total tip_amount
+SELECT
+    TO_CHAR(t.pickup_datetime, 'Day') AS day_of_week,
+    EXTRACT(DOW FROM t.pickup_datetime) AS day_num,
+    SUM(t.tip_amount) AS total_tip_amount
+FROM vw_fact_trips AS t
+GROUP BY
+    TO_CHAR(t.pickup_datetime, 'Day'),
+    EXTRACT(DOW FROM t.pickup_datetime)
+ORDER BY total_tip_amount DESC
+LIMIT 1;
+
+
+-- 4b. Time patterns: hour of the day with the highest average tip_amount
+SELECT
+    EXTRACT(HOUR FROM t.pickup_datetime) AS hour_of_day,
+    AVG(t.tip_amount) AS avg_tip_amount,
+    COUNT(*) AS trip_count
+FROM vw_fact_trips AS t
+GROUP BY EXTRACT(HOUR FROM t.pickup_datetime)
+ORDER BY avg_tip_amount DESC
+LIMIT 1;