diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 07efd00..1181568 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -1,21 +1,59 @@ # AI Assistance Log -Document one session where you used an LLM to help with a query or a design decision while completing Tasks 1-4. Replace every TODO. +## The problem -> ⚠️ Never paste real customer data or PII into an LLM. The NYC taxi dataset used here is public, so sample rows are safe to share. +I was working on Task 4, question 1b, where I needed to count the number of trips per pickup borough. Borough names are stored in `vw_dim_zones`, while trip records are stored in `vw_fact_trips`, so I needed to join the fact view to the dimension view. -## The problem +My first version used an inner join: -TODO: What were you trying to solve? Paste the relevant SQL or schema fragment. +```sql +SELECT + z.borough, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +INNER JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY z.borough; +``` ## The prompt -TODO: What did you ask the AI? Include the context you provided. +I am completing an analytics engineering SQL assignment using NYC taxi data in PostgreSQL. I created a fact view called `vw_fact_trips` and a dimension view called `vw_dim_zones`. For Task 4, question 1b, I need to count rows per pickup borough. +My first idea was to use this query: + +```sql +SELECT + z.borough, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +INNER JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY z.borough; +``` + +However, my validation queries showed that some trips have `NULL` `pickup_location_id`, and `pickup_location_id = 999` exists in the trips table but does not exist in the zones table. Is my query still a good choice for the borough count, or would it hide some data-quality issues? What would be a better way to write this query for analytics reporting, and why? ## The response -TODO: What did it suggest? Did it work first try? +The AI explained that my original `INNER JOIN` query would run, but it would hide trips where the pickup location does not match a row in `vw_dim_zones`. This includes trips with `NULL` `pickup_location_id` and trips with invalid pickup IDs such as `999`. + +The AI suggested changing the query to use a `LEFT JOIN` so that all rows from `vw_fact_trips` are kept in the result. It also introduced `COALESCE(z.borough, 'Unknown') AS borough`, which replaces a missing borough value with `Unknown` instead of leaving it as `NULL`. + +The suggested query was: + +```sql +SELECT + COALESCE(z.borough, 'Unknown') AS borough, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +LEFT JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY COALESCE(z.borough, 'Unknown') +ORDER BY trip_count DESC; +``` ## Reflection -TODO: Did you understand *why* the suggestion worked, or did you accept it blindly? +I understood why the suggestion worked after comparing it with my validation results. Before this, I was thinking only about joining the fact view to the dimension view, so an `INNER JOIN` seemed normal. However, the validation step showed that some pickup locations were missing or invalid. + +Using an `INNER JOIN` would silently remove those trips from the borough count. Using a `LEFT JOIN` keeps all fact rows, and `COALESCE(z.borough, 'Unknown')` makes unmatched rows visible and readable in the result. This is better for analytics reporting because it shows the data-quality issue instead of hiding it. I did not accept the suggestion blindly; I checked it against the data issues found in Task 1 and understood why it was a better choice for this query. diff --git a/assets/borough_count.png.png b/assets/borough_count.png.png new file mode 100644 index 0000000..d626984 Binary files /dev/null and b/assets/borough_count.png.png differ diff --git a/data_dictionary.md b/data_dictionary.md index 5e44612..9059f26 100644 --- a/data_dictionary.md +++ b/data_dictionary.md @@ -1,17 +1,15 @@ # Data Dictionary -Document both views. State the grain in one sentence, identify the keys, and list the measures (the columns you can aggregate). Replace every TODO. - ## vw_fact_trips -- **Grain:** TODO (one sentence, e.g. "One row per ...") -- **Primary key:** TODO -- **Foreign keys:** TODO -- **Measures:** TODO (columns you would SUM or AVG) +- **Grain:** One row represents one NYC taxi trip after basic cleaning, where trips with negative `fare_amount` have been excluded. +- **Primary key:** No declared primary key is included in this view. I checked the database constraints for `nyc_taxi.raw_trips`, and no primary key or unique constraint was returned. The view is at trip-event grain, but the selected columns do not provide a guaranteed unique trip identifier. A possible natural key could be a combination of attributes such as `vendor_id`, `pickup_datetime`, `dropoff_datetime`, `pickup_location_id`, and `dropoff_location_id`, but this should not be treated as guaranteed unique without further validation. +- **Foreign keys:** `pickup_location_id` and `dropoff_location_id` reference `vw_dim_zones.location_id`. +- **Measures:** `passenger_count`, `trip_distance`, `fare_amount`, `extra`, `mta_tax`, `tip_amount`, `tolls_amount`, `improvement_surcharge`, and `total_amount`. ## vw_dim_zones -- **Grain:** TODO -- **Primary key:** TODO -- **Foreign keys:** TODO (or "none") -- **Measures:** TODO (or "none, descriptive attributes only") +- **Grain:** One row represents one NYC taxi zone/location. +- **Primary key:** `location_id`. +- **Foreign keys:** none. +- **Measures:** none, descriptive attributes only. The descriptive columns are `borough`, `zone`, and `service_zone`. diff --git a/schema_setup.sql b/schema_setup.sql index a7ae1ad..f86a930 100644 --- a/schema_setup.sql +++ b/schema_setup.sql @@ -2,25 +2,46 @@ -- CREATE OR REPLACE VIEW lets you re-run this script while you iterate. -- Dimension: one row per location_id. Treat location_id as the primary key. --- TODO: complete the SELECT (location_id, zone, borough). CREATE OR REPLACE VIEW vw_dim_zones AS SELECT - -- TODO + location_id, + borough, + zone, + service_zone FROM nyc_taxi.raw_zones; -- Fact: one row per taxi trip. --- - Exclude rows where fare_amount is less than 0. --- - Cast pickup_datetime to TIMESTAMP. --- - Keep the location IDs so the view can join to vw_dim_zones. --- TODO: complete the SELECT and the WHERE. +-- Negative fare_amount rows are excluded. +-- pickup_datetime is explicitly cast as TIMESTAMP. CREATE OR REPLACE VIEW vw_fact_trips AS SELECT - -- TODO + vendor_id, + pickup_datetime::TIMESTAMP AS pickup_datetime, + dropoff_datetime::TIMESTAMP AS dropoff_datetime, + passenger_count, + trip_distance, + pickup_location_id, + dropoff_location_id, + fare_amount, + extra, + mta_tax, + tip_amount, + tolls_amount, + improvement_surcharge, + total_amount, + payment_type, + trip_type FROM nyc_taxi.raw_trips --- TODO: WHERE fare_amount >= 0 -; +WHERE fare_amount >= 0; --- Join-readiness test (run after creating the views; it must run without error --- and return a count close to the vw_fact_trips row count): --- SELECT COUNT(*) FROM vw_fact_trips f --- JOIN vw_dim_zones d ON f.pickup_location_id = d.location_id; + +-- Verification: fact view row count after removing negative fares. +SELECT COUNT(*) AS fact_trip_count +FROM vw_fact_trips; + + +-- Verification: join-readiness test. +SELECT COUNT(*) AS joined_trip_count +FROM vw_fact_trips AS f +INNER JOIN vw_dim_zones AS d + ON f.pickup_location_id = d.location_id; diff --git a/validation_queries.sql b/validation_queries.sql index 301b194..4851f77 100644 --- a/validation_queries.sql +++ b/validation_queries.sql @@ -1,20 +1,68 @@ -- Task 1: Data Quality Audit --- Run every query against nyc_taxi.raw_trips / nyc_taxi.raw_zones in YOUR OWN schema (not public). +-- Run every query against nyc_taxi.raw_trips / nyc_taxi.raw_zones in YOUR OWN schema (not public). -- noqa: LT05 -- The shared pattern is a query that returns the bad rows (or a count). -- Zero rows back means the check passed. --- 1. Duplicate check: are there rows with the same vendor_id, pickup_datetime, dropoff_datetime? --- TODO: GROUP BY the three columns and keep only groups with HAVING COUNT(*) > 1. +-- 1. Duplicate check +SELECT + vendor_id, + pickup_datetime, + dropoff_datetime, + COUNT(*) AS duplicate_count +FROM nyc_taxi.raw_trips +GROUP BY + vendor_id, + pickup_datetime, + dropoff_datetime +HAVING COUNT(*) > 1 +ORDER BY duplicate_count DESC; +-- Finding: +-- Duplicate records exist. Some combinations appear 2 times --- 2. Null integrity: how many rows have a NULL pickup_location_id or dropoff_location_id? --- TODO: count the NULLs (COUNT(*) FILTER (WHERE ... IS NULL) is handy for several columns at once). +-- 2. Null integrity +SELECT + COUNT(*) FILTER ( + WHERE pickup_location_id IS NULL + ) AS null_pickup_location_id_count, + COUNT(*) FILTER ( + WHERE dropoff_location_id IS NULL + ) AS null_dropoff_location_id_count, + COUNT(*) FILTER ( + WHERE pickup_location_id IS NULL + OR dropoff_location_id IS NULL + ) AS rows_with_any_null_location_id +FROM nyc_taxi.raw_trips; +-- Finding: +-- 5 rows have NULL pickup_location_id. +-- 0 rows have NULL dropoff_location_id. +-- In total, 5 rows have at least one missing location ID. --- 3. Range validation: what are the min and max fare_amount? Are there negative values? --- TODO: SELECT MIN(fare_amount), MAX(fare_amount), and a count of rows where fare_amount < 0. +-- 3. Range validation +SELECT + MIN(fare_amount) AS min_fare_amount, + MAX(fare_amount) AS max_fare_amount, + COUNT(*) FILTER (WHERE fare_amount < 0) AS negative_fare_count +FROM nyc_taxi.raw_trips; --- 4. Relationship check: which pickup_location_id values in nyc_taxi.raw_trips do NOT exist in nyc_taxi.raw_zones? --- TODO: LEFT JOIN nyc_taxi.raw_zones ... WHERE z.location_id IS NULL (or NOT EXISTS). --- Do NOT use NOT IN: a single NULL in the subquery hides every orphan. +-- Finding: +-- fare_amount ranges from -70 to 1422.6. +-- There are 182 rows with negative fare_amount. + +-- 4. Relationship check +SELECT + t.pickup_location_id, + COUNT(*) AS trip_count +FROM nyc_taxi.raw_trips AS t +LEFT JOIN nyc_taxi.raw_zones AS z + ON t.pickup_location_id = z.location_id +WHERE + t.pickup_location_id IS NOT NULL + AND z.location_id IS NULL +GROUP BY t.pickup_location_id +ORDER BY trip_count DESC; + +-- Finding: +-- pickup_location_id 999 appears in 5 trips but does not exist in nyc_taxi.raw_zones. -- noqa: LT05 diff --git a/verification_results.sql b/verification_results.sql index 68a8d26..8e88c75 100644 --- a/verification_results.sql +++ b/verification_results.sql @@ -1,22 +1,97 @@ -- Task 4: Verification Queries. --- Query your views and label each query with the question it answers. --- Borough and zone names live in vw_dim_zones, so join on pickup_location_id = location_id. --- 1. Volume: how many total rows in vw_fact_trips? How many rows per borough? --- What is the most common pickup/dropoff location combination? --- TODO --- (Take a screenshot of the per-borough counts and save it as assets/borough_count.png.) +-- 1a. Volume: total rows in vw_fact_trips +SELECT COUNT(*) AS total_trips +FROM vw_fact_trips; --- 2. Revenue: which pickup zone (name, not ID) generated the highest total fare_amount? --- Which pickup zone collected the highest total fare_amount on any single day? --- TODO +-- 1b. Volume: rows per pickup borough +-- Screenshot this result and save it as assets/borough_count.png. +SELECT + COALESCE(z.borough, 'Unknown') AS borough, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +LEFT JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY COALESCE(z.borough, 'Unknown') +ORDER BY trip_count DESC; --- 3. Geospatial: total number of trips and average trip_distance for each borough. --- TODO +-- 1c. Volume: most common pickup/dropoff location combination +SELECT + z_pickup.zone AS pickup_zone, + z_dropoff.zone AS dropoff_zone, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +INNER JOIN vw_dim_zones AS z_pickup + ON t.pickup_location_id = z_pickup.location_id +INNER JOIN vw_dim_zones AS z_dropoff + ON t.dropoff_location_id = z_dropoff.location_id +GROUP BY + z_pickup.zone, + z_dropoff.zone +ORDER BY trip_count DESC +LIMIT 1; --- 4. Time patterns: which day of the week had the highest total tip_amount? --- What hour of the day has the highest average tip? --- TODO +-- 2a. Revenue: pickup zone with highest total fare_amount +SELECT + z.zone AS pickup_zone, + SUM(t.fare_amount) AS total_fare +FROM vw_fact_trips AS t +INNER JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY z.zone +ORDER BY total_fare DESC +LIMIT 1; + + +-- 2b. Revenue: pickup zone with highest total fare_amount on any single day +SELECT + z.zone AS pickup_zone, + DATE(t.pickup_datetime) AS trip_date, + SUM(t.fare_amount) AS total_fare +FROM vw_fact_trips AS t +INNER JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY + z.zone, + DATE(t.pickup_datetime) +ORDER BY total_fare DESC +LIMIT 1; + + +-- 3. Geospatial: total number of trips and average trip_distance for each pickup borough -- noqa: LT05 +SELECT + COALESCE(z.borough, 'Unknown') AS borough, + COUNT(*) AS total_trips, + AVG(t.trip_distance) AS avg_trip_distance +FROM vw_fact_trips AS t +LEFT JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY COALESCE(z.borough, 'Unknown') +ORDER BY total_trips DESC; + + +-- 4a. Time patterns: day of the week with the highest total tip_amount +SELECT + TO_CHAR(t.pickup_datetime, 'Day') AS day_of_week, + EXTRACT(DOW FROM t.pickup_datetime) AS day_num, + SUM(t.tip_amount) AS total_tip_amount +FROM vw_fact_trips AS t +GROUP BY + TO_CHAR(t.pickup_datetime, 'Day'), + EXTRACT(DOW FROM t.pickup_datetime) +ORDER BY total_tip_amount DESC +LIMIT 1; + + +-- 4b. Time patterns: hour of the day with the highest average tip_amount +SELECT + EXTRACT(HOUR FROM t.pickup_datetime) AS hour_of_day, + AVG(t.tip_amount) AS avg_tip_amount, + COUNT(*) AS trip_count +FROM vw_fact_trips AS t +GROUP BY EXTRACT(HOUR FROM t.pickup_datetime) +ORDER BY avg_tip_amount DESC +LIMIT 1;