diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 07efd00..dae1a80 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -6,16 +6,65 @@ Document one session where you used an LLM to help with a query or a design deci ## The problem -TODO: What were you trying to solve? Paste the relevant SQL or schema fragment. +I was confused about the meaning of primary keys and foreign keys in the Week 9 data model. In particular, I did not understand whether `vendor_id` could be used as the primary key in `vw_fact_trips`. + +The relevant views were: + +```sql +CREATE OR REPLACE VIEW dev_halyna.vw_dim_zones AS +SELECT + location_id, + zone, + borough +FROM nyc_taxi.raw_zones; + +CREATE OR REPLACE VIEW dev_halyna.vw_fact_trips AS +SELECT + vendor_id, + pickup_datetime::TIMESTAMP AS pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + passenger_count, + trip_distance, + fare_amount, + tip_amount, + total_amount, + payment_type +FROM nyc_taxi.raw_trips +WHERE fare_amount >= 0; +``` ## The prompt -TODO: What did you ask the AI? Include the context you provided. +I asked the AI to explain the two views in beginner-friendly language and to clarify whether vendor_id is a primary key. + +I also asked what should be written in the data dictionary for: + +- the grain of each view +- the primary key +- the foreign keys +- the measures ## The response -TODO: What did it suggest? Did it work first try? +The AI explained that vendor_id is not a primary key because it does not identify one unique trip. The same vendor can appear in many different taxi trips. + +The AI explained the difference between the two views: + +vw_fact_trips is the fact view. One row represents one cleaned taxi trip. It contains measurable values such as passenger_count, trip_distance, fare_amount, tip_amount, and total_amount. +vw_dim_zones is the dimension view. One row represents one taxi zone. It contains descriptive information such as location_id, zone, and borough. + +The AI also explained that: + +vw_fact_trips has no explicit primary key because the raw trips table does not provide a single unique trip identifier. +pickup_location_id and dropoff_location_id are foreign keys because they refer to vw_dim_zones.location_id. +vw_dim_zones.location_id can be treated as the primary key because each taxi zone has one unique location ID. ## Reflection -TODO: Did you understand *why* the suggestion worked, or did you accept it blindly? +I understood why vendor_id is not a primary key. A primary key must uniquely identify one row, but many trips can have the same vendor_id. Therefore, vendor_id only describes the vendor, not the individual trip. + +I also understood the role of the two views in the star schema. The fact view stores trip events and numeric measures. The dimension view stores descriptive zone information. The fact view can be joined to the dimension view using pickup_location_id or dropoff_location_id. + +This helped me understand the data model, not just copy the SQL. diff --git a/assets/borough_count.png b/assets/borough_count.png new file mode 100644 index 0000000..8c2a2ef Binary files /dev/null and b/assets/borough_count.png differ diff --git a/data_dictionary.md b/data_dictionary.md index 5e44612..aba1618 100644 --- a/data_dictionary.md +++ b/data_dictionary.md @@ -1,17 +1,27 @@ # Data Dictionary -Document both views. State the grain in one sentence, identify the keys, and list the measures (the columns you can aggregate). Replace every TODO. +This file documents the two analytical views created for the Week 9 NYC Taxi star schema. ## vw_fact_trips -- **Grain:** TODO (one sentence, e.g. "One row per ...") -- **Primary key:** TODO -- **Foreign keys:** TODO -- **Measures:** TODO (columns you would SUM or AVG) +- **Grain:** One row represents one NYC green taxi trip from the raw trips table after removing rows where `fare_amount < 0`. +- **Primary key:** No explicit primary key is available in this view. The raw trips table does not provide a single unique trip identifier. `vendor_id` is not a primary key because the same vendor can appear in many trips. +- **Foreign keys:** + - `pickup_location_id` references `vw_dim_zones.location_id` + - `dropoff_location_id` references `vw_dim_zones.location_id` +- **Measures:** + - `passenger_count` + - `trip_distance` + - `fare_amount` + - `tip_amount` + - `total_amount` ## vw_dim_zones -- **Grain:** TODO -- **Primary key:** TODO -- **Foreign keys:** TODO (or "none") -- **Measures:** TODO (or "none, descriptive attributes only") +- **Grain:** One row represents one NYC taxi zone. +- **Primary key:** `location_id` +- **Foreign keys:** None. +- **Measures:** None. This view contains descriptive attributes only: + - `location_id` + - `zone` + - `borough` diff --git a/schema_setup.sql b/schema_setup.sql index a7ae1ad..4f158b2 100644 --- a/schema_setup.sql +++ b/schema_setup.sql @@ -2,25 +2,41 @@ -- CREATE OR REPLACE VIEW lets you re-run this script while you iterate. -- Dimension: one row per location_id. Treat location_id as the primary key. --- TODO: complete the SELECT (location_id, zone, borough). -CREATE OR REPLACE VIEW vw_dim_zones AS +CREATE OR REPLACE VIEW dev_halyna.vw_dim_zones AS SELECT - -- TODO + location_id, + zone, + borough FROM nyc_taxi.raw_zones; -- Fact: one row per taxi trip. -- - Exclude rows where fare_amount is less than 0. -- - Cast pickup_datetime to TIMESTAMP. -- - Keep the location IDs so the view can join to vw_dim_zones. --- TODO: complete the SELECT and the WHERE. -CREATE OR REPLACE VIEW vw_fact_trips AS + +CREATE OR REPLACE VIEW dev_halyna.vw_fact_trips AS SELECT - -- TODO + vendor_id, + pickup_datetime::TIMESTAMP AS pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + passenger_count, + trip_distance, + fare_amount, + tip_amount, + total_amount, + payment_type FROM nyc_taxi.raw_trips --- TODO: WHERE fare_amount >= 0 -; +WHERE fare_amount >= 0; + -- Join-readiness test (run after creating the views; it must run without error -- and return a count close to the vw_fact_trips row count): -- SELECT COUNT(*) FROM vw_fact_trips f -- JOIN vw_dim_zones d ON f.pickup_location_id = d.location_id; + +SELECT COUNT(*) AS joined_pickup_trip_count +FROM dev_halyna.vw_fact_trips AS f +JOIN dev_halyna.vw_dim_zones AS d + ON f.pickup_location_id = d.location_id; diff --git a/validation_queries.sql b/validation_queries.sql index 301b194..4d15eaf 100644 --- a/validation_queries.sql +++ b/validation_queries.sql @@ -4,17 +4,46 @@ -- Zero rows back means the check passed. -- 1. Duplicate check: are there rows with the same vendor_id, pickup_datetime, dropoff_datetime? --- TODO: GROUP BY the three columns and keep only groups with HAVING COUNT(*) > 1. - +SELECT + vendor_id, + pickup_datetime, + dropoff_datetime, + COUNT(*) AS duplicate_count +FROM nyc_taxi.raw_trips +GROUP BY + vendor_id, + pickup_datetime, + dropoff_datetime +HAVING COUNT(*) > 1 +ORDER BY duplicate_count DESC; -- 2. Null integrity: how many rows have a NULL pickup_location_id or dropoff_location_id? --- TODO: count the NULLs (COUNT(*) FILTER (WHERE ... IS NULL) is handy for several columns at once). - +SELECT + COUNT(*) FILTER (WHERE pickup_location_id IS NULL) AS null_pickup_location_id, + COUNT(*) FILTER (WHERE dropoff_location_id IS NULL) AS null_dropoff_location_id, + COUNT(*) FILTER ( + WHERE pickup_location_id IS NULL + OR dropoff_location_id IS NULL + ) AS rows_with_any_null_location_id +FROM nyc_taxi.raw_trips; -- 3. Range validation: what are the min and max fare_amount? Are there negative values? --- TODO: SELECT MIN(fare_amount), MAX(fare_amount), and a count of rows where fare_amount < 0. +SELECT + MIN(fare_amount) AS min_fare_amount, + MAX(fare_amount) AS max_fare_amount, + COUNT(*) FILTER (WHERE fare_amount < 0) AS negative_fare_amount_count +FROM nyc_taxi.raw_trips; -- 4. Relationship check: which pickup_location_id values in nyc_taxi.raw_trips do NOT exist in nyc_taxi.raw_zones? --- TODO: LEFT JOIN nyc_taxi.raw_zones ... WHERE z.location_id IS NULL (or NOT EXISTS). +SELECT + t.pickup_location_id, + COUNT(*) AS orphaned_trip_count +FROM nyc_taxi.raw_trips AS t +LEFT JOIN nyc_taxi.raw_zones AS z + ON t.pickup_location_id = z.location_id +WHERE t.pickup_location_id IS NOT NULL + AND z.location_id IS NULL +GROUP BY t.pickup_location_id +ORDER BY orphaned_trip_count DESC; -- Do NOT use NOT IN: a single NULL in the subquery hides every orphan. diff --git a/verification_results.sql b/verification_results.sql index 68a8d26..fff5688 100644 --- a/verification_results.sql +++ b/verification_results.sql @@ -4,19 +4,92 @@ -- 1. Volume: how many total rows in vw_fact_trips? How many rows per borough? -- What is the most common pickup/dropoff location combination? --- TODO +SELECT + COUNT(*) AS total_fact_trip_rows +FROM vw_fact_trips; -- (Take a screenshot of the per-borough counts and save it as assets/borough_count.png.) +SELECT + d.borough, + COUNT(*) AS trip_count +FROM vw_fact_trips AS f +JOIN vw_dim_zones AS d + ON f.pickup_location_id = d.location_id +GROUP BY d.borough +ORDER BY trip_count DESC; + +SELECT + pickup_zone.zone AS pickup_zone, + dropoff_zone.zone AS dropoff_zone, + COUNT(*) AS trip_count +FROM vw_fact_trips AS f +JOIN vw_dim_zones AS pickup_zone + ON f.pickup_location_id = pickup_zone.location_id +JOIN vw_dim_zones AS dropoff_zone + ON f.dropoff_location_id = dropoff_zone.location_id +GROUP BY + pickup_zone.zone, + dropoff_zone.zone +ORDER BY trip_count DESC +LIMIT 1; -- 2. Revenue: which pickup zone (name, not ID) generated the highest total fare_amount? -- Which pickup zone collected the highest total fare_amount on any single day? --- TODO +SELECT + d.zone AS pickup_zone, + d.borough AS pickup_borough, + SUM(f.fare_amount) AS total_fare_amount +FROM vw_fact_trips AS f +JOIN vw_dim_zones AS d + ON f.pickup_location_id = d.location_id +GROUP BY + d.zone, + d.borough +ORDER BY total_fare_amount DESC +LIMIT 1; + +SELECT + DATE(f.pickup_datetime) AS pickup_date, + d.zone AS pickup_zone, + d.borough AS pickup_borough, + SUM(f.fare_amount) AS daily_total_fare_amount +FROM vw_fact_trips AS f +JOIN vw_dim_zones AS d + ON f.pickup_location_id = d.location_id +GROUP BY + DATE(f.pickup_datetime), + d.zone, + d.borough +ORDER BY daily_total_fare_amount DESC +LIMIT 1; -- 3. Geospatial: total number of trips and average trip_distance for each borough. --- TODO +SELECT + d.borough, + COUNT(*) AS trip_count, + AVG(f.trip_distance) AS avg_trip_distance +FROM vw_fact_trips AS f +JOIN vw_dim_zones AS d + ON f.pickup_location_id = d.location_id +GROUP BY d.borough +ORDER BY trip_count DESC; -- 4. Time patterns: which day of the week had the highest total tip_amount? -- What hour of the day has the highest average tip? --- TODO +SELECT + TO_CHAR(pickup_datetime, 'Day') AS day_of_week, + SUM(tip_amount) AS total_tip_amount +FROM vw_fact_trips +GROUP BY TO_CHAR(pickup_datetime, 'Day') +ORDER BY total_tip_amount DESC +LIMIT 1; + +SELECT + DATE_PART('hour', pickup_datetime) AS pickup_hour, + AVG(tip_amount) AS avg_tip_amount +FROM vw_fact_trips +GROUP BY DATE_PART('hour', pickup_datetime) +ORDER BY avg_tip_amount DESC +LIMIT 1; \ No newline at end of file