diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 07efd00..b80286b 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -6,16 +6,74 @@ Document one session where you used an LLM to help with a query or a design deci ## The problem -TODO: What were you trying to solve? Paste the relevant SQL or schema fragment. +I was creating the `vw_dim_zones` view for the star schema. The starter comment suggested selecting `location_id`, `zone`, and `borough`, but when I tried to run the view with this column order, PostgreSQL returned an error. + +Relevant SQL: + +```sql +CREATE OR REPLACE VIEW vw_dim_zones AS +SELECT + location_id, + zone, + borough +FROM nyc_taxi.raw_zones; +``` + +The error was: + +```text +ERROR: cannot change name of view column "borough" to "zone" +Hint: Use ALTER VIEW ... RENAME COLUMN ... to change name of view column instead. +``` ## The prompt -TODO: What did you ask the AI? Include the context you provided. +I asked the LLM why this query failed and whether the issue was related to the `zone` column being highlighted in DBeaver. + +I provided the context that this was a PostgreSQL view for the NYC Taxi assignment, based on `nyc_taxi.raw_zones`, and that the existing view seemed to work when I used this order: + +```sql +CREATE OR REPLACE VIEW vw_dim_zones AS +SELECT + location_id, + borough, + zone +FROM nyc_taxi.raw_zones; +``` ## The response -TODO: What did it suggest? Did it work first try? +The LLM explained that the problem was not the DBeaver highlighting of `zone`. The issue was that `CREATE OR REPLACE VIEW` in PostgreSQL does not allow changing the names/order of existing view columns in a way that would rename an existing column. + +The view had already been created with this column order: + +```text +location_id, borough, zone +``` + +So trying to replace it with this order: + +```text +location_id, zone, borough +``` + +made PostgreSQL interpret the second column as being renamed from `borough` to `zone`. + +The LLM suggested either dropping and recreating the view, or keeping the existing column order. I kept the existing column order: + +```sql +CREATE OR REPLACE VIEW vw_dim_zones AS +SELECT + location_id, + borough, + zone +FROM nyc_taxi.raw_zones; +``` + +This worked. ## Reflection -TODO: Did you understand *why* the suggestion worked, or did you accept it blindly? +I understood why the suggestion worked. The problem was not with the raw data or with the `zone` column name itself. The problem was how PostgreSQL handles `CREATE OR REPLACE VIEW`: it can replace the query behind a view, but it does not freely allow changing the existing view column names by reordering columns. + +Keeping the same column order solved the issue because PostgreSQL no longer interpreted the replacement as a column rename. In this case, the view still contains all required dimension columns: `location_id`, `borough`, and `zone`, so the star schema remains valid. diff --git a/assets/borough_count.png b/assets/borough_count.png new file mode 100644 index 0000000..2647f47 Binary files /dev/null and b/assets/borough_count.png differ diff --git a/data_dictionary.md b/data_dictionary.md index 5e44612..8f6def5 100644 --- a/data_dictionary.md +++ b/data_dictionary.md @@ -4,14 +4,14 @@ Document both views. State the grain in one sentence, identify the keys, and lis ## vw_fact_trips -- **Grain:** TODO (one sentence, e.g. "One row per ...") -- **Primary key:** TODO -- **Foreign keys:** TODO -- **Measures:** TODO (columns you would SUM or AVG) +- **Grain:** One row per taxi trip from `nyc_taxi.raw_trips`, excluding rows where `fare_amount` is less than 0 +- **Primary key:** No declared primary key. The source table does not provide a unique trip ID +- **Foreign keys:** `pickup_location_id` and `dropoff_location_id` reference `vw_dim_zones.location_id` +- **Measures:** `passenger_count`, `trip_distance`, `fare_amount`, `extra`, `mta_tax`, `tip_amount`, `tolls_amount`, `improvement_surcharge`, `total_amount`, `congestion_surcharge` ## vw_dim_zones -- **Grain:** TODO -- **Primary key:** TODO -- **Foreign keys:** TODO (or "none") -- **Measures:** TODO (or "none, descriptive attributes only") +- **Grain:** One row per taxi zone location +- **Primary key:** `location_id` +- **Foreign keys:** None +- **Measures:** None, descriptive attributes only diff --git a/schema_setup.sql b/schema_setup.sql index a7ae1ad..2140d24 100644 --- a/schema_setup.sql +++ b/schema_setup.sql @@ -5,7 +5,9 @@ -- TODO: complete the SELECT (location_id, zone, borough). CREATE OR REPLACE VIEW vw_dim_zones AS SELECT - -- TODO + location_id, + borough, + zone FROM nyc_taxi.raw_zones; -- Fact: one row per taxi trip. @@ -15,10 +17,27 @@ FROM nyc_taxi.raw_zones; -- TODO: complete the SELECT and the WHERE. CREATE OR REPLACE VIEW vw_fact_trips AS SELECT - -- TODO + vendor_id, + CAST(pickup_datetime AS TIMESTAMP) AS pickup_datetime, + dropoff_datetime, + store_and_fwd_flag, + rate_type_id, + pickup_location_id, + dropoff_location_id, + passenger_count, + trip_distance, + fare_amount, + extra, + mta_tax, + tip_amount, + tolls_amount, + improvement_surcharge, + total_amount, + payment_type, + congestion_surcharge, + trip_type FROM nyc_taxi.raw_trips --- TODO: WHERE fare_amount >= 0 -; +WHERE fare_amount >= 0; -- Join-readiness test (run after creating the views; it must run without error -- and return a count close to the vw_fact_trips row count): diff --git a/validation_queries.sql b/validation_queries.sql index 301b194..d7fb2ce 100644 --- a/validation_queries.sql +++ b/validation_queries.sql @@ -5,16 +5,42 @@ -- 1. Duplicate check: are there rows with the same vendor_id, pickup_datetime, dropoff_datetime? -- TODO: GROUP BY the three columns and keep only groups with HAVING COUNT(*) > 1. - +SELECT + vendor_id, + pickup_datetime, + dropoff_datetime, + COUNT(*) AS duplicate_count +FROM nyc_taxi.raw_trips +GROUP BY + vendor_id, + pickup_datetime, + dropoff_datetime +HAVING COUNT(*) > 1; -- 2. Null integrity: how many rows have a NULL pickup_location_id or dropoff_location_id? -- TODO: count the NULLs (COUNT(*) FILTER (WHERE ... IS NULL) is handy for several columns at once). - +SELECT + COUNT(*) FILTER (WHERE pickup_location_id IS NULL) AS null_pickup_location_id_count, + COUNT(*) FILTER (WHERE dropoff_location_id IS NULL) AS null_dropoff_location_id_count +FROM nyc_taxi.raw_trips; -- 3. Range validation: what are the min and max fare_amount? Are there negative values? -- TODO: SELECT MIN(fare_amount), MAX(fare_amount), and a count of rows where fare_amount < 0. - +SELECT + MIN(fare_amount) AS min_fare_amount, + MAX(fare_amount) AS max_fare_amount, + COUNT(*) FILTER (WHERE fare_amount < 0) AS negative_fares +FROM nyc_taxi.raw_trips; -- 4. Relationship check: which pickup_location_id values in nyc_taxi.raw_trips do NOT exist in nyc_taxi.raw_zones? -- TODO: LEFT JOIN nyc_taxi.raw_zones ... WHERE z.location_id IS NULL (or NOT EXISTS). -- Do NOT use NOT IN: a single NULL in the subquery hides every orphan. +SELECT + t.pickup_location_id, + COUNT(*) AS row_count +FROM nyc_taxi.raw_trips AS t +LEFT JOIN nyc_taxi.raw_zones AS z + ON t.pickup_location_id = z.location_id +WHERE t.pickup_location_id IS NOT NULL + AND z.location_id IS NULL +GROUP BY t.pickup_location_id \ No newline at end of file diff --git a/verification_results.sql b/verification_results.sql index 68a8d26..911ebd1 100644 --- a/verification_results.sql +++ b/verification_results.sql @@ -4,19 +4,101 @@ -- 1. Volume: how many total rows in vw_fact_trips? How many rows per borough? -- What is the most common pickup/dropoff location combination? --- TODO +SELECT + COUNT(*) AS total_rows +FROM vw_fact_trips; + +SELECT + z.borough, + COUNT(*) AS row_count +FROM vw_fact_trips AS t +JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY + z.borough +ORDER BY + row_count DESC; + +SELECT + pickup_zones.zone AS pickup_zone, + dropoff_zones.zone AS dropoff_zone, + COUNT(*) AS row_count +FROM vw_fact_trips AS t +JOIN vw_dim_zones AS pickup_zones + ON t.pickup_location_id = pickup_zones.location_id +JOIN vw_dim_zones AS dropoff_zones + ON t.dropoff_location_id = dropoff_zones.location_id +GROUP BY + pickup_zones.zone, + dropoff_zones.zone +ORDER BY + row_count DESC +LIMIT 1 + -- (Take a screenshot of the per-borough counts and save it as assets/borough_count.png.) -- 2. Revenue: which pickup zone (name, not ID) generated the highest total fare_amount? -- Which pickup zone collected the highest total fare_amount on any single day? --- TODO +SELECT + z.zone AS pickup_zone, + SUM(t.fare_amount) AS total_fare_amount +FROM vw_fact_trips AS t +JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY + z.zone +ORDER BY + total_fare_amount DESC +LIMIT 1; + +SELECT + CAST(t.pickup_datetime AS DATE) AS pickup_date, + z.zone AS pickup_zone, + SUM(t.fare_amount) AS total_fare_amount +FROM vw_fact_trips AS t +JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY + CAST(t.pickup_datetime AS DATE), + z.zone +ORDER BY + total_fare_amount DESC +LIMIT 1; -- 3. Geospatial: total number of trips and average trip_distance for each borough. --- TODO +SELECT + z.borough, + COUNT(*) AS total_trips, + AVG(t.trip_distance) AS avg_trip_distance +FROM vw_fact_trips AS t +JOIN vw_dim_zones AS z + ON t.pickup_location_id = z.location_id +GROUP BY + z.borough +ORDER BY + total_trips DESC; -- 4. Time patterns: which day of the week had the highest total tip_amount? -- What hour of the day has the highest average tip? --- TODO +SELECT + TO_CHAR(pickup_datetime, 'Day') AS day_of_week, + SUM(tip_amount) AS total_tip_amount +FROM vw_fact_trips +GROUP BY + TO_CHAR(pickup_datetime, 'Day') +ORDER BY + total_tip_amount DESC +LIMIT 1; + +SELECT + EXTRACT(HOUR FROM pickup_datetime) AS pickup_hour, + AVG(tip_amount) AS avg_tip_amount +FROM vw_fact_trips +GROUP BY + EXTRACT(HOUR FROM pickup_datetime) +ORDER BY + avg_tip_amount DESC +LIMIT 1;