Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 53 additions & 4 deletions AI_ASSIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,65 @@ Document one session where you used an LLM to help with a query or a design deci

## The problem

TODO: What were you trying to solve? Paste the relevant SQL or schema fragment.
I was confused about the meaning of primary keys and foreign keys in the Week 9 data model. In particular, I did not understand whether `vendor_id` could be used as the primary key in `vw_fact_trips`.

The relevant views were:

```sql
CREATE OR REPLACE VIEW dev_halyna.vw_dim_zones AS
SELECT
location_id,
zone,
borough
FROM nyc_taxi.raw_zones;

CREATE OR REPLACE VIEW dev_halyna.vw_fact_trips AS
SELECT
vendor_id,
pickup_datetime::TIMESTAMP AS pickup_datetime,
dropoff_datetime,
pickup_location_id,
dropoff_location_id,
passenger_count,
trip_distance,
fare_amount,
tip_amount,
total_amount,
payment_type
FROM nyc_taxi.raw_trips
WHERE fare_amount >= 0;
```

## The prompt

TODO: What did you ask the AI? Include the context you provided.
I asked the AI to explain the two views in beginner-friendly language and to clarify whether vendor_id is a primary key.

I also asked what should be written in the data dictionary for:

- the grain of each view
- the primary key
- the foreign keys
- the measures

## The response

TODO: What did it suggest? Did it work first try?
The AI explained that vendor_id is not a primary key because it does not identify one unique trip. The same vendor can appear in many different taxi trips.

The AI explained the difference between the two views:

vw_fact_trips is the fact view. One row represents one cleaned taxi trip. It contains measurable values such as passenger_count, trip_distance, fare_amount, tip_amount, and total_amount.
vw_dim_zones is the dimension view. One row represents one taxi zone. It contains descriptive information such as location_id, zone, and borough.

The AI also explained that:

vw_fact_trips has no explicit primary key because the raw trips table does not provide a single unique trip identifier.
pickup_location_id and dropoff_location_id are foreign keys because they refer to vw_dim_zones.location_id.
vw_dim_zones.location_id can be treated as the primary key because each taxi zone has one unique location ID.

## Reflection

TODO: Did you understand *why* the suggestion worked, or did you accept it blindly?
I understood why vendor_id is not a primary key. A primary key must uniquely identify one row, but many trips can have the same vendor_id. Therefore, vendor_id only describes the vendor, not the individual trip.

I also understood the role of the two views in the star schema. The fact view stores trip events and numeric measures. The dimension view stores descriptive zone information. The fact view can be joined to the dimension view using pickup_location_id or dropoff_location_id.

This helped me understand the data model, not just copy the SQL.
Binary file added assets/borough_count.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 19 additions & 9 deletions data_dictionary.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,27 @@
# Data Dictionary

Document both views. State the grain in one sentence, identify the keys, and list the measures (the columns you can aggregate). Replace every TODO.
This file documents the two analytical views created for the Week 9 NYC Taxi star schema.

## vw_fact_trips

- **Grain:** TODO (one sentence, e.g. "One row per ...")
- **Primary key:** TODO
- **Foreign keys:** TODO
- **Measures:** TODO (columns you would SUM or AVG)
- **Grain:** One row represents one NYC green taxi trip from the raw trips table after removing rows where `fare_amount < 0`.
- **Primary key:** No explicit primary key is available in this view. The raw trips table does not provide a single unique trip identifier. `vendor_id` is not a primary key because the same vendor can appear in many trips.
- **Foreign keys:**
- `pickup_location_id` references `vw_dim_zones.location_id`
- `dropoff_location_id` references `vw_dim_zones.location_id`
- **Measures:**
- `passenger_count`
- `trip_distance`
- `fare_amount`
- `tip_amount`
- `total_amount`

## vw_dim_zones

- **Grain:** TODO
- **Primary key:** TODO
- **Foreign keys:** TODO (or "none")
- **Measures:** TODO (or "none, descriptive attributes only")
- **Grain:** One row represents one NYC taxi zone.
- **Primary key:** `location_id`
- **Foreign keys:** None.
- **Measures:** None. This view contains descriptive attributes only:
- `location_id`
- `zone`
- `borough`
32 changes: 24 additions & 8 deletions schema_setup.sql
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,41 @@
-- CREATE OR REPLACE VIEW lets you re-run this script while you iterate.

-- Dimension: one row per location_id. Treat location_id as the primary key.
-- TODO: complete the SELECT (location_id, zone, borough).
CREATE OR REPLACE VIEW vw_dim_zones AS
CREATE OR REPLACE VIEW dev_halyna.vw_dim_zones AS
SELECT
-- TODO
location_id,
zone,
borough
FROM nyc_taxi.raw_zones;

-- Fact: one row per taxi trip.
-- - Exclude rows where fare_amount is less than 0.
-- - Cast pickup_datetime to TIMESTAMP.
-- - Keep the location IDs so the view can join to vw_dim_zones.
-- TODO: complete the SELECT and the WHERE.
CREATE OR REPLACE VIEW vw_fact_trips AS

CREATE OR REPLACE VIEW dev_halyna.vw_fact_trips AS
SELECT
-- TODO
vendor_id,
pickup_datetime::TIMESTAMP AS pickup_datetime,
dropoff_datetime,
pickup_location_id,
dropoff_location_id,
passenger_count,
trip_distance,
fare_amount,
tip_amount,
total_amount,
payment_type
FROM nyc_taxi.raw_trips
-- TODO: WHERE fare_amount >= 0
;
WHERE fare_amount >= 0;


-- Join-readiness test (run after creating the views; it must run without error
-- and return a count close to the vw_fact_trips row count):
-- SELECT COUNT(*) FROM vw_fact_trips f
-- JOIN vw_dim_zones d ON f.pickup_location_id = d.location_id;

SELECT COUNT(*) AS joined_pickup_trip_count
FROM dev_halyna.vw_fact_trips AS f
JOIN dev_halyna.vw_dim_zones AS d
ON f.pickup_location_id = d.location_id;
41 changes: 35 additions & 6 deletions validation_queries.sql
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,46 @@
-- Zero rows back means the check passed.

-- 1. Duplicate check: are there rows with the same vendor_id, pickup_datetime, dropoff_datetime?
-- TODO: GROUP BY the three columns and keep only groups with HAVING COUNT(*) > 1.

SELECT
vendor_id,
pickup_datetime,
dropoff_datetime,
COUNT(*) AS duplicate_count
FROM nyc_taxi.raw_trips
GROUP BY
vendor_id,
pickup_datetime,
dropoff_datetime
HAVING COUNT(*) > 1
ORDER BY duplicate_count DESC;

-- 2. Null integrity: how many rows have a NULL pickup_location_id or dropoff_location_id?
-- TODO: count the NULLs (COUNT(*) FILTER (WHERE ... IS NULL) is handy for several columns at once).

SELECT
COUNT(*) FILTER (WHERE pickup_location_id IS NULL) AS null_pickup_location_id,
COUNT(*) FILTER (WHERE dropoff_location_id IS NULL) AS null_dropoff_location_id,
COUNT(*) FILTER (
WHERE pickup_location_id IS NULL
OR dropoff_location_id IS NULL
) AS rows_with_any_null_location_id
FROM nyc_taxi.raw_trips;

-- 3. Range validation: what are the min and max fare_amount? Are there negative values?
-- TODO: SELECT MIN(fare_amount), MAX(fare_amount), and a count of rows where fare_amount < 0.
SELECT
MIN(fare_amount) AS min_fare_amount,
MAX(fare_amount) AS max_fare_amount,
COUNT(*) FILTER (WHERE fare_amount < 0) AS negative_fare_amount_count
FROM nyc_taxi.raw_trips;


-- 4. Relationship check: which pickup_location_id values in nyc_taxi.raw_trips do NOT exist in nyc_taxi.raw_zones?
-- TODO: LEFT JOIN nyc_taxi.raw_zones ... WHERE z.location_id IS NULL (or NOT EXISTS).
SELECT
t.pickup_location_id,
COUNT(*) AS orphaned_trip_count
FROM nyc_taxi.raw_trips AS t
LEFT JOIN nyc_taxi.raw_zones AS z
ON t.pickup_location_id = z.location_id
WHERE t.pickup_location_id IS NOT NULL

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The join already make sure the left side is never null, so this statement doesn't hurt but is not strictly needed.

AND z.location_id IS NULL
GROUP BY t.pickup_location_id
ORDER BY orphaned_trip_count DESC;
-- Do NOT use NOT IN: a single NULL in the subquery hides every orphan.
81 changes: 77 additions & 4 deletions verification_results.sql
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,92 @@

-- 1. Volume: how many total rows in vw_fact_trips? How many rows per borough?
-- What is the most common pickup/dropoff location combination?
-- TODO
SELECT
COUNT(*) AS total_fact_trip_rows
FROM vw_fact_trips;
-- (Take a screenshot of the per-borough counts and save it as assets/borough_count.png.)

SELECT
d.borough,
COUNT(*) AS trip_count
FROM vw_fact_trips AS f
JOIN vw_dim_zones AS d
ON f.pickup_location_id = d.location_id
GROUP BY d.borough
ORDER BY trip_count DESC;

SELECT
pickup_zone.zone AS pickup_zone,
dropoff_zone.zone AS dropoff_zone,
COUNT(*) AS trip_count
FROM vw_fact_trips AS f
JOIN vw_dim_zones AS pickup_zone
ON f.pickup_location_id = pickup_zone.location_id
JOIN vw_dim_zones AS dropoff_zone
ON f.dropoff_location_id = dropoff_zone.location_id
GROUP BY
pickup_zone.zone,
dropoff_zone.zone
ORDER BY trip_count DESC
LIMIT 1;

-- 2. Revenue: which pickup zone (name, not ID) generated the highest total fare_amount?
-- Which pickup zone collected the highest total fare_amount on any single day?
-- TODO
SELECT
d.zone AS pickup_zone,
d.borough AS pickup_borough,
SUM(f.fare_amount) AS total_fare_amount
FROM vw_fact_trips AS f
JOIN vw_dim_zones AS d
ON f.pickup_location_id = d.location_id
GROUP BY
d.zone,
d.borough
ORDER BY total_fare_amount DESC
LIMIT 1;

SELECT
DATE(f.pickup_datetime) AS pickup_date,
d.zone AS pickup_zone,
d.borough AS pickup_borough,
SUM(f.fare_amount) AS daily_total_fare_amount
FROM vw_fact_trips AS f
JOIN vw_dim_zones AS d
ON f.pickup_location_id = d.location_id
GROUP BY
DATE(f.pickup_datetime),
d.zone,
d.borough
ORDER BY daily_total_fare_amount DESC
LIMIT 1;


-- 3. Geospatial: total number of trips and average trip_distance for each borough.
-- TODO
SELECT
d.borough,
COUNT(*) AS trip_count,
AVG(f.trip_distance) AS avg_trip_distance
FROM vw_fact_trips AS f
JOIN vw_dim_zones AS d
ON f.pickup_location_id = d.location_id
GROUP BY d.borough
ORDER BY trip_count DESC;


-- 4. Time patterns: which day of the week had the highest total tip_amount?
-- What hour of the day has the highest average tip?
-- TODO
SELECT
TO_CHAR(pickup_datetime, 'Day') AS day_of_week,
SUM(tip_amount) AS total_tip_amount
FROM vw_fact_trips
GROUP BY TO_CHAR(pickup_datetime, 'Day')
ORDER BY total_tip_amount DESC
LIMIT 1;

SELECT
DATE_PART('hour', pickup_datetime) AS pickup_hour,
AVG(tip_amount) AS avg_tip_amount
FROM vw_fact_trips
GROUP BY DATE_PART('hour', pickup_datetime)
ORDER BY avg_tip_amount DESC
LIMIT 1;