diff --git a/CHANGELOG.md b/CHANGELOG.md index e45d82f63..f02b605f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,11 +14,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `viz smart` now auto-wires the new chart types: a **3D scatter** of the strongest-correlation triple when there are 3+ numeric columns; a **2D density contour** instead of the correlated-pair scatter for large datasets (where a scatter overplots); and an offline **ScatterGeo projection** world-overview instead of mapbox tiles when the coordinates span a continental/global extent ([#302](https://github.com/dathere/qsv/issues/302)). - `viz smart` box plots now overlay sample points via a size-based heuristic — all points for small data, Tukey outliers for medium, none for large (a fast cache-only quartile box) — overridable with `--box-points` (now accepted by `smart`, not just `box`) ([#302](https://github.com/dathere/qsv/issues/302)). - `viz smart` frequency bar charts now show a `(NULL)` bar for empty cells and an `Other (N)` aggregate bar for the categories beyond `--limit` (N = the count of distinct categories rolled up), matching `qsv frequency`'s default output. Both aggregate bars are drawn in a muted grey so they read as summaries rather than real categories. New `--no-nulls` and `--no-other` flags suppress them ([#302](https://github.com/dathere/qsv/issues/302)). +- `viz choropleth` & `viz smart` can now build a choropleth from a user-supplied GeoJSON by **point-in-polygon binning**: each row's `--lat`/`--lon` is tested directly against the GeoJSON polygons (even-odd ray casting, handling holes & MultiPolygon) and the matched feature id becomes the location — exact, works for any country/admin level, and needs no geocoding or GeoNames lookup. Points outside every region snap to the nearest feature by default (`--no-snap` drops them instead); either way a coverage note reports how many points missed every polygon. Wired into the `viz smart` dashboard as the "Regions" panel when a `--geojson` is supplied. Zero new dependencies ([#302](https://github.com/dathere/qsv/issues/302)). +- `viz choropleth` & `viz smart` choropleths now have **richer hover tooltips**. Instead of a bare feature id and value, each region shows a human-readable name + id (e.g. `Kagoshima (JP46)`), the value labeled with its measure (`count: 65`), the share of total (`15.6% of total`, for count/sum aggregations only), and the rank (`rank 1 of 47`). Region names are read from the GeoJSON via the new `--feature-name-key` flag, or auto-detected from common name properties (`properties.name`, etc.) when omitted. Applies to all paths (point-in-polygon, literal `--locations`, geocoded) and both the geo and MapLibre (`--map`) basemaps ([#302](https://github.com/dathere/qsv/issues/302)). ### Changed - `viz smart`: the leading **overview panels** (map/geo, correlation heatmap and its scatter/contour/3D drill-downs, and the time-series trend) now each span the **full dashboard width** on their own row, instead of being squeezed into a half-width grid cell. The per-column box/bar/histogram panels still flow in the `--grid-cols`-wide grid below. Applies to all render paths (typed subplot grid, raw-JSON static export, and the inline-div HTML grid) ([#302](https://github.com/dathere/qsv/issues/302)). - `viz smart`: static image export (PNG/SVG/PDF/…) now **renders the geographic panel** instead of dropping it. Since the Mapbox tile basemap can't be statically exported (it needs network tiles), the map is drawn as an offline ScatterGeo projection fit to the data's extent — with an `albers-usa` projection auto-selected when the coordinates span the US. The Mapbox tile map and 3D scene panels remain HTML-only ([#302](https://github.com/dathere/qsv/issues/302)). - `viz smart`: dashboards now auto-fit the data for **both** HTML and static image export. `--max-charts` defaults to `0` (auto), drawing every eligible column (up to 64). Up to 8 cartesian panels render as a single typed subplot grid (plotly's typed subplot-axis limit); beyond 8, HTML switches to an inline-div grid of independent plots while static image export (PNG/SVG/PDF/…) assembles the grid as raw Plotly JSON with domain-positioned `xaxis9+`/`yaxis9+` axes — so it's no longer capped at 8 panels. Rendered via the static exporter's JSON path (no new dependency; plotly re-exports `plotly_static`). Subplot gaps now scale with the grid size so tall dashboards (e.g. a 42-panel, 21-row export) lay out correctly instead of collapsing to negative cell heights. Set a positive `--max-charts N` to cap the count ([#302](https://github.com/dathere/qsv/issues/302)). +- `viz` **geo maps are now theme-aware**. The `geo`-subplot charts (`choropleth`, `geo`, and the `viz smart` map/region panels) previously always rendered light-gray land on a (possibly dark) page; under a dark `--theme` (e.g. `plotly_dark`) they now use a dark land fill and a dark subplot background (`geo.bgcolor`, newly exposed on the plotly fork's `LayoutGeo`) so the whole map — sea and the area outside the projection included — matches the page. The `viz smart` light/dark toggle also recolors the geo land/sea (not just the page) when switching modes live ([#302](https://github.com/dathere/qsv/issues/302)). --- diff --git a/Cargo.lock b/Cargo.lock index 1501818b6..1c2508eb7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5818,7 +5818,7 @@ dependencies = [ [[package]] name = "plotly" version = "0.14.1" -source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#f4f9f3a4f184b459a3e7bddb271ff08aa5f6eb68" +source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#c933b51fba05b77893201daa61bf06d2ac50e38f" dependencies = [ "askama", "async-trait", @@ -5841,7 +5841,7 @@ dependencies = [ [[package]] name = "plotly_derive" version = "0.14.1" -source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#f4f9f3a4f184b459a3e7bddb271ff08aa5f6eb68" +source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#c933b51fba05b77893201daa61bf06d2ac50e38f" dependencies = [ "darling 0.23.0", "proc-macro2", @@ -5852,7 +5852,7 @@ dependencies = [ [[package]] name = "plotly_static" version = "0.1.0" -source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#f4f9f3a4f184b459a3e7bddb271ff08aa5f6eb68" +source = "git+https://github.com/dathere/plotly?branch=feat%2Fchoropleth-maps#c933b51fba05b77893201daa61bf06d2ac50e38f" dependencies = [ "anyhow", "base64 0.22.1", @@ -6841,6 +6841,7 @@ dependencies = [ "futures", "futures-util", "gender_guesser", + "geojson", "geosuggest-core", "geosuggest-utils", "geozero", diff --git a/Cargo.toml b/Cargo.toml index 9a2b2a110..09e296efd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -209,6 +209,7 @@ flexi_logger = { version = "0.31", features = [ futures = "0.3" futures-util = "0.3" gender_guesser = { version = "0.2", optional = true } +geojson = { version = "0.24", default-features = false, optional = true } geosuggest-core = { version = "0.8", features = ["geoip2"], optional = true } geosuggest-utils = { version = "0.8", optional = true } geozero = { version = "0.15", features = [ @@ -632,7 +633,13 @@ to = ["csvs_convert"] # (`viz smart`) from CSV data via the plotly crate. Base feature = self-contained # interactive HTML output (plotly_embed_js); NO polars dependency. `viz smart` # reuses qsv's in-process stats + frequency caches. See `qsv viz --help` and #302. -viz = ["dep:plotly", "dep:opener", "plotly/plotly_embed_js", "base64-simd"] +viz = [ + "dep:plotly", + "dep:opener", + "dep:geojson", + "plotly/plotly_embed_js", + "base64-simd", +] # viz_static: adds static PNG/SVG/PDF/JPEG/WebP export to `viz` via plotly_static, # which drives a headless Chromium/Firefox (webdriver auto-downloaded). Requires a # browser at runtime; keep out of big-endian / headless-only publish targets. diff --git a/docs/help/viz.md b/docs/help/viz.md index 1612deb8e..3a5331275 100644 --- a/docs/help/viz.md +++ b/docs/help/viz.md @@ -259,6 +259,12 @@ qsv viz choropleth counties.csv --locations fips --value pop --map --geojson cou qsv viz choropleth stops.csv --geocode --lat lat --lon lon -o by_country.html ``` +> Point-in-polygon: bin lat/lon points into custom GeoJSON regions by count (no geocode) + +```console +qsv viz choropleth quakes.csv --lat lat --lon lon --geojson prefectures.geojson --feature-id-key properties.id -o by_pref.html +``` + For more examples, see [tests](https://github.com/dathere/qsv/blob/master/tests/test_viz.rs). See also @@ -340,15 +346,17 @@ qsv viz --help ## Choropleth Options [↩](#nav) -|       Option       | Type | Description | Default | +|        Option        | Type | Description | Default | |--------|------|-------------|--------| |  `‑‑locations`  | string | Column holding the region key for each row (an ISO-3 country code, a 2-letter US state code, a country name, or a GeoJSON feature id, per --location-mode). With --geocode, this instead names a place-name column to forward-geocode into region codes. | | |  `‑‑location‑mode`  | string | How --locations values are matched to regions. One of: iso3 (the default, ISO-3166-1 alpha-3 country codes), usa-states (2-letter US state codes), country-names (full country names), geojson-id (match a --geojson feature id). | `iso3` | |  `‑‑color‑scale`  | string | Colorscale for the region fill. One of: viridis (the default), cividis, greys, greens, blues, reds, ylgnbu, ylorrd, bluered, rdbu, portland, electric, jet, hot, blackbody, earth, picnic, rainbow. | `viridis` | |  `‑‑map`  | flag | Render on a token-free MapLibre tile basemap (a ChoroplethMap) instead of the default projection basemap. Requires --geojson and --feature-id-key. Reuses --style for the basemap. | | -|  `‑‑geojson`  | string | Custom region polygons as a local file path or an http(s) URL to a GeoJSON FeatureCollection. Required for --map, and for the geojson-id location mode. | | -|  `‑‑feature‑id‑key`  | string | Property path in each GeoJSON feature whose value matches an entry in the locations column (e.g. id, properties.fips). | `id` | +|  `‑‑geojson`  | string | Custom region polygons as a local file path or an http(s) URL to a GeoJSON FeatureCollection. Required for --map, and for the geojson-id location mode. Also enables point-in-polygon binning: with --lat/--lon (and without --geocode), each row's point is binned into the region whose polygon contains it (exact, no geocoding) and colored by --value/--agg or counts. | | +|  `‑‑feature‑id‑key`  | string | Property path in each GeoJSON feature whose value matches an entry in the locations column, or that labels each binned region (e.g. id, properties.fips). | `id` | +|  `‑‑feature‑name‑key`  | string | GeoJSON property path whose value is shown as the human-readable region label in choropleth hover (e.g. properties.name). When omitted, common name keys are auto-detected; falls back to the feature id when absent. | | |  `‑‑geocode`  | flag | Derive the region codes by reusing qsv's geocode engine (needs a build with the geocode feature). Either reverse-geocode the lat/lon points, or forward-geocode the locations name column. Only valid with location modes iso3 or usa-states. `viz choropleth` also reuses --value, --agg, --style and the lat/lon options. | | +|  `‑‑no‑snap`  | flag | For point-in-polygon binning (lat/lon points binned into a custom GeoJSON without geocoding): drop points that fall outside every region instead of snapping each to its nearest region (the default). A stderr note reports coverage either way. | | diff --git a/examples/viz/README.md b/examples/viz/README.md index 76314ff74..1afec5547 100644 --- a/examples/viz/README.md +++ b/examples/viz/README.md @@ -56,7 +56,7 @@ as `text/plain`, so a browser won't render it): | `world_cities.csv` | 33 major cities spanning all **seven continents** (incl. two Antarctic stations): `country`, `continent`, `lat`/`lon`, `metro_population_m`, `elevation_m`, `avg_annual_temp_c` | `smart --dictionary infer` (global geo map + per-COUNTRY choropleth via `fitbounds` with `geocode` + a seven-continent bar + box panels) | | `us_cities.csv` | 54 US cities across ~35 states: `lat`/`lon`, `census_region`, `population_m`, `median_age` | `smart` (US point map + per-US-STATE choropleth with `geocode` + box/bar/correlation panels) | | `customer_spend.csv` | 300 customers: a bimodal `monthly_spend`, a right-skewed `account_age_days`, plan/region categoricals, an ID | `smart --smarter` (moarstats-informed: histogram + box hints) | -| `seismic_events.csv` | 417 synthetic Japanese earthquakes: `timestamp`, `lat`/`lon`, a bimodal `depth_km`, a right-skewed `magnitude` correlated with `felt_reports`, a `tsunami` boolean, `region`, an ID | `smart --smarter` (the full geospatial dashboard: map + time-series + correlation + scatter + histogram + boxes + bars) | +| `seismic_events.csv` + `japan_prefectures.geojson` | 417 synthetic Japanese earthquakes (`timestamp`, `lat`/`lon`, a bimodal `depth_km`, a right-skewed `magnitude` correlated with `felt_reports`, a `tsunami` boolean, `region`, an ID), plus a GeoJSON of the 47 prefectures keyed by `properties.id` (ISO 3166-2) | `smart --smarter --geojson japan_prefectures.geojson --feature-id-key properties.id` (the full geospatial dashboard: map + **prefecture choropleth via point-in-polygon binning** + time-series + correlation + scatter + histogram + boxes + bars) | | `delivery_stops.csv` | 90 delivery stops clustered in metro Denver + 4 bad-geocode strays in neighboring states, with `zone`/`vehicle` categoricals, `packages`, and correlated `weight_kg`/`distance_km`/`delivery_minutes` numerics over a `delivered_date` | `smart` (geographic outlier markers + core/full extent boxes, Core/Full zoom buttons & spatial-extent call-out with `geocode`; plus boxes, bars, correlation heatmap, strongest-pair scatter & a time-series — no `--smarter` needed) | ## The smart dashboard @@ -162,11 +162,16 @@ qsv viz smart quakes.csv -o quakes_dashboard.html # them out in the spatial-extent label, e.g. "... — 4 outliers (Wyoming, Kansas & Nebraska)" qsv viz smart delivery_stops.csv -o delivery_dashboard.html -# the full geospatial dashboard: a map, a time-series, a correlation heatmap + drill-down -# scatter, a bimodal-depth histogram, annotated boxes and frequency bars — all auto-chosen. +# the full geospatial dashboard: a map, a prefecture choropleth, a time-series, a correlation +# heatmap + drill-down scatter, a bimodal-depth histogram, annotated boxes and frequency bars — +# all auto-chosen. --geojson + --feature-id-key add a point-in-polygon prefecture choropleth: each +# quake is binned into the GeoJSON region that contains it (no geocoding). This catalog is mostly +# offshore, so each such quake snaps to its nearest prefecture; add --no-snap to drop offshore +# points and color on-land prefectures only. A stderr note reports coverage either way. # Recognized lat/lon columns are charted on the map only, not as redundant distribution panels. # Rendered with the built-in plotly_dark theme (--theme works on every chart type, incl. smart). -qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3 -o seismic_dashboard.html +qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3 \ + --geojson japan_prefectures.geojson --feature-id-key properties.id -o seismic_dashboard.html ``` ### dictionary-guided hierarchy panels (treemap / sunburst) diff --git a/examples/viz/gallery.html b/examples/viz/gallery.html index 2bf81ccf0..ec94709be 100644 --- a/examples/viz/gallery.html +++ b/examples/viz/gallery.html @@ -29,7 +29,7 @@

qsv viz — chart gallery

examples/viz/. Generated with the viz feature; each chart is fully interactive.

-
smart dashboard (--smarter, geospatial)One `qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3` command, 10 auto-chosen panels — nearly every panel type at once on a synthetic catalog of Japanese earthquakes. Things the raw table hides but the dashboard makes obvious: depth_km is bimodal (two populations — shallow interplate quakes ~20 km and the deep Wadati-Benioff slab ~450 km — so --smarter draws a histogram, not a box that would average the peaks away); the points trace Japan's subduction arcs on the map; magnitude vs felt_reports is almost perfectly correlated (r=0.95); magnitude and felt_reports are right-skewed with flagged outliers; and the magnitude-over-time trend spikes during a September aftershock sequence. Coordinate columns are shown on the map only, not re-charted as distributions. Rendered with the built-in plotly_dark theme.
+
smart dashboard (--smarter, geospatial)One `qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3 --geojson japan_prefectures.geojson --feature-id-key properties.id` command, 11 auto-chosen panels — nearly every panel type at once on a synthetic catalog of Japanese earthquakes. Things the raw table hides but the dashboard makes obvious: depth_km is bimodal (two populations — shallow interplate quakes ~20 km and the deep Wadati-Benioff slab ~450 km — so --smarter draws a histogram, not a box that would average the peaks away); the points trace Japan's subduction arcs on the map; and a prefecture choropleth bins each quake into the GeoJSON region that contains it (point-in-polygon, no geocoding) — most of this catalog is offshore Pacific seismicity, so each such quake is snapped to its nearest prefecture (coloring the Tōhoku/Hokkaidō coast; pass --no-snap for an on-land-only view). magnitude vs felt_reports is almost perfectly correlated (r=0.95); magnitude and felt_reports are right-skewed with flagged outliers; and the magnitude-over-time trend spikes during a September aftershock sequence. Coordinate columns are shown on the map only, not re-charted as distributions. Rendered with the built-in plotly_dark theme.
smart dashboard (geographic outliers)`qsv viz smart delivery_stops.csv` — delivery stops clustered in metro Denver with four bad-geocode strays. Points far from the cluster centroid (beyond the Tukey far-out fence of their distances) are flagged as geographic outliers: drawn as distinct amber markers, drawn outside the purple (filled) spatial-extent box, and excluded from the auto-zoom — so the default view stays tight on the core cluster. A second, dashed-magenta no-fill box marks the full extent (core + outliers); use the Core extent / Full extent buttons at the top-left of the map to jump between the tight core view and the full spread (where the strays and the magenta box become visible). In the full qsv viz smart HTML output the spatial-extent label calls them out — Colorado, United States — 4 outliers (Wyoming, Kansas & Nebraska) — while strays within the core's own jurisdiction are folded back in silently instead. Each stop also carries delivery attributes (packages, weight_kg, distance_km, delivery_minutes, a vehicle class and a delivered_date), so beyond the map the auto-profiler fills the dashboard out with box plots, frequency bars, a correlation heatmap, the strongest-pair scatter (packages vs weight_kg) and a delivered-over-time trend — all without --smarter.
smart dashboardAuto-profiled overview: correlation heatmap + box plots + frequency bars, led by a drill-down sunburst. `viz smart` now SKIPS an auto hierarchy when the candidate dimensions are statistically independent (nesting them would just replicate each level's marginal); sales_sample's region/payment_method/product_category are independent, so `--hierarchy-style sunburst` is passed to deliberately showcase the interactive sunburst.
smart dashboard (--smarter)Same auto-profiler with `--smarter`, which runs `qsv moarstats --advanced` itself to enrich the stats cache in one step: the bimodal monthly_spend column renders as a histogram (a box plot would hide its two peaks), and the skewed account_age_days box is annotated with its skew direction and outlier share.
@@ -65,7 +65,7 @@

qsv viz — chart gallery

smart dashboard (--dictionary infer, world choropleth)`qsv viz smart world_cities.csv --dictionary infer` — cities across all seven continents: `viz smart` reverse-geocodes the points and adds a per-country choropleth (cities-per-country, ISO-3) framed to the filled-country geometries via Plotly fitbounds — so the regions are never clipped at the viewport edge — beside the natural-earth point map (crimson markers so coastal/island points read against the ocean), plus a seven-continent breakdown. A describegpt-inferred Data Dictionary supplies the friendly field labels (e.g. Metro Population, Avg Annual Temp). Note: the choropleth is reverse-geocoded from lat/lon, so the two Antarctic stations — which have no sovereign country — snap to the nearest administering territory (McMurdo → NZ's Ross Dependency, Rothera → the Argentine sector); the seven-continent grouping instead comes from the dataset's own continent column. Requires a local LLM; the committed HTML is reused on regen.
@@ -42,7 +42,7 @@

sales_sample.csv — data overview

@@ -58,7 +58,7 @@

sales_sample.csv — data overview

@@ -66,7 +66,7 @@

sales_sample.csv — data overview

@@ -74,7 +74,7 @@

sales_sample.csv — data overview

@@ -82,7 +82,7 @@

sales_sample.csv — data overview

@@ -90,7 +90,7 @@

sales_sample.csv — data overview

@@ -98,7 +98,7 @@

sales_sample.csv — data overview

@@ -106,7 +106,7 @@

sales_sample.csv — data overview

@@ -114,7 +114,7 @@

sales_sample.csv — data overview

@@ -122,7 +122,7 @@

sales_sample.csv — data overview

@@ -130,7 +130,7 @@

sales_sample.csv — data overview

@@ -138,7 +138,7 @@

sales_sample.csv — data overview

@@ -146,8 +146,8 @@

sales_sample.csv — data overview

@@ -42,7 +42,7 @@

customer_spend.csv — data overview

@@ -50,7 +50,7 @@

customer_spend.csv — data overview

@@ -58,15 +58,7 @@

customer_spend.csv — data overview

-
- -
-
-
-
@@ -74,8 +66,8 @@

customer_spend.csv — data overview

@@ -51,7 +51,7 @@

seismic_events.csv — data overview

@@ -59,7 +59,7 @@

seismic_events.csv — data overview

@@ -67,15 +67,15 @@

seismic_events.csv — data overview

-
-
+
+
@@ -83,7 +83,7 @@

seismic_events.csv — data overview

@@ -91,7 +91,7 @@

seismic_events.csv — data overview

@@ -99,7 +99,7 @@

seismic_events.csv — data overview

@@ -107,7 +107,15 @@

seismic_events.csv — data overview

+
+ +
+
+
+
@@ -115,8 +123,8 @@

seismic_events.csv — data overview

@@ -83,8 +83,8 @@

us_cities.csv — data overview

@@ -42,7 +42,7 @@

world_cities.csv — data overview

@@ -106,8 +106,8 @@

world_cities.csv — data overview