Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `viz smart` now auto-wires the new chart types: a **3D scatter** of the strongest-correlation triple when there are 3+ numeric columns; a **2D density contour** instead of the correlated-pair scatter for large datasets (where a scatter overplots); and an offline **ScatterGeo projection** world-overview instead of mapbox tiles when the coordinates span a continental/global extent ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz smart` box plots now overlay sample points via a size-based heuristic — all points for small data, Tukey outliers for medium, none for large (a fast cache-only quartile box) — overridable with `--box-points` (now accepted by `smart`, not just `box`) ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz smart` frequency bar charts now show a `(NULL)` bar for empty cells and an `Other (N)` aggregate bar for the categories beyond `--limit` (N = the count of distinct categories rolled up), matching `qsv frequency`'s default output. Both aggregate bars are drawn in a muted grey so they read as summaries rather than real categories. New `--no-nulls` and `--no-other` flags suppress them ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz choropleth` & `viz smart` can now build a choropleth from a user-supplied GeoJSON by **point-in-polygon binning**: each row's `--lat`/`--lon` is tested directly against the GeoJSON polygons (even-odd ray casting, handling holes & MultiPolygon) and the matched feature id becomes the location — exact, works for any country/admin level, and needs no geocoding or GeoNames lookup. Points outside every region snap to the nearest feature by default (`--no-snap` drops them instead); either way a coverage note reports how many points missed every polygon. Wired into the `viz smart` dashboard as the "Regions" panel when a `--geojson` is supplied. Zero new dependencies ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz choropleth` & `viz smart` choropleths now have **richer hover tooltips**. Instead of a bare feature id and value, each region shows a human-readable name + id (e.g. `Kagoshima (JP46)`), the value labeled with its measure (`count: 65`), the share of total (`15.6% of total`, for count/sum aggregations only), and the rank (`rank 1 of 47`). Region names are read from the GeoJSON via the new `--feature-name-key` flag, or auto-detected from common name properties (`properties.name`, etc.) when omitted. Applies to all paths (point-in-polygon, literal `--locations`, geocoded) and both the geo and MapLibre (`--map`) basemaps ([#302](https://github.com/dathere/qsv/issues/302)).

### Changed
- `viz smart`: the leading **overview panels** (map/geo, correlation heatmap and its scatter/contour/3D drill-downs, and the time-series trend) now each span the **full dashboard width** on their own row, instead of being squeezed into a half-width grid cell. The per-column box/bar/histogram panels still flow in the `--grid-cols`-wide grid below. Applies to all render paths (typed subplot grid, raw-JSON static export, and the inline-div HTML grid) ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz smart`: static image export (PNG/SVG/PDF/…) now **renders the geographic panel** instead of dropping it. Since the Mapbox tile basemap can't be statically exported (it needs network tiles), the map is drawn as an offline ScatterGeo projection fit to the data's extent — with an `albers-usa` projection auto-selected when the coordinates span the US. The Mapbox tile map and 3D scene panels remain HTML-only ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz smart`: dashboards now auto-fit the data for **both** HTML and static image export. `--max-charts` defaults to `0` (auto), drawing every eligible column (up to 64). Up to 8 cartesian panels render as a single typed subplot grid (plotly's typed subplot-axis limit); beyond 8, HTML switches to an inline-div grid of independent plots while static image export (PNG/SVG/PDF/…) assembles the grid as raw Plotly JSON with domain-positioned `xaxis9+`/`yaxis9+` axes — so it's no longer capped at 8 panels. Rendered via the static exporter's JSON path (no new dependency; plotly re-exports `plotly_static`). Subplot gaps now scale with the grid size so tall dashboards (e.g. a 42-panel, 21-row export) lay out correctly instead of collapsing to negative cell heights. Set a positive `--max-charts N` to cap the count ([#302](https://github.com/dathere/qsv/issues/302)).
- `viz` **geo maps are now theme-aware**. The `geo`-subplot charts (`choropleth`, `geo`, and the `viz smart` map/region panels) previously always rendered light-gray land on a (possibly dark) page; under a dark `--theme` (e.g. `plotly_dark`) they now use a dark land fill and a dark subplot background (`geo.bgcolor`, newly exposed on the plotly fork's `LayoutGeo`) so the whole map — sea and the area outside the projection included — matches the page. The `viz smart` light/dark toggle also recolors the geo land/sea (not just the page) when switching modes live ([#302](https://github.com/dathere/qsv/issues/302)).

---

Expand Down
7 changes: 4 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@ flexi_logger = { version = "0.31", features = [
futures = "0.3"
futures-util = "0.3"
gender_guesser = { version = "0.2", optional = true }
geojson = { version = "0.24", default-features = false, optional = true }
geosuggest-core = { version = "0.8", features = ["geoip2"], optional = true }
geosuggest-utils = { version = "0.8", optional = true }
geozero = { version = "0.15", features = [
Expand Down Expand Up @@ -632,7 +633,13 @@ to = ["csvs_convert"]
# (`viz smart`) from CSV data via the plotly crate. Base feature = self-contained
# interactive HTML output (plotly_embed_js); NO polars dependency. `viz smart`
# reuses qsv's in-process stats + frequency caches. See `qsv viz --help` and #302.
viz = ["dep:plotly", "dep:opener", "plotly/plotly_embed_js", "base64-simd"]
viz = [
"dep:plotly",
"dep:opener",
"dep:geojson",
"plotly/plotly_embed_js",
"base64-simd",
]
# viz_static: adds static PNG/SVG/PDF/JPEG/WebP export to `viz` via plotly_static,
# which drives a headless Chromium/Firefox (webdriver auto-downloaded). Requires a
# browser at runtime; keep out of big-endian / headless-only publish targets.
Expand Down
14 changes: 11 additions & 3 deletions docs/help/viz.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,12 @@ qsv viz choropleth counties.csv --locations fips --value pop --map --geojson cou
qsv viz choropleth stops.csv --geocode --lat lat --lon lon -o by_country.html
```

> Point-in-polygon: bin lat/lon points into custom GeoJSON regions by count (no geocode)

```console
qsv viz choropleth quakes.csv --lat lat --lon lon --geojson prefectures.geojson --feature-id-key properties.id -o by_pref.html
```

For more examples, see [tests](https://github.com/dathere/qsv/blob/master/tests/test_viz.rs).

See also <https://github.com/dathere/qsv/wiki/Visualization>
Expand Down Expand Up @@ -340,15 +346,17 @@ qsv viz --help

## Choropleth Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑locations`&nbsp; | string | Column holding the region key for each row (an ISO-3 country code, a 2-letter US state code, a country name, or a GeoJSON feature id, per --location-mode). With --geocode, this instead names a place-name column to forward-geocode into region codes. | |
| &nbsp;`‑‑location‑mode`&nbsp; | string | How --locations values are matched to regions. One of: iso3 (the default, ISO-3166-1 alpha-3 country codes), usa-states (2-letter US state codes), country-names (full country names), geojson-id (match a --geojson feature id). | `iso3` |
| &nbsp;`‑‑color‑scale`&nbsp; | string | Colorscale for the region fill. One of: viridis (the default), cividis, greys, greens, blues, reds, ylgnbu, ylorrd, bluered, rdbu, portland, electric, jet, hot, blackbody, earth, picnic, rainbow. | `viridis` |
| &nbsp;`‑‑map`&nbsp; | flag | Render on a token-free MapLibre tile basemap (a ChoroplethMap) instead of the default projection basemap. Requires --geojson and --feature-id-key. Reuses --style for the basemap. | |
| &nbsp;`‑‑geojson`&nbsp; | string | Custom region polygons as a local file path or an http(s) URL to a GeoJSON FeatureCollection. Required for --map, and for the geojson-id location mode. | |
| &nbsp;`‑‑feature‑id‑key`&nbsp; | string | Property path in each GeoJSON feature whose value matches an entry in the locations column (e.g. id, properties.fips). | `id` |
| &nbsp;`‑‑geojson`&nbsp; | string | Custom region polygons as a local file path or an http(s) URL to a GeoJSON FeatureCollection. Required for --map, and for the geojson-id location mode. Also enables point-in-polygon binning: with --lat/--lon (and without --geocode), each row's point is binned into the region whose polygon contains it (exact, no geocoding) and colored by --value/--agg or counts. | |
| &nbsp;`‑‑feature‑id‑key`&nbsp; | string | Property path in each GeoJSON feature whose value matches an entry in the locations column, or that labels each binned region (e.g. id, properties.fips). | `id` |
| &nbsp;`‑‑feature‑name‑key`&nbsp; | string | GeoJSON property path whose value is shown as the human-readable region label in choropleth hover (e.g. properties.name). When omitted, common name keys are auto-detected; falls back to the feature id when absent. | |
| &nbsp;`‑‑geocode`&nbsp; | flag | Derive the region codes by reusing qsv's geocode engine (needs a build with the geocode feature). Either reverse-geocode the lat/lon points, or forward-geocode the locations name column. Only valid with location modes iso3 or usa-states. `viz choropleth` also reuses --value, --agg, --style and the lat/lon options. | |
| &nbsp;`‑‑no‑snap`&nbsp; | flag | For point-in-polygon binning (lat/lon points binned into a custom GeoJSON without geocoding): drop points that fall outside every region instead of snapping each to its nearest region (the default). A stderr note reports coverage either way. | |

<a name="smart-options"></a>

Expand Down
13 changes: 9 additions & 4 deletions examples/viz/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ as `text/plain`, so a browser won't render it):
| `world_cities.csv` | 33 major cities spanning all **seven continents** (incl. two Antarctic stations): `country`, `continent`, `lat`/`lon`, `metro_population_m`, `elevation_m`, `avg_annual_temp_c` | `smart --dictionary infer` (global geo map + per-COUNTRY choropleth via `fitbounds` with `geocode` + a seven-continent bar + box panels) |
| `us_cities.csv` | 54 US cities across ~35 states: `lat`/`lon`, `census_region`, `population_m`, `median_age` | `smart` (US point map + per-US-STATE choropleth with `geocode` + box/bar/correlation panels) |
| `customer_spend.csv` | 300 customers: a bimodal `monthly_spend`, a right-skewed `account_age_days`, plan/region categoricals, an ID | `smart --smarter` (moarstats-informed: histogram + box hints) |
| `seismic_events.csv` | 417 synthetic Japanese earthquakes: `timestamp`, `lat`/`lon`, a bimodal `depth_km`, a right-skewed `magnitude` correlated with `felt_reports`, a `tsunami` boolean, `region`, an ID | `smart --smarter` (the full geospatial dashboard: map + time-series + correlation + scatter + histogram + boxes + bars) |
| `seismic_events.csv` + `japan_prefectures.geojson` | 417 synthetic Japanese earthquakes (`timestamp`, `lat`/`lon`, a bimodal `depth_km`, a right-skewed `magnitude` correlated with `felt_reports`, a `tsunami` boolean, `region`, an ID), plus a GeoJSON of the 47 prefectures keyed by `properties.id` (ISO&nbsp;3166-2) | `smart --smarter --geojson japan_prefectures.geojson --feature-id-key properties.id` (the full geospatial dashboard: map + **prefecture choropleth via point-in-polygon binning** + time-series + correlation + scatter + histogram + boxes + bars) |
| `delivery_stops.csv` | 90 delivery stops clustered in metro Denver + 4 bad-geocode strays in neighboring states, with `zone`/`vehicle` categoricals, `packages`, and correlated `weight_kg`/`distance_km`/`delivery_minutes` numerics over a `delivered_date` | `smart` (geographic outlier markers + core/full extent boxes, Core/Full zoom buttons & spatial-extent call-out with `geocode`; plus boxes, bars, correlation heatmap, strongest-pair scatter & a time-series — no `--smarter` needed) |

## The smart dashboard
Expand Down Expand Up @@ -162,11 +162,16 @@ qsv viz smart quakes.csv -o quakes_dashboard.html
# them out in the spatial-extent label, e.g. "... — 4 outliers (Wyoming, Kansas & Nebraska)"
qsv viz smart delivery_stops.csv -o delivery_dashboard.html

# the full geospatial dashboard: a map, a time-series, a correlation heatmap + drill-down
# scatter, a bimodal-depth histogram, annotated boxes and frequency bars — all auto-chosen.
# the full geospatial dashboard: a map, a prefecture choropleth, a time-series, a correlation
# heatmap + drill-down scatter, a bimodal-depth histogram, annotated boxes and frequency bars —
# all auto-chosen. --geojson + --feature-id-key add a point-in-polygon prefecture choropleth: each
# quake is binned into the GeoJSON region that contains it (no geocoding). This catalog is mostly
# offshore, so each such quake snaps to its nearest prefecture; add --no-snap to drop offshore
# points and color on-land prefectures only. A stderr note reports coverage either way.
# Recognized lat/lon columns are charted on the map only, not as redundant distribution panels.
# Rendered with the built-in plotly_dark theme (--theme works on every chart type, incl. smart).
qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3 -o seismic_dashboard.html
qsv viz smart seismic_events.csv --smarter --theme plotly_dark --grid-cols 3 \
--geojson japan_prefectures.geojson --feature-id-key properties.id -o seismic_dashboard.html
```

### dictionary-guided hierarchy panels (treemap / sunburst)
Expand Down
Loading
Loading