Conversation
Seven additions on top of the initial bench harness: - Per-row info ⓘ popovers explaining each Configuration field. - Swift Charts stacked-bar showing decode/sample/quantize means. - Photo source — pick from library, center-cropped to each grid size. - '.auto' quantizer added so the gating decision itself is measured. - Run note field, flowed into CSV header. - CSV header gains palettekit_version, device_marketing, source. - Summary CSV gains decode_mean / sample_mean / quantize_mean. Plus polish: marketing-name device labels, tighter fonts, Grid-based summary table, spinner progress, failure surfacing, Run/Reset/Cancel state separation.
…mark/ Three small repo-level changes for the v1.1 release: - README gains a 'Benchmark on your device' section so the new bench harness is discoverable. - 'API at a glance' replaced by a concise method skeleton plus a link to the DocC reference. ExtractionOptions defaults move to Options.md (single source of truth, no README drift). - benchmark/ is gitignored — the convention for storing CSV exports locally without leaking device-specific results into the repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Examples/PaletteKitDemoso PaletteKit performance claims can be measured per-device, per-content. Pick a real photo or a synthesized fixture, vary size /quantizer / downsample, and export per-stage timings as CSV.
progressive extraction is not worth shipping; v1.1 ships the harness itself instead.
What's included
Bench harness (
Examples/PaletteKitDemo/PaletteKitDemo/Bench/)BenchModels— case definitions (size × quantizer × downsample), per-sample structures, summary aggregates with mean per-stage timings, marketing-name mapping for hardware identifiers.BenchFixture— deterministic synthesized image (gradient + 5 colored blobs + per-pixel noise), plusresizeToSquare(_:side:)for real-photo input center-cropped to each grid size.BenchRunner(@MainActorObservableObject) — orchestrates warmup + measured runs, capturesExtractionTimingsper sample, computes p50/p95/min/max + per-stage means, surfaces failurecounts.
BenchView— Source picker (Synthesized / Photo) withPhotosPicker, Configuration card with per-rowInfoButtonpopovers, Run/Reset/Cancel state machine,Grid-based summary table,stacked-bar chart, raw samples disclosure.
BenchChart(Swift Charts) — horizontal stacked bars showing decode / sample / quantize means per case.BenchExport— Raw and Summary CSV with headers including PaletteKit version, device identifier, marketing name, source descriptor (synthesizedorphoto WxH), and optional run note.BenchInfo— popover content for each Configuration field; reusableInfoButtoncomponent (compact-adapted on iPhone, fixed-width with text wrap).Repo
ExtractionOptionsdefaults now live inOptions.mdas the single source of truth.benchmark/convention..gitignoreaddsbenchmark/so CSV exports stay local without leaking device-specific results into the repo.Methodology notes
The harness exists because v1.0 shipped with provisional CPU/GPU thresholds (
metalAutoThreshold = 500_000) and aquality.stridedefault that no one had measured under load. v1.1 ships themeasurement infrastructure first; v1.x feature work queues behind it.
Findings from the first round (iPhone 15 Pro, both synthesized and a 12 MP photo):
Downsample.automatic(maxPixels: 1_000_000)) flatten quantize cost to ~80–140 ms regardless of input size. Metal vs CPU differ by <5 ms (within noise).Downsample.disabledAND sampled pixel count ≥ ~1M — essentially the 4096²+ raw case.Findings stay local (
memory/); README / DESIGN_SPEC / threshold corrections accumulate for a single batch update around v1.2 instead of churning per-discovery.Test plan
swift test— existing PaletteKit tests pass (no library changes in this PR).make demo-appregenerates the Xcode project; iOS simulator build succeeds.PhotosPickerloads CGImage and shows thumbnail with original dimensions.renders bars correctly.