Explorer fixes for Eric (#295) and Andrea (#311, #312)

EXPLORER_QUERIES.md

@@ -0,0 +1,168 @@
+# How the Interactive Explorer queries data (plain-English walkthrough)
+
+*Written for Eric (#268: "where's the SQL and what parquet files are getting
+queried"). This is a guided tour, not the full reference — for the exhaustive
+file-by-file schema catalog see `SERIALIZATIONS.md`, and for how those files
+get built from the raw export see `DATA_PROVENANCE.md`.*
+
+## The one-sentence version
+
+There is no server and no database. The Explorer is a static webpage
+(`explorer.qmd`, compiled to `explorer.html`) that runs a real SQL engine
+**inside your browser tab** (DuckDB-WASM) and points it at plain `.parquet`
+files sitting on a public URL. Every "query" is your browser fetching just
+the byte-ranges of a file it needs and running SQL against them locally — no
+request ever goes to an iSamples server.
+
+## Where the files live
+
+Every file is served from `https://data.isamples.org/<filename>.parquet` —
+that's a Cloudflare Worker in front of a storage bucket, not a database
+server. You can open any of these URLs directly, or point DuckDB at them
+(see "Try it yourself" below). The Explorer picks the current filenames in
+one place, `explorer.qmd` around **line 800-864**, e.g.:
+
+```js
+lite_url    = `${R2_BASE}/isamples_202608_samples_map_lite_v2.parquet`   // map points + table
+wide_url    = `${R2_BASE}/isamples_202608_wide.parquet`                  // full sample detail
+facets_url  = `${R2_BASE}/isamples_202608_sample_facets_v3.parquet`      // material/context/object_type + search text
+h3_res4_url = `${R2_BASE}/isamples_202608_h3_summary_res4.parquet`       // pre-counted globe dots (world zoom)
+```
+
+| File | Plain-English role | Roughly how big |
+|---|---|---|
+| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~280 MB |
+| `..._samples_map_lite_v2.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~50-60 MB |
+| `..._sample_facets_v3.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~60 MB |
+| `..._h3_summary_res{4,6,8}.parquet` | Pre-counted dots for the globe at 3 zoom tiers (continent / region / neighborhood), so zooming out never counts 6M rows live | tiny–few MB |
+| `..._facet_summaries.parquet`, `..._facet_cross_filter.parquet`, `..._facet_tree_*.parquet` | Pre-computed facet-checkbox counts at various levels of "how many filters are active" — the whole point of these is to avoid a live COUNT over millions of rows | KB–tens of MB |
+| `..._sample_facet_masks.parquet`, `..._sample_facet_index.parquet` | Bitmask tricks so 2+ facet filters at once are still fast (see `SERIALIZATIONS.md` §4.12 if you want the gory detail) | ~10 MB each |
+| `vocab_labels_*.parquet` | URI → human-readable label lookup (e.g. `.../material/1.0/rock` → "Rock") | ~60 KB |
+
+*Full list with exact schemas: `SERIALIZATIONS.md`. This table is the subset
+that matters for "what happens when I click around the Explorer."*
+
+## What happens when you...
+
+### ...load the page
+
+The globe draws immediately from `h3_res4_url` — one query, pre-aggregated,
+instant:
+
+```sql
+SELECT h3_cell, sample_count, center_lat, center_lng, dominant_source
+FROM read_parquet('h3_summary_res4.parquet')
+```
+*(`explorer.qmd` — the `phase1` cell, ~line 2209.)* This is why the globe
+never feels like it's "loading 6 million points" — it's loading ~38,000
+pre-counted hexagon summaries instead.
+
+### ...zoom in
+
+As you zoom past a threshold, the Explorer swaps to res6, then res8 H3
+tiles (same idea, finer hexagons), and eventually to individual points from
+`samples_map_lite_v2.parquet` once there are few enough in view to draw
+directly.
+
+### ...click a facet checkbox (Material / Sampled Feature / Object Type / Source)
+
+Two things happen. First, the map/table re-query with an added `WHERE`
+clause built from your selection (`facetFilterSQL()` in `explorer.qmd`).
+Second, the OTHER facets' counts need to update ("if I also filtered by
+SESAR, how many rocks would there be") — that's the expensive part, and it's
+answered by pre-computed cross-filter tables (`facet_cross_filter`,
+`facet_tree_cross_filter`) rather than a live scan, *unless* you have 2+
+filters active at world zoom, in which case a bitmask index
+(`sample_facet_masks` + `facet_node_bits`) does a fast columnar AND/OR
+instead of scanning the full membership table. This part has had the most
+engineering attention (issues #290/#293/#304/#305/#306) because it's the
+slowest possible query shape — "count matches under an arbitrary combination
+of filters" doesn't pre-aggregate cleanly.
+
+### ...search for text
+
+Search runs `ILIKE`-style matching against `sample_facets_v3.parquet`'s
+description column (which has vocabulary-concept labels appended at build
+time, so a search for "pottery" also matches samples only tagged with a
+pottery *concept*, not just the word):
+
+```sql
+SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v3.parquet')
+WHERE description ILIKE '%pottery%'
+```
+*(`buildSearchFilter()`, `explorer.qmd` ~line 5370-5415.)* Matching pids are
+staged into a table (`search_pids`) that every other query — map, table,
+facet counts — then filters against, so search composes with facets instead
+of being a separate mode.
+
+### ...view the Samples table
+
+The table pages through `samples_map_lite_v2.parquet` (coords/label/place/
+date) and, as of #311, joins in `sample_facets_v3.parquet` for
+material/object type/sampled feature — one query per page (default page
+size), not the whole result set:
+
+```sql
+WITH page AS (
+    SELECT pid, label, source, latitude, longitude, place_name, result_time
+    FROM read_parquet('samples_map_lite_v2.parquet')
+    WHERE <your active filters>
+    ORDER BY pid LIMIT 50 OFFSET 0
+)
+SELECT page.*, f.material, f.context, f.object_type
+FROM page LEFT JOIN read_parquet('sample_facets_v3.parquet') AS f ON f.pid = page.pid
+```
+*(`loadPage()`, `explorer.qmd` ~line 2755-2775.)* "Download CSV" (#312) runs
+the same shape without the `LIMIT`/`OFFSET` (capped at 50,000 rows so an
+unfiltered world-zoom export can't hang the tab).
+
+### ...click a sample point
+
+One `wide.parquet` self-join to pull the full detail row plus its material/
+object-type concept labels (`explorer.qmd` ~line 2114-2135):
+
+```sql
+SELECT s.description, s.thumbnail_url, mat_lbl.pref_label AS material_label, obj_lbl.pref_label AS object_type_label
+FROM read_parquet('wide.parquet') s
+LEFT JOIN read_parquet('wide.parquet') mat ON mat.row_id = s.p__has_material_category[1]
+LEFT JOIN read_parquet('vocab_labels.parquet') mat_lbl ON mat_lbl.uri = mat.pid
+WHERE s.pid = '<clicked pid>'
+```
+This is the one query that reads from `wide.parquet` on click (everything
+above deliberately avoids touching the 280 MB wide file until you actually
+need full detail on one sample).
+
+## Try it yourself
+
+You don't need the browser — any of this works from the DuckDB CLI or
+`isamples-python`, against the exact same public URLs the Explorer uses:
+
+```sql
+-- how many samples per source, right now, live off the public URL
+SELECT n AS source, COUNT(*)
+FROM read_parquet('https://data.isamples.org/isamples_202608_wide.parquet')
+WHERE otype = 'MaterialSampleRecord'
+GROUP BY n ORDER BY 2 DESC;
+
+-- same "pottery" search the Explorer runs
+SELECT pid, label, source
+FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v3.parquet')
+WHERE description ILIKE '%pottery%'
+LIMIT 20;
+```
+
+DuckDB only fetches the byte ranges it needs (via HTTP range requests), so
+this is fast even though the files are hundreds of MB — you're not
+downloading the whole thing to run a filtered query.
+
+## Where to go deeper
+
+- **`SERIALIZATIONS.md`** — every file, full schema, exact row/byte counts, the URL convention (versioned vs `/current/` alias).
+- **`DATA_PROVENANCE.md`** — how `wide.parquet` itself gets built from the frozen Zenodo export, and the sidecar-enrichment pattern Eric's OpenContext PQG feeds into.
+- **`query-spec.qmd`** — the dimension-to-file binding table (source/material/bbox/h3/time/text → which file answers each).
+- **`scripts/build_frontend_derived.py`** — the actual SQL that produces every file in the table above, if you want the ground truth rather than my paraphrase of it.
+
+---
+*Written 2026-07 in response to #268. If something above doesn't match what
+you're actually seeing when you click around, that's a bug in this doc (or a
+real bug) — file it.*

ISSUE_313_GUIDANCE_2026-07-03.md

@@ -0,0 +1,79 @@
+# #313 — browser/bandwidth guidance (draft, post-#317 verification)
+
+*Drafted 2026-07-03. This is the deliverable #313 actually asked for
+("recommend what browser the explorer works best in", "recommended minimum
+bandwidth") — written now because the prerequisite fix (#317) is confirmed
+deployed as of today. Draft for Raymond's review before posting to #313 or
+folding into `how-to-use.qmd`.*
+
+## What changed since Andrea filed this
+
+Andrea's original repro: two Material facets active at world zoom on Firefox
+→ counts show `--` instead of numbers. Root cause (traced 2026-06-26,
+`ISSUE_313_FINDINGS_2026-06-26.md`): the boot-time readiness check was
+scanning ~20 MB of index data (`sample_facet_index` full-table distinct scan
++ a full coverage GROUP BY) before multi-filter counts would activate. On a
+slow connection or a backgrounded tab, that took 20-80+ seconds; select a
+2nd facet inside that window and you get the dash.
+
+**PR #317 (merged 2026-07-01, R2 upload confirmed live today) fixes the two
+biggest contributors:**
+- The readiness check now reads a ~1 KB trusted manifest instead of
+  scanning the 9.68 MB index file.
+- The 9.67 MB mask-scan is decoupled from the readiness gate (runs after,
+  not blocking it).
+
+**Verified live on isamples.org just now** (Playwright): `window.__facetIndexStatus`
+reads `"ready"`, and reproducing Andrea's exact scenario — two Material
+facets active at world zoom — returns real counts (e.g. "Anthropogenic
+environment (157,333)"), not dashes, for every Sampled Feature / Specimen
+Type row checked. The dash bug, as originally reported, is fixed.
+
+## What's still true (not fixed by #317, don't overclaim)
+
+**DuckDB-WASM's range-request fallback (#190) is still open**, and it's not
+Firefox-specific — I saw it fire in a Chromium session today too: every
+parquet fetch, including the new ~1 KB manifest, logs `falling back to full
+HTTP read for: ...`. Before #317 this meant full-HTTP-reading a combined
+~20 MB of index+mask data on every load. After #317, the same fallback now
+mostly applies to files that are KB-to-low-MB, so the fallback is far
+cheaper even though it isn't fixed. Net effect: meaningfully better, but
+"why does the network tab show full reads instead of range requests" is a
+real, separate, still-open question (#190) — don't tell Andrea it's gone.
+
+## Recommended guidance for Andrea (and how-to-use.qmd, pending Raymond's OK)
+
+**Minimum bandwidth:** the boot-critical data (globe tiles + facet-ready
+manifest) is now ~0.5-0.6 MB total (measured live today: 505 KB H3 tiles +
+59 KB vocab labels + ~3 KB of small manifests). On a typical broadband or
+LTE connection this is sub-second; even on a slow/throttled connection
+(think old hotel wifi) it should resolve in single-digit seconds, not the
+tens-of-seconds the pre-#317 dash bug required. The DuckDB engine itself
+(a WASM module + worker script, roughly 1 MB combined per today's
+measurement) is a one-time download your browser caches after the first
+visit — it doesn't re-download on later visits or page navigations within
+the site.
+
+**Browser recommendation:** no browser is currently *broken* — the dash bug
+reproduced on Firefox but the root cause (slow boot-time scan) was
+connection-speed-dependent, not Firefox-specific, and #317's fix applies
+equally to every browser. That said, Chrome/Chromium-based browsers remain
+the most-tested path for this project (the CI smoke gate runs Chromium;
+Firefox got one targeted spec in #317's P6, not full coverage). If you hit
+something that looks Firefox-specific, it's still the least-covered browser
+here — worth flagging which one you're on when filing.
+
+**What to expect when clicking facets / spinning the globe:** brief loading
+states are normal and intentional — the table dims and shows a spinner,
+facet counts may briefly hold their previous value or show "Loading…"
+rather than jump straight to a new number. A permanent dash that never
+recovers, or a hard freeze, is not expected behavior post-#317 and is worth
+a fresh bug report with browser + connection details.
+
+## Suggested next step
+
+Post a version of the "what changed" + "what's still true" sections above
+as a comment on #313 (not closing it — #190 and the Firefox-coverage gap
+are legitimate open follow-ups), and consider folding the "recommended
+guidance" section into `how-to-use.qmd` as a short "Performance tips"
+callout. Holding both for Raymond's sign-off before posting/publishing.

_quarto.yml

@@ -5,6 +5,16 @@ project:
     - assets/js/source-palette.js
     - assets/js/sql-builders.js
     - assets/js/explorer-utils.js
+    # #295: quarto.js unconditionally fetches /listings.json on every page
+    # whose frontmatter declares `categories:` (explorer.qmd and others do,
+    # for the tag chips under the title). Quarto only ever GENERATES that
+    # file for sites with an actual `listing:` page — this project has none,
+    # so the fetch 404s on every load (harmless — the JS no-ops on non-200 —
+    # but it's a real, reported console error, see #295/#297). Shipping a
+    # static empty-array listings.json is exactly what Quarto itself would
+    # emit for a site with zero listings: same shape, zero behavior change,
+    # just a 200 instead of a 404.
+    - listings.json
 
 website:
   title: "iSamples"