diff --git a/EXPLORER_QUERIES.md b/EXPLORER_QUERIES.md new file mode 100644 index 0000000..5bc9248 --- /dev/null +++ b/EXPLORER_QUERIES.md @@ -0,0 +1,168 @@ +# How the Interactive Explorer queries data (plain-English walkthrough) + +*Written for Eric (#268: "where's the SQL and what parquet files are getting +queried"). This is a guided tour, not the full reference — for the exhaustive +file-by-file schema catalog see `SERIALIZATIONS.md`, and for how those files +get built from the raw export see `DATA_PROVENANCE.md`.* + +## The one-sentence version + +There is no server and no database. The Explorer is a static webpage +(`explorer.qmd`, compiled to `explorer.html`) that runs a real SQL engine +**inside your browser tab** (DuckDB-WASM) and points it at plain `.parquet` +files sitting on a public URL. Every "query" is your browser fetching just +the byte-ranges of a file it needs and running SQL against them locally — no +request ever goes to an iSamples server. + +## Where the files live + +Every file is served from `https://data.isamples.org/.parquet` — +that's a Cloudflare Worker in front of a storage bucket, not a database +server. You can open any of these URLs directly, or point DuckDB at them +(see "Try it yourself" below). The Explorer picks the current filenames in +one place, `explorer.qmd` around **line 800-864**, e.g.: + +```js +lite_url = `${R2_BASE}/isamples_202608_samples_map_lite_v2.parquet` // map points + table +wide_url = `${R2_BASE}/isamples_202608_wide.parquet` // full sample detail +facets_url = `${R2_BASE}/isamples_202608_sample_facets_v3.parquet` // material/context/object_type + search text +h3_res4_url = `${R2_BASE}/isamples_202608_h3_summary_res4.parquet` // pre-counted globe dots (world zoom) +``` + +| File | Plain-English role | Roughly how big | +|---|---|---| +| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~280 MB | +| `..._samples_map_lite_v2.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~50-60 MB | +| `..._sample_facets_v3.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~60 MB | +| `..._h3_summary_res{4,6,8}.parquet` | Pre-counted dots for the globe at 3 zoom tiers (continent / region / neighborhood), so zooming out never counts 6M rows live | tiny–few MB | +| `..._facet_summaries.parquet`, `..._facet_cross_filter.parquet`, `..._facet_tree_*.parquet` | Pre-computed facet-checkbox counts at various levels of "how many filters are active" — the whole point of these is to avoid a live COUNT over millions of rows | KB–tens of MB | +| `..._sample_facet_masks.parquet`, `..._sample_facet_index.parquet` | Bitmask tricks so 2+ facet filters at once are still fast (see `SERIALIZATIONS.md` §4.12 if you want the gory detail) | ~10 MB each | +| `vocab_labels_*.parquet` | URI → human-readable label lookup (e.g. `.../material/1.0/rock` → "Rock") | ~60 KB | + +*Full list with exact schemas: `SERIALIZATIONS.md`. This table is the subset +that matters for "what happens when I click around the Explorer."* + +## What happens when you... + +### ...load the page + +The globe draws immediately from `h3_res4_url` — one query, pre-aggregated, +instant: + +```sql +SELECT h3_cell, sample_count, center_lat, center_lng, dominant_source +FROM read_parquet('h3_summary_res4.parquet') +``` +*(`explorer.qmd` — the `phase1` cell, ~line 2209.)* This is why the globe +never feels like it's "loading 6 million points" — it's loading ~38,000 +pre-counted hexagon summaries instead. + +### ...zoom in + +As you zoom past a threshold, the Explorer swaps to res6, then res8 H3 +tiles (same idea, finer hexagons), and eventually to individual points from +`samples_map_lite_v2.parquet` once there are few enough in view to draw +directly. + +### ...click a facet checkbox (Material / Sampled Feature / Object Type / Source) + +Two things happen. First, the map/table re-query with an added `WHERE` +clause built from your selection (`facetFilterSQL()` in `explorer.qmd`). +Second, the OTHER facets' counts need to update ("if I also filtered by +SESAR, how many rocks would there be") — that's the expensive part, and it's +answered by pre-computed cross-filter tables (`facet_cross_filter`, +`facet_tree_cross_filter`) rather than a live scan, *unless* you have 2+ +filters active at world zoom, in which case a bitmask index +(`sample_facet_masks` + `facet_node_bits`) does a fast columnar AND/OR +instead of scanning the full membership table. This part has had the most +engineering attention (issues #290/#293/#304/#305/#306) because it's the +slowest possible query shape — "count matches under an arbitrary combination +of filters" doesn't pre-aggregate cleanly. + +### ...search for text + +Search runs `ILIKE`-style matching against `sample_facets_v3.parquet`'s +description column (which has vocabulary-concept labels appended at build +time, so a search for "pottery" also matches samples only tagged with a +pottery *concept*, not just the word): + +```sql +SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v3.parquet') +WHERE description ILIKE '%pottery%' +``` +*(`buildSearchFilter()`, `explorer.qmd` ~line 5370-5415.)* Matching pids are +staged into a table (`search_pids`) that every other query — map, table, +facet counts — then filters against, so search composes with facets instead +of being a separate mode. + +### ...view the Samples table + +The table pages through `samples_map_lite_v2.parquet` (coords/label/place/ +date) and, as of #311, joins in `sample_facets_v3.parquet` for +material/object type/sampled feature — one query per page (default page +size), not the whole result set: + +```sql +WITH page AS ( + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('samples_map_lite_v2.parquet') + WHERE + ORDER BY pid LIMIT 50 OFFSET 0 +) +SELECT page.*, f.material, f.context, f.object_type +FROM page LEFT JOIN read_parquet('sample_facets_v3.parquet') AS f ON f.pid = page.pid +``` +*(`loadPage()`, `explorer.qmd` ~line 2755-2775.)* "Download CSV" (#312) runs +the same shape without the `LIMIT`/`OFFSET` (capped at 50,000 rows so an +unfiltered world-zoom export can't hang the tab). + +### ...click a sample point + +One `wide.parquet` self-join to pull the full detail row plus its material/ +object-type concept labels (`explorer.qmd` ~line 2114-2135): + +```sql +SELECT s.description, s.thumbnail_url, mat_lbl.pref_label AS material_label, obj_lbl.pref_label AS object_type_label +FROM read_parquet('wide.parquet') s +LEFT JOIN read_parquet('wide.parquet') mat ON mat.row_id = s.p__has_material_category[1] +LEFT JOIN read_parquet('vocab_labels.parquet') mat_lbl ON mat_lbl.uri = mat.pid +WHERE s.pid = '' +``` +This is the one query that reads from `wide.parquet` on click (everything +above deliberately avoids touching the 280 MB wide file until you actually +need full detail on one sample). + +## Try it yourself + +You don't need the browser — any of this works from the DuckDB CLI or +`isamples-python`, against the exact same public URLs the Explorer uses: + +```sql +-- how many samples per source, right now, live off the public URL +SELECT n AS source, COUNT(*) +FROM read_parquet('https://data.isamples.org/isamples_202608_wide.parquet') +WHERE otype = 'MaterialSampleRecord' +GROUP BY n ORDER BY 2 DESC; + +-- same "pottery" search the Explorer runs +SELECT pid, label, source +FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v3.parquet') +WHERE description ILIKE '%pottery%' +LIMIT 20; +``` + +DuckDB only fetches the byte ranges it needs (via HTTP range requests), so +this is fast even though the files are hundreds of MB — you're not +downloading the whole thing to run a filtered query. + +## Where to go deeper + +- **`SERIALIZATIONS.md`** — every file, full schema, exact row/byte counts, the URL convention (versioned vs `/current/` alias). +- **`DATA_PROVENANCE.md`** — how `wide.parquet` itself gets built from the frozen Zenodo export, and the sidecar-enrichment pattern Eric's OpenContext PQG feeds into. +- **`query-spec.qmd`** — the dimension-to-file binding table (source/material/bbox/h3/time/text → which file answers each). +- **`scripts/build_frontend_derived.py`** — the actual SQL that produces every file in the table above, if you want the ground truth rather than my paraphrase of it. + +--- +*Written 2026-07 in response to #268. If something above doesn't match what +you're actually seeing when you click around, that's a bug in this doc (or a +real bug) — file it.* diff --git a/ISSUE_313_GUIDANCE_2026-07-03.md b/ISSUE_313_GUIDANCE_2026-07-03.md new file mode 100644 index 0000000..61e80aa --- /dev/null +++ b/ISSUE_313_GUIDANCE_2026-07-03.md @@ -0,0 +1,79 @@ +# #313 — browser/bandwidth guidance (draft, post-#317 verification) + +*Drafted 2026-07-03. This is the deliverable #313 actually asked for +("recommend what browser the explorer works best in", "recommended minimum +bandwidth") — written now because the prerequisite fix (#317) is confirmed +deployed as of today. Draft for Raymond's review before posting to #313 or +folding into `how-to-use.qmd`.* + +## What changed since Andrea filed this + +Andrea's original repro: two Material facets active at world zoom on Firefox +→ counts show `--` instead of numbers. Root cause (traced 2026-06-26, +`ISSUE_313_FINDINGS_2026-06-26.md`): the boot-time readiness check was +scanning ~20 MB of index data (`sample_facet_index` full-table distinct scan ++ a full coverage GROUP BY) before multi-filter counts would activate. On a +slow connection or a backgrounded tab, that took 20-80+ seconds; select a +2nd facet inside that window and you get the dash. + +**PR #317 (merged 2026-07-01, R2 upload confirmed live today) fixes the two +biggest contributors:** +- The readiness check now reads a ~1 KB trusted manifest instead of + scanning the 9.68 MB index file. +- The 9.67 MB mask-scan is decoupled from the readiness gate (runs after, + not blocking it). + +**Verified live on isamples.org just now** (Playwright): `window.__facetIndexStatus` +reads `"ready"`, and reproducing Andrea's exact scenario — two Material +facets active at world zoom — returns real counts (e.g. "Anthropogenic +environment (157,333)"), not dashes, for every Sampled Feature / Specimen +Type row checked. The dash bug, as originally reported, is fixed. + +## What's still true (not fixed by #317, don't overclaim) + +**DuckDB-WASM's range-request fallback (#190) is still open**, and it's not +Firefox-specific — I saw it fire in a Chromium session today too: every +parquet fetch, including the new ~1 KB manifest, logs `falling back to full +HTTP read for: ...`. Before #317 this meant full-HTTP-reading a combined +~20 MB of index+mask data on every load. After #317, the same fallback now +mostly applies to files that are KB-to-low-MB, so the fallback is far +cheaper even though it isn't fixed. Net effect: meaningfully better, but +"why does the network tab show full reads instead of range requests" is a +real, separate, still-open question (#190) — don't tell Andrea it's gone. + +## Recommended guidance for Andrea (and how-to-use.qmd, pending Raymond's OK) + +**Minimum bandwidth:** the boot-critical data (globe tiles + facet-ready +manifest) is now ~0.5-0.6 MB total (measured live today: 505 KB H3 tiles + +59 KB vocab labels + ~3 KB of small manifests). On a typical broadband or +LTE connection this is sub-second; even on a slow/throttled connection +(think old hotel wifi) it should resolve in single-digit seconds, not the +tens-of-seconds the pre-#317 dash bug required. The DuckDB engine itself +(a WASM module + worker script, roughly 1 MB combined per today's +measurement) is a one-time download your browser caches after the first +visit — it doesn't re-download on later visits or page navigations within +the site. + +**Browser recommendation:** no browser is currently *broken* — the dash bug +reproduced on Firefox but the root cause (slow boot-time scan) was +connection-speed-dependent, not Firefox-specific, and #317's fix applies +equally to every browser. That said, Chrome/Chromium-based browsers remain +the most-tested path for this project (the CI smoke gate runs Chromium; +Firefox got one targeted spec in #317's P6, not full coverage). If you hit +something that looks Firefox-specific, it's still the least-covered browser +here — worth flagging which one you're on when filing. + +**What to expect when clicking facets / spinning the globe:** brief loading +states are normal and intentional — the table dims and shows a spinner, +facet counts may briefly hold their previous value or show "Loading…" +rather than jump straight to a new number. A permanent dash that never +recovers, or a hard freeze, is not expected behavior post-#317 and is worth +a fresh bug report with browser + connection details. + +## Suggested next step + +Post a version of the "what changed" + "what's still true" sections above +as a comment on #313 (not closing it — #190 and the Firefox-coverage gap +are legitimate open follow-ups), and consider folding the "recommended +guidance" section into `how-to-use.qmd` as a short "Performance tips" +callout. Holding both for Raymond's sign-off before posting/publishing. diff --git a/_quarto.yml b/_quarto.yml index c6d0035..7bafcc4 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -5,6 +5,16 @@ project: - assets/js/source-palette.js - assets/js/sql-builders.js - assets/js/explorer-utils.js + # #295: quarto.js unconditionally fetches /listings.json on every page + # whose frontmatter declares `categories:` (explorer.qmd and others do, + # for the tag chips under the title). Quarto only ever GENERATES that + # file for sites with an actual `listing:` page — this project has none, + # so the fetch 404s on every load (harmless — the JS no-ops on non-200 — + # but it's a real, reported console error, see #295/#297). Shipping a + # static empty-array listings.json is exactly what Quarto itself would + # emit for a site with zero listings: same shape, zero behavior change, + # just a 200 instead of a 404. + - listings.json website: title: "iSamples" diff --git a/explorer.qmd b/explorer.qmd index 3af0c0d..5057ff2 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -473,11 +473,31 @@ format: @media (prefers-reduced-motion: reduce) { .table-loading-spinner { animation: none; border-top-color: #1565c0; } } + .samples-section-header { + display: flex; + justify-content: space-between; + align-items: baseline; + gap: 8px; + margin: 18px 0 8px; + } .samples-section-heading { font-size: 15px; font-weight: 600; color: #234; - margin: 18px 0 8px; + margin: 0; + } + .table-download-btn { + background: #1565c0; + color: white; + border: 0; + border-radius: 4px; + padding: 5px 10px; + font-size: 12px; + cursor: pointer; + } + .table-download-btn:disabled { + background: #c7d2df; + cursor: default; } .samples-table tbody tr { cursor: pointer; @@ -721,7 +741,10 @@ Loading H3 global overview... +

Samples table

+ +
Loading samples matching the current filters...
@@ -2537,6 +2560,7 @@ tableView = { const pageInfoEl = document.getElementById('tablePageInfo'); const tableEl = document.getElementById('samplesTable'); const containerEl = document.getElementById('tableContainer'); + const downloadBtn = document.getElementById('tableDownloadCsv'); function totalPagesFor(total) { return Math.max(1, Math.ceil(total / TABLE_PAGE_SIZE)); @@ -2629,18 +2653,30 @@ tableView = { const labelHtml = `${safeLabel}`; const pidAttr = escapeHtml(r.pid || ''); const selectedClass = (r.pid && r.pid === selectedPid) ? ' class="selected"' : ''; + // #311: material/context/object_type are URIs (from facets_url, + // joined in loadPage()); resolve to human labels the same way + // the facet tree and search results already do. context is + // Andrea's "Sampled feature" (matches #291's facet-tree naming: + // context -> Sampled Feature, object_type -> Specimen Type). + const uriLabel = (uri) => uri + ? escapeHtml((typeof window !== 'undefined' && window.conceptLabelForUri) + ? window.conceptLabelForUri(uri) : uri) + : ''; return ` ${tableSourceBadge(r.source)} ${labelHtml} ${escapeHtml(place)} ${escapeHtml(r.result_time || '')} + ${uriLabel(r.material)} + ${uriLabel(r.object_type)} + ${uriLabel(r.context)} ${escapeHtml(lat)} ${escapeHtml(lng)} `; }).join(''); tableEl.innerHTML = `
- + ${body}
SourceLabelPlaceDateLatLon
SourceLabelPlaceDateMaterialObject typeSampled featureLatLon
`; @@ -2823,16 +2859,26 @@ tableView = { // that contains nulls is only deterministic by accident on a // read-only parquet snapshot. Filter them defensively so the // page contract holds even if a future parquet rev allows nulls. + // #311: material/context/object_type come from facets_url (scalar + // URI columns keyed by pid), joined AFTER paging (CTE) so the join + // only ever touches TABLE_PAGE_SIZE rows, not a full facets scan. + // Labels are resolved client-side via window.conceptLabelForUri() + // in renderTable() rather than joining vocab_labels here too. const data = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE pid IS NOT NULL - ${sourceFilterSQL('source')} - ${facetFilterSQL()} - ${bboxSQL} - ${searchFilterSQL('pid')} - ORDER BY pid - LIMIT ${TABLE_PAGE_SIZE} OFFSET ${offset} + WITH page AS ( + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('${lite_url}') + WHERE pid IS NOT NULL + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + ${bboxSQL} + ${searchFilterSQL('pid')} + ORDER BY pid + LIMIT ${TABLE_PAGE_SIZE} OFFSET ${offset} + ) + SELECT page.*, f.material, f.context, f.object_type + FROM page + LEFT JOIN read_parquet('${facets_url}') AS f ON f.pid = page.pid `); if (genAtStart !== pageGen) return true; const arr = Array.from(data); @@ -2944,6 +2990,109 @@ tableView = { } } + // #312: "Download CSV" exports the CURRENT FILTERED result set (viewport + // + source/facet/search filters), not just the visible page. Capped at + // CSV_ROW_CAP rows so a broad/unfiltered export at world zoom can't hang + // the tab on a multi-million-row pull; the meta line says so honestly + // (same "honesty" convention as the rest of this table) rather than + // silently truncating. + const CSV_ROW_CAP = 50000; + const CSV_HEADERS = ['pid', 'source', 'label', 'place', 'date', 'material', 'object_type', 'sampled_feature', 'latitude', 'longitude']; + + function csvField(v) { + let s = v == null ? '' : String(v); + // Formula-injection hardening: a label/place_name is public data we + // don't control, and this file is expected to land in Excel/Sheets. + // A cell starting with =, +, -, or @ is interpreted as a formula + // there — prefix with a single quote (a no-op in spreadsheet apps, + // rendered as plain text) so no downstream formula ever executes. + // Skipped for actual JS numbers (latitude/longitude): a negative + // coordinate like -122.27 stringifies to "-122.27", which would + // otherwise be misidentified as a formula and corrupted. + if (typeof v !== 'number' && /^[=+\-@]/.test(s)) s = `'${s}`; + // \r included (not just \n) — a bare carriage return in a field + // still breaks row structure for RFC 4180 parsers (Codex review). + return /[",\r\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s; + } + + function csvRow(r) { + const placeParts = r.place_name; + const place = Array.isArray(placeParts) && placeParts.length > 0 + ? placeParts.filter(Boolean).join(' › ') : ''; + const labelFor = (uri) => uri + ? ((typeof window !== 'undefined' && window.conceptLabelForUri) ? window.conceptLabelForUri(uri) : uri) + : ''; + return [r.pid, r.source, r.label, place, r.result_time, + labelFor(r.material), labelFor(r.object_type), labelFor(r.context), + r.latitude, r.longitude].map(csvField).join(','); + } + + let downloadInFlight = false; + + async function downloadCsv() { + if (downloadInFlight || !downloadBtn) return; + const bboxSQL = viewerBboxSQL('latitude', 'longitude', VIEWPORT_PAD_FACTOR); + if (bboxSQL === null) { + setMeta('No globe area in view; pan or zoom the globe before downloading.', true); + return; + } + downloadInFlight = true; + downloadBtn.disabled = true; + const prevLabel = downloadBtn.textContent; + downloadBtn.textContent = 'Preparing…'; + try { + const where = ` + WHERE pid IS NOT NULL + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + ${bboxSQL} + ${searchFilterSQL('pid')} + `; + const countData = await db.query(`SELECT COUNT(*) AS n FROM read_parquet('${lite_url}') ${where}`); + const total = Number(Array.from(countData)[0]?.n ?? 0); + if (total === 0) { + setMeta('No samples match the current filters — nothing to download.', true); + return; + } + const capped = total > CSV_ROW_CAP; + const rowsData = await db.query(` + WITH page AS ( + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('${lite_url}') + ${where} + ORDER BY pid + LIMIT ${CSV_ROW_CAP} + ) + SELECT page.*, f.material, f.context, f.object_type + FROM page + LEFT JOIN read_parquet('${facets_url}') AS f ON f.pid = page.pid + `); + const rows = Array.from(rowsData); + const csv = [CSV_HEADERS.join(','), ...rows.map(csvRow)].join('\r\n'); + const blob = new Blob([csv], { type: 'text/csv;charset=utf-8;' }); + const url = URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + a.download = `isamples-explorer-samples-${rows.length}${capped ? `-of-${total}` : ''}.csv`; + document.body.appendChild(a); + a.click(); + a.remove(); + URL.revokeObjectURL(url); + setMeta(capped + ? `Downloaded first ${rows.length.toLocaleString()} of ${total.toLocaleString()} matching samples as CSV — narrow your filters for a complete export.` + : `Downloaded ${rows.length.toLocaleString()} samples as CSV.`); + } catch (err) { + console.error('CSV download failed:', err); + setMeta('CSV download failed; adjust filters and try again.', true); + } finally { + downloadInFlight = false; + downloadBtn.disabled = false; + downloadBtn.textContent = prevLabel; + } + } + + if (downloadBtn) downloadBtn.addEventListener('click', downloadCsv); + if (prevBtn) prevBtn.addEventListener('click', () => { if (currentPage <= 0) return; refreshPage(currentPage - 1); diff --git a/listings.json b/listings.json new file mode 100644 index 0000000..fe51488 --- /dev/null +++ b/listings.json @@ -0,0 +1 @@ +[] diff --git a/scripts/build_frontend_derived.py b/scripts/build_frontend_derived.py index ae6d3fd..417d892 100755 --- a/scripts/build_frontend_derived.py +++ b/scripts/build_frontend_derived.py @@ -7,14 +7,17 @@ machine-identifiable and re-verifiable. INPUT CONTRACT (enforced/handled): - A wide PQG parquet with entity rows incl. MaterialSampleRecord + - IdentifiedConcept. The `geometry` column may be **WKB BLOB** or DuckDB - **GEOMETRY** — both are handled (detected at runtime). Concept references - live in `p__has_{material,context,sample_object}_category` row-id arrays. + A wide PQG parquet with entity rows incl. MaterialSampleRecord, + IdentifiedConcept, SamplingEvent, and SamplingSite. The `geometry` column + may be **WKB BLOB** or DuckDB **GEOMETRY** — both are handled (detected at + runtime). Concept references live in + `p__has_{material,context,sample_object}_category` row-id arrays. + `p__produced_by[1]` (Sample->SamplingEvent) and SamplingEvent's + `p__sampling_site[1]` (->SamplingSite) resolve place_name/result_time (#311). OUTPUTS (into --outdir, prefixed --tag): - - {tag}_sample_facets_v2.parquet pid, source, material, context, object_type, label, description (search-only; includes appended concept labels), place_name(VARCHAR) - - {tag}_samples_map_lite.parquet pid, label, source, latitude, longitude, place_name(VARCHAR[]), result_time, h3_res4(UBIGINT), h3_res6(UBIGINT), h3_res8(UBIGINT), h3_res8_hex + - {tag}_sample_facets_v2.parquet pid, source, material, context, object_type, label, description (search-only; includes appended concept labels), place_name(VARCHAR) [resolved via SamplingEvent/SamplingSite traversal, #311] + - {tag}_samples_map_lite.parquet pid, label, source, latitude, longitude, place_name(VARCHAR[]) [SamplingSite, #311], result_time [SamplingEvent, #311], h3_res4(UBIGINT), h3_res6(UBIGINT), h3_res8(UBIGINT), h3_res8_hex - {tag}_h3_summary_res{4,6,8}.parquet h3_cell(UBIGINT), sample_count(INT), center_lat, center_lng, dominant_source, source_count(INT), resolution(INT) - {tag}_facet_summaries.parquet facet_type, facet_value, scheme, count - {tag}_facet_cross_filter.parquet filter_source/material/context/object_type, facet_type, facet_value, count @@ -176,6 +179,29 @@ def build_base_tables(con, wide, t0): WHERE ic.label IS NOT NULL AND TRIM(ic.label) != '' GROUP BY r.pid; + -- #311: place_name and result_time are declared on MaterialSampleRecord's + -- own schema, but empirically (production wide, 2026-07) they are 100% + -- NULL there for BOTH fields — s.place_name/s.result_time is a dead read. + -- The real values live one (result_time) or two (place_name) hops away in + -- the standard iSamples graph: Sample -produced_by-> SamplingEvent + -- [-sampling_site-> SamplingSite]. Verified against production data: + -- SamplingEvent.result_time is ~95% populated over the located universe; + -- SamplingSite.place_name (via SamplingEvent.p__sampling_site) ~37%. + -- `ev`/`site` mirror the existing `mat`/`ctx`/`obj` row_id-join pattern, + -- including its same [1]-pick limitation: a sample with MULTIPLE + -- produced_by events would only ever consult the first. Verified against + -- production (202608 wide): 0 of 6.68M located samples have more than one + -- p__produced_by entry today, so this matches reality now; if that ever + -- stops being true, revisit alongside context/object_type's [1]-pick + -- (same tradeoff, tracked in the pipeline epic). + CREATE OR REPLACE TEMP TABLE ev AS + SELECT row_id, result_time, p__sampling_site[1] AS site_row_id + FROM read_parquet('{wide}') WHERE otype='SamplingEvent'; + + CREATE OR REPLACE TEMP TABLE site AS + SELECT row_id, place_name + FROM read_parquet('{wide}') WHERE otype='SamplingSite'; + -- one row per MaterialSampleRecord; all concept resolution via JOINs (decorrelated). CREATE OR REPLACE TEMP TABLE samp AS SELECT @@ -183,8 +209,20 @@ def build_base_tables(con, wide, t0): s.n AS source, s.label, s.description, - s.place_name, -- VARCHAR[] - s.result_time, + -- CASE (not COALESCE) so an EMPTY s.place_name array doesn't win over a + -- populated site.place_name — COALESCE(([]::VARCHAR[]), [...]) returns + -- [], not [...], because [] is non-NULL (Codex review catch). Keeps the + -- same backward-compat intent: a future wide build that populates + -- place_name/result_time directly on the sample row still wins over + -- the traversal, for both NULL and empty-array/empty-string forms. + CASE WHEN s.place_name IS NOT NULL AND len(s.place_name) > 0 + THEN s.place_name ELSE site.place_name END AS place_name, -- VARCHAR[] + -- CAST to VARCHAR before TRIM: s.result_time's declared type varies + -- (VARCHAR in production wide; the test fixtures below use TIMESTAMP + -- literals) and TRIM only accepts VARCHAR — cast explicitly so this + -- works regardless of which type this particular wide build carries. + CASE WHEN s.result_time IS NOT NULL AND TRIM(CAST(s.result_time AS VARCHAR)) != '' + THEN s.result_time ELSE ev.result_time END AS result_time, ROUND(ST_Y({geom}), 6) AS latitude, ROUND(ST_X({geom}), 6) AS longitude, mat.material AS material, @@ -196,6 +234,8 @@ def build_base_tables(con, wide, t0): LEFT JOIN ic AS ctx ON ctx.row_id = s.p__has_context_category[1] LEFT JOIN ic AS obj ON obj.row_id = s.p__has_sample_object_type[1] LEFT JOIN concept_labels cl ON cl.pid = s.pid + LEFT JOIN ev ON ev.row_id = s.p__produced_by[1] + LEFT JOIN site ON site.row_id = ev.site_row_id WHERE s.otype='MaterialSampleRecord'; CREATE OR REPLACE TEMP TABLE samp_geo AS diff --git a/tests/test_frontend_derived.py b/tests/test_frontend_derived.py index b90f2c4..b273217 100644 --- a/tests/test_frontend_derived.py +++ b/tests/test_frontend_derived.py @@ -63,12 +63,20 @@ def build_fixture_wide(path, geom_mode): geom = (lambda lng, lat: f"ST_AsWKB(ST_Point({lng},{lat}))") if geom_mode == "blob" \ else (lambda lng, lat: f"ST_Point({lng},{lat})") + # p__produced_by / p__sampling_site (#311): NULL for every row here — this + # fixture never links to a SamplingEvent/SamplingSite, so the new + # traversal joins in build_frontend_derived.py are no-ops and every + # existing assertion in this file (place_name/result_time as set + # directly on the MSR row below) is unaffected. See + # test_place_name_and_result_time_via_graph_traversal for the + # traversal-populated case. ic_rows = " UNION ALL ".join( f"SELECT 'IdentifiedConcept' AS otype, '{uri}' AS pid, {rid}::BIGINT AS row_id, NULL::VARCHAR AS n, " f"NULL::VARCHAR AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, " f"NULL::TIMESTAMP AS result_time, NULL AS geometry, " f"NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " - f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords" + f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site" for rid, uri in CONCEPTS) msr = [] @@ -79,13 +87,15 @@ def build_fixture_wide(path, geom_mode): f"'label {pid}' AS label, 'desc {pid}' AS description, ['plc-{pid}','x''q']::VARCHAR[] AS place_name, " f"NULL::TIMESTAMP AS result_time, {geom(lng, lat)} AS geometry, " f"{_arr(marr)} AS p__has_material_category, [10]::BIGINT[] AS p__has_context_category, " - f"[20]::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords") + f"[20]::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site") # one NULL-geometry sample -> must be EXCLUDED from located outputs msr.append( "SELECT 'MaterialSampleRecord' AS otype, 'm-nogeo' AS pid, NULL::BIGINT AS row_id, 'TEST' AS n, " "'l' AS label, 'd' AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " "NULL AS geometry, [4]::BIGINT[] AS p__has_material_category, [10]::BIGINT[] AS p__has_context_category, " - "[20]::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords") + "[20]::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site") con.execute(f"COPY ({ic_rows} UNION ALL {' UNION ALL '.join(msr)}) " f"TO '{path}' (FORMAT PARQUET)") @@ -143,6 +153,138 @@ def test_place_name_serialized_and_quotes(tmp_path): assert isinstance(pn, str) and "plc-m-real-first" in pn # VARCHAR, not array, embedded quote survived +# --------------------------------------------------------------------------- +# #311 — place_name/result_time are declared on MaterialSampleRecord but are +# 100% NULL there in production; the real values live on SamplingEvent +# (result_time) and SamplingSite (place_name), reached via +# Sample -produced_by-> SamplingEvent [-sampling_site-> SamplingSite]. +# Verified against production wide.parquet 2026-07 before writing this fix: +# SamplingEvent.result_time ~95% populated over the located universe; +# SamplingSite.place_name (via SamplingEvent.p__sampling_site) ~37%. +# --------------------------------------------------------------------------- + +def build_traversal_fixture(path, geom_mode="blob"): + """A wide parquet with 3 MaterialSampleRecords exercising every + traversal case: full chain (event + site), event with no site, and no + produced_by link at all (nothing to traverse -> stays NULL).""" + con = duckdb.connect() + con.execute("INSTALL spatial; LOAD spatial;") + geom = (lambda lng, lat: f"ST_AsWKB(ST_Point({lng},{lat}))") if geom_mode == "blob" \ + else (lambda lng, lat: f"ST_Point({lng},{lat})") + + # otype, pid, row_id, n, label, description, place_name, result_time, + # geometry, p__has_material_category, p__has_context_category, + # p__has_sample_object_type, p__keywords, p__produced_by, p__sampling_site + rows = [ + # t-full: MSR has NULL place_name/result_time directly; produced_by=[100] + # -> SamplingEvent row_id=100 (result_time set, sampling_site=[200]) + # -> SamplingSite row_id=200 (place_name set). Both must resolve. + "SELECT 'MaterialSampleRecord' AS otype, 't-full' AS pid, NULL::BIGINT AS row_id, 'TEST' AS n, " + "'label' AS label, 'desc' AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " + f"{geom(10.0, 40.0)} AS geometry, NULL::BIGINT[] AS p__has_material_category, " + "NULL::BIGINT[] AS p__has_context_category, NULL::BIGINT[] AS p__has_sample_object_type, " + "NULL::BIGINT[] AS p__keywords, [100]::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + "SELECT 'SamplingEvent' AS otype, 'ev-100' AS pid, 100::BIGINT AS row_id, NULL::VARCHAR AS n, " + "NULL::VARCHAR AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, " + "TIMESTAMP '2020-06-15 00:00:00' AS result_time, NULL AS geometry, " + "NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " + "NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, [200]::BIGINT[] AS p__sampling_site", + + "SELECT 'SamplingSite' AS otype, 'site-200' AS pid, 200::BIGINT AS row_id, NULL::VARCHAR AS n, " + "NULL::VARCHAR AS label, NULL::VARCHAR AS description, ['Full Chain Site']::VARCHAR[] AS place_name, " + "NULL::TIMESTAMP AS result_time, NULL AS geometry, " + "NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " + "NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + # t-event-only: produced_by=[101] -> SamplingEvent with result_time but + # NO sampling_site -> result_time resolves, place_name stays NULL. + "SELECT 'MaterialSampleRecord' AS otype, 't-event-only' AS pid, NULL::BIGINT AS row_id, 'TEST' AS n, " + "'label' AS label, 'desc' AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " + f"{geom(11.0, 41.0)} AS geometry, NULL::BIGINT[] AS p__has_material_category, " + "NULL::BIGINT[] AS p__has_context_category, NULL::BIGINT[] AS p__has_sample_object_type, " + "NULL::BIGINT[] AS p__keywords, [101]::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + "SELECT 'SamplingEvent' AS otype, 'ev-101' AS pid, 101::BIGINT AS row_id, NULL::VARCHAR AS n, " + "NULL::VARCHAR AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, " + "TIMESTAMP '2021-03-01 00:00:00' AS result_time, NULL AS geometry, " + "NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " + "NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + # t-no-event: no produced_by link at all -> both stay NULL (no crash). + "SELECT 'MaterialSampleRecord' AS otype, 't-no-event' AS pid, NULL::BIGINT AS row_id, 'TEST' AS n, " + "'label' AS label, 'desc' AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " + f"{geom(12.0, 42.0)} AS geometry, NULL::BIGINT[] AS p__has_material_category, " + "NULL::BIGINT[] AS p__has_context_category, NULL::BIGINT[] AS p__has_sample_object_type, " + "NULL::BIGINT[] AS p__keywords, NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + # t-empty-direct (Codex review catch): MSR carries an EMPTY array/string + # directly (not NULL) for place_name/result_time — a naive COALESCE + # would let the empty value win (COALESCE([]::VARCHAR[], [...]) = [], + # not [...], since [] is non-NULL). The CASE form must still fall + # through to the traversal here, same as the NULL case. + "SELECT 'MaterialSampleRecord' AS otype, 't-empty-direct' AS pid, NULL::BIGINT AS row_id, 'TEST' AS n, " + "'label' AS label, 'desc' AS description, []::VARCHAR[] AS place_name, ''::VARCHAR AS result_time, " + f"{geom(13.0, 43.0)} AS geometry, NULL::BIGINT[] AS p__has_material_category, " + "NULL::BIGINT[] AS p__has_context_category, NULL::BIGINT[] AS p__has_sample_object_type, " + "NULL::BIGINT[] AS p__keywords, [102]::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + + "SELECT 'SamplingEvent' AS otype, 'ev-102' AS pid, 102::BIGINT AS row_id, NULL::VARCHAR AS n, " + "NULL::VARCHAR AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, " + "TIMESTAMP '2022-11-20 00:00:00' AS result_time, NULL AS geometry, " + "NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " + "NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, [201]::BIGINT[] AS p__sampling_site", + + "SELECT 'SamplingSite' AS otype, 'site-201' AS pid, 201::BIGINT AS row_id, NULL::VARCHAR AS n, " + "NULL::VARCHAR AS label, NULL::VARCHAR AS description, ['Empty-Direct Fallthrough Site']::VARCHAR[] AS place_name, " + "NULL::TIMESTAMP AS result_time, NULL AS geometry, " + "NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " + "NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + "NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site", + ] + con.execute(f"COPY ({' UNION ALL '.join(rows)}) TO '{path}' (FORMAT PARQUET)") + con.close() + + +def test_place_name_and_result_time_via_graph_traversal(tmp_path): + wide = str(tmp_path / "wide.parquet") + build_traversal_fixture(wide) + r = run_builder(wide, str(tmp_path), "t") + assert r.returncode == 0, f"builder failed:\n{r.stdout}\n{r.stderr}" + + con = duckdb.connect() + ml = f"read_parquet('{tmp_path / 't_samples_map_lite.parquet'}')" + rows = {r[0]: (r[1], r[2]) for r in + con.sql(f"SELECT pid, place_name, result_time FROM {ml}").fetchall()} + + place, result_time = rows["t-full"] + assert place is not None and "Full Chain Site" in list(place), \ + f"t-full place_name should resolve via SamplingEvent->SamplingSite, got {place!r}" + assert result_time is not None and "2020-06-15" in str(result_time), \ + f"t-full result_time should resolve via SamplingEvent, got {result_time!r}" + + place, result_time = rows["t-event-only"] + assert result_time is not None and "2021-03-01" in str(result_time), \ + f"t-event-only result_time should resolve via SamplingEvent, got {result_time!r}" + assert not place, f"t-event-only has no sampling_site; place_name should stay NULL, got {place!r}" + + # Codex review catch: an EMPTY (not NULL) direct value must still fall + # through to the traversal — COALESCE would have let '' / [] win here. + place, result_time = rows["t-empty-direct"] + assert place is not None and "Empty-Direct Fallthrough Site" in list(place), \ + f"t-empty-direct's EMPTY place_name array must fall through to the traversal, got {place!r}" + assert result_time is not None and "2022-11-20" in str(result_time), \ + f"t-empty-direct's EMPTY '' result_time must fall through to the traversal, got {result_time!r}" + + place, result_time = rows["t-no-event"] + assert not place and not result_time, \ + f"t-no-event has no produced_by link; both should stay NULL, got place={place!r} result_time={result_time!r}" + + def test_cli_rejects_unknown_only(tmp_path): wide = str(tmp_path / "wide.parquet"); build_fixture_wide(wide, "blob") r = run_builder(wide, str(tmp_path), "t", extra=["--only", "bogus_name"]) @@ -342,7 +484,8 @@ def build_tree_fixture(wide, vocab): f"SELECT 'IdentifiedConcept' AS otype, '{uri}' AS pid, {rid}::BIGINT AS row_id, NULL::VARCHAR AS n, " f"'lbl' AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " f"NULL AS geometry, NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " - f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords" + f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site" for rid, uri in TREE_CONCEPTS) msr = [] for i, (pid, src, m, c, o) in enumerate(TREE_SAMPLES): @@ -351,7 +494,8 @@ def build_tree_fixture(wide, vocab): f"'label {pid}' AS label, 'desc {pid}' AS description, ['plc-{pid}']::VARCHAR[] AS place_name, " f"NULL::TIMESTAMP AS result_time, ST_AsWKB(ST_Point({10.0+i},{40.0+i})) AS geometry, " f"{_arr(m)} AS p__has_material_category, {_arr(c)} AS p__has_context_category, " - f"{_arr(o)} AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords") + f"{_arr(o)} AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site") con.execute(f"COPY ({ic_rows} UNION ALL {' UNION ALL '.join(msr)}) TO '{wide}' (FORMAT PARQUET)") edges = " UNION ALL ".join( f"SELECT '{u}' uri, {('NULL' if p is None else repr(p))}::VARCHAR broader, 'data_v1' uri_form" @@ -559,7 +703,8 @@ def build_index_fixture(wide, vocab): f"SELECT 'IdentifiedConcept' AS otype, '{uri}' AS pid, {rid}::BIGINT AS row_id, NULL::VARCHAR AS n, " f"'lbl' AS label, NULL::VARCHAR AS description, NULL::VARCHAR[] AS place_name, NULL::TIMESTAMP AS result_time, " f"NULL AS geometry, NULL::BIGINT[] AS p__has_material_category, NULL::BIGINT[] AS p__has_context_category, " - f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords" + f"NULL::BIGINT[] AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site" for rid, uri in TREE_CONCEPTS) msr = [] for i, (pid, src, m, c, o) in enumerate(INDEX_SAMPLES): @@ -568,7 +713,8 @@ def build_index_fixture(wide, vocab): f"'label {pid}' AS label, 'desc {pid}' AS description, ['plc-{pid}']::VARCHAR[] AS place_name, " f"NULL::TIMESTAMP AS result_time, ST_AsWKB(ST_Point({10.0+i},{40.0+i})) AS geometry, " f"{_arr(m)} AS p__has_material_category, {_arr(c)} AS p__has_context_category, " - f"{_arr(o)} AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords") + f"{_arr(o)} AS p__has_sample_object_type, NULL::BIGINT[] AS p__keywords, " + f"NULL::BIGINT[] AS p__produced_by, NULL::BIGINT[] AS p__sampling_site") con.execute(f"COPY ({ic_rows} UNION ALL {' UNION ALL '.join(msr)}) TO '{wide}' (FORMAT PARQUET)") edges = " UNION ALL ".join( f"SELECT '{u}' uri, {('NULL' if p is None else repr(p))}::VARCHAR broader, 'data_v1' uri_form"