TIMX 646 - avoid virtual filename column reaning in read_parquet

[ghukill]

Purpose and background context

Please see ticket for background: https://mitlibraries.atlassian.net/browse/TIMX-646.

In summary:

• a new application was performing a where filename = ... clause that had not been done before because it was not needed for day-to-day ETL usage of TDA
• it reveals a potential bug in DuckDB where the virtual column filename when reading parquet files is potentially treated differently for selecting vs filtering against

This PR updates TDA to stop renaming the virtual filename column when reading append deltas for data types (records, embeddings, fulltexts) to avoid subtle bugs in downstream applications. Instead, we use a simple glob() approach, which we were doing already in other contexts.

As noted in the Jira ticket, a DuckDB issue has been created. This may or may not result in a change in DuckDB, but either way we need to address it in the short-term.

The effect of these changes in TDA are minimal to none. These changes are mostly for downstream applications that may want to filter on the metadata.records table by the parquet filepaths; the column filename should be predictably for the data parquet files, not append deltas.

How can a reviewer manually see the effects of these changes?

Observe the bug in DuckDB

1- Ensure you are on DuckDB 1.5.2 or higher

2- Open a DuckDB shell

duckdb

3- Create a couple of parquet files:

COPY (
    select
        'foo' as name,
        42 as number,
        '/path/to/a/related/thing1' as filename
)
TO '/tmp/duckdb-fake-data-1.parquet';

COPY (
    select
        'bar' as name,
        101 as number,
        '/path/to/a/related/thing2' as filename
)
TO '/tmp/duckdb-fake-data-2.parquet';

4- Create a view that projects over these, renaming the virtual filename column:

create view deltas as
select *
from read_parquet(
    '/tmp/duckdb-fake-data-*.parquet',
    filename = 'append_delta_filename'
);

5- Note that a select statement treats these columns correctly:

select filename, append_delta_filename
from deltas;
/*
┌─────────┬───────────────────────────┬─────────────────────────────────┐
│  name   │         filename          │      append_delta_filename      │
│ varchar │          varchar          │             varchar             │
├─────────┼───────────────────────────┼─────────────────────────────────┤
│ foo     │ /path/to/a/related/thing1 │ /tmp/duckdb-fake-data-1.parquet │
│ bar     │ /path/to/a/related/thing2 │ /tmp/duckdb-fake-data-2.parquet │
└─────────┴───────────────────────────┴─────────────────────────────────┘
*/

The filename column is the "real" data from the parquet file, and the virtual/dynamic append_delta_filename column is the filepath of the parquet files we're reading from using read_parquet(). At this point, all is working as expected.

6- Apply a filter to the filename column:

select name
from deltas
where filename = '/path/to/a/related/thing1';

/*
┌─────────┐
│  name   │
│ varchar │
└─────────┘
  0 rows 
*/

Given the results from our select statement above, we should have expected to find the "foo" record. What happened instead was DuckDB applied a File Filter, ignoring our renamed virtual column, and actually filters on what we intended to be the append_delta_filename column not the filename column in the view:

explain select name
from deltas
where filename = '/path/to/a/related/thing1';

/*
┌─────────────────────────────┐
│┌───────────────────────────┐│
││       Physical Plan       ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│        READ_PARQUET       │
│    ────────────────────   │
│         Function:         │
│        READ_PARQUET       │
│                           │
│     Projections: name     │
│                           │
│       File Filters:       │   <-----------------
│  (filename = '/path/to/a  │
│     /related/thing1')     │
│                           │
│    Scanning Files: 0/2    │
│                           │
│          ~0 rows          │
└───────────────────────────┘
*/

Confirm that new glob() approach works in TDA

1- Start ipython shell and load a local, temp dataset:

from timdex_dataset_api import TIMDEXDataset
from timdex_dataset_api.config import configure_dev_logger

configure_dev_logger()

td = TIMDEXDataset("/tmp/tda-timx-646")

2- Write some records, build metadatda, and write some more. This will result in some append deltas:

from tests.utils import generate_sample_records

# write first records
records = generate_sample_records(10)
td.records.write(records)

# rebuild
td.metadata.rebuild_dataset_metadata()

# write more records
records = generate_sample_records(10)
td.records.write(records)

# refresh
td.refresh()

3- Count append deltas:

from timdex_dataset_api import TIMDEXRecords

td.metadata.append_deltas_count_for(TIMDEXRecords)

4- Merge append deltas:

td.metadata.merge_append_deltas()

5- Count again, should be zero:

td.metadata.append_deltas_count_for(TIMDEXRecords)

This demonstrates that counting + merging of append delta remains the same after these changes.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES: append_delta_filename is no longer a column in the metadata.<data_type> views

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-646

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ehanson8]

Interesting bug you discovered and this seems like a solid fix, good manual tests to illustrate the bug and prove the fix didn't break anything

[Copilot]

Pull request overview

This PR adjusts how TIMDEX Dataset API builds and uses append-delta metadata views to avoid a DuckDB edge case where renaming the virtual filename column can cause WHERE filename = ... filters to be applied against the wrong (virtual) column. The change is aimed at making metadata.<data_type>/metadata.<data_type>_append_deltas behave predictably for downstream consumers that filter on filename.

Changes:

• Stop renaming DuckDB’s virtual parquet filename column when creating metadata.<type>_append_deltas views; rely on the real filename column stored in append-delta parquet files.
• Update append-delta merge logic to enumerate delta files via DuckDB glob() rather than selecting the previously-exposed append_delta_filename column.
• Update tests to reflect the removal of append_delta_filename and add coverage for filtering by filename and for missing/merged delta scenarios.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
uv.lock	Bumps local package version reference to 5.2.0.
timdex_dataset_api/metadata.py	Removes append_delta_filename surfacing, switches merge filename collection to glob(), and adds safer counting behavior for missing delta views.
tests/test_metadata.py	Updates expected schema, adds regression test for filename filtering, and adds count-for behavior tests.
pyproject.toml	Bumps project version to 5.2.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.