Reconcile own indices with pre-existing OHDSI-standard indices - #36
Open
nicoloesch wants to merge 13 commits into
Open
Reconcile own indices with pre-existing OHDSI-standard indices#36nicoloesch wants to merge 13 commits into
nicoloesch wants to merge 13 commits into
Conversation
Collaborator
Author
|
Converted to "draft" stage as the structure is entirely dependent on the outcome of AustralianCancerDataNetwork/oa-configurator#18 |
Member
|
the solution presented here is fine, but might be more simply addressed if we change what omop_index returns to match the existing pattern of indices in https://github.com/OHDSI/CommonDataModel/tree/main/inst/ddl/5.4 exceptions include: missing episode / episode event index that should have been pulled in when those tables were merged into 5.4 |
nicoloesch
force-pushed
the
index-reconciliation
branch
from
August 3, 2026 06:19
579f547 to
a465a16
Compare
nicoloesch
marked this pull request as ready for review
August 3, 2026 07:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why structural reconciliation instead of a hardcoded name map
Before merging this we checked whether the naming mismatch could be solved more simply, by making
omop_indexjust emit the same literal names as the official OHDSI CDM DDL. It can't, cleanly:visit_detforvisit_detail,eeforepisode_event.idx_genderindexesgender_concept_id.idx_concept_vocabluary_id.A hardcoded lookup table would only ever match a database built from the exact DDL text it was transcribed from, and would fail silently against an older CDM point release, a hand-modified schema, or a future OHDSI edit.
Structural matching by column-set and uniqueness is the only axis that's actually stable against naming that is, by inspection, not principled. We're keeping the reconciliation approach in this PR for that reason.
Summary
_is_plain_index,_find_equivalent_index,_find_shape_conflictincli_indexes.py) soindexes enablerecognizes an index that already exists under a foreign name (e.g. from the official OHDSI CDM DDL script) and never creates a duplicate.dropped_indexestable in a new reserved schema,omop_alchemy_maintenanceon PostgreSQL) soindexes disablecan safely drop such a foreign-named index for bulk-load speed, and a laterindexes enablerestores it under its original name. Includes schema-scoping to prevent collisions across two schemas.ReservedSchemaStrEnum (_cli_utils.py) centralizingSTAGINGandMAINTENANCE, guarded viareject_reserved_schema()wired into theomop_commanddecorator so every maintenance CLI command is protected, not justvocab load-vocab-source.vocabulary,domain,concept_class,relationshiponto the same pattern already used byperson/location/care_site/provider/concept: cluster directly on the primary key's own index instead of declaring a separate, redundant same-column index. Closes #. See background above on why the official DDL's version of that redundant index needs reconciliation, not replication, to support.cli_schema_reconcile.pyso a foreign-named equivalent index reports a single"renamed"issue instead of a misleading"missing"+"unexpected"pair, and doesn't count as drift. This includes the CLUSTER comparison and in thedoctor/summary rendering layers.cli_vocab.py's bulk vocabulary loader silently discardingindexes disable's warning results; they now surface onVocabularyLoadReport.index_warningsand render as a dedicated panel.indexes disablerun (without an interveningenable) when a different foreign index appears on the same columns now degrades gracefully instead of raising an uncaughtIntegrityError.load_vocab_source()'s reserved-schema guard only coveringSTAGING, notMAINTENANCE, when called directly (bypassing the CLI)._resolve_physical_cluster_name()helper used by bothmanage_indexes()'s cluster step and the standaloneindexes clustercommand, fixing a hardcodedunique=Falsein the latter and removing a duplicated resolution policy that could have drifted out of sync.status: strfield across all tencli_*.pymaintenance modules with one shared typedStatusStrEnum, each member carrying aSeverity(OK/INFO/WARNING/ERROR, itself carrying a renderstyle).FullTextResult/FullTextActionfromomop_alchemy/backends/base.pytoomop_alchemy/maintenance/cli_fulltext.py.manage_indexes()'s independentOptionaloutcome locals into a single_IndexOutcomediscriminated dataclass, so two of them being set simultaneously by a future edit becomes structurally impossible.