Ensure all migrations use schema defaults

[melton-jason]

Fixes #8307

The most direct cause of the #8307 was that migrations were not using the schema_localization_en.json file.
This meant that explicit Django migrations that worked with the Schema Config would not "default" to our expected Schema Defaults.

Take for example the migration for specify/0020:, which would trigger an update for the PaleoContext -> tectonicUnit relationship in the Schema Config:

specify7/specifyweb/specify/migration_utils/migration_helpers/helper_0020_add_tectonicunit_to_pc_in_schema_config.py

Lines 7 to 9 in 27cb105

	MIGRATION_0020_FIELDS = {
	'PaleoContext': ['tectonicUnit'],
	}

Specify would first attempt to check whether the PaleoContext table already existed in the Schema Config and create the table if it didn't exist:

specify7/specifyweb/specify/migration_utils/update_schema_config.py

Lines 1388 to 1389 in 4c80482

	update_table_field_schema_config_with_defaults(
	table, discipline.id, field, apps)

specify7/specifyweb/specify/migration_utils/update_schema_config.py

Lines 513 to 516 in 4c80482

	sp_local_container, _ = _get_or_create_schema_config_row(
	Splocalecontainer,
	container_attrs,
	)

The Issue here is that we are generally expecting PaleoContext to have the description defined in the default schema file:

specify7/config/common/schema_localization_en.json

Lines 4589 to 4590 in 4c80482

	"name": "Paleo Context",
	"desc": "Provides paleontological context for a Collection Object, Collecting Event, or Locality, linking to chronostratigraphy, lithostratigraphy, and biostratigraphy data."

However, the table will be created with the "heuristically" generated default description and not look at the Schema Defaults file.

This PR seeks to address this Issue and have all functions that interact with the Schema Config via the helper functions go through an additional layer of defaults: those defined in the JSON file.

In addition, this PR simplifies the Schema Configuration interface for the application.

Primarily (from an internal Slack message. Also see e258b74):

• There was a lot of redundant and duplicated work in our Schema Config functions, and no real "helpers" to abstract or simplify the complicated logic
  • I've made a SchemaWriter class that abstracts away the underlying Schema structure and database queries so callers don't need to worry about the underlying logic or tables when working with the Schema Config
  • The SchemaWriter class also batches/combines/merges queries where possible that were previously performed separately. This leads to a lot less queries hitting the database, and less overall work for the database

Because the changes are relatively significant, I measured the proportional performance impact of these changes on a variety of databases.
The tests are not performed on a production-ready instance of Specify (using the production image/Gunicorn), but the effects should be relatively proportional.

In terms of speed, I measured the total start up time for Specify (which is largely dominated by the time it takes to apply the Schema Defaults) on my local machine, and measured the following results:

Specify Start Up Time (seconds)	v7_12_0_7_base	v7_12_0_7_schema_config
SAIAB	78.83	13.67
HUJINNHC	155.26	17.33
MHNG	150.33	22.20
KUFish	19	8.10

This shows a massive improvement in how long it takes Specify to apply and/or check a potentially large amount of Schema Configuration records. On average, I saw a 6x improvement in speed.

Knowing it would likely be worse/higher, I also measured the memory impact of these changes. Below are some measurements taken after two actions in Specify which measure the PSS of the Specify process (captured via smem) after each action.

Development ENV PSS Usage (MB)	v7_12_0_7_base	v7_12_0_7_schema_config
HUJINNHC (startup only)	134.749	136.016
HUJINNHC (after creating Discipline)	146.797	148.786
KUFish (startup only)	136.000	136.689
KUFish (after creating Discipline)	147.528	146.847

While there is on average a negative change in memory impact thus far, the change seems small at the moment (in the worse case when profiling a sufficiently large database taking an extra ~2 MB of space).
If this proves to be a concern, we can reduce this memory usage even further by tweaking the batching of queries within SchemaWriter.

Checklist

• Self-review the PR after opening it to make sure the changes look good and
  self-explanatory (or properly documented)
• Add relevant issue to release milestone
• Add pr to documentation list
• Add automated tests
• Add a reverse migration if a migration is present in the PR
• Add migration function to

  specify7/specifyweb/specify/management/commands/run_key_migration_functions.py

  Line 50 in ea04665

  def fix_schema_config(stdout: WriteToStdOut | None = None):

Testing instructions

• On a blank database created with Guided Setup
• Ensure that the Table Labels, Descriptions, and Hidden values are correct for PaleoContext, CollectionObject, Loan, CollectionObjectGroup, TectonicUnit, etc.(any table which is modified in the Schema Config as part of migrations: I can compile a complete list if required).
  • The "correct" values are defined for each table in the schema_localization_en.json file
• Create a Discipline with a type of "fish" and ensure that the field labels and descriptions for the following fields are correct:

Table Name	Field Name	Expected Label and Description
Collection Object Attribute	text8	Color in Life
Collection Object Attribute	text5	TL
Collection Object	text1	Other number
Collecting Event Attribute	text4	Current
Collecting Event Attribute	text7	Turbidity

• On a database that had an existing Discipline
  • Ensure existing Schema Config edits/customizations are not lost on startup
    • If needed, look at the tables and fields from the prior set of testing instructions and ensure that customizations to the Tables/Fields in Schema Config are not lost when the instance is restarted

Summary by CodeRabbit

• New Features

  • Schema configuration now supports discipline-specific defaults and localization overrides.
  • Added clearer handling for missing or ambiguous schema tables and fields.
  • Schema updates now apply labels, descriptions, visibility, and other settings more consistently.

• Bug Fixes

  • Improved case-insensitive detection of existing schema fields and tables.
  • Updated migrations to preserve correct discipline-specific schema settings.

• Tests

  • Expanded integration coverage for schema defaults, localization, and migration reversions.

[coderabbitai]

📝 Walkthrough

Walkthrough

Schema configuration handling now uses cached, discipline-aware defaults and builder-based persistence. Migration callers pass explicit discipline identifiers and types, schema readers enforce deterministic lookups, and tests validate persisted configuration through integration-style assertions.

Changes

Schema configuration workflow

Layer / File(s)	Summary
Defaults and schema reader contracts specifyweb/backend/setup_tool/schema_defaults.py, specifyweb/specify/migration_utils/schema_reader.py	Typed, cached defaults and discipline overrides are merged centrally; schema reads enforce single-record selection and foreign-key payloads use *_id values.
Schema writer builders and persistence specifyweb/specify/migration_utils/schema_writer.py	New builders and SchemaWriter coordinate table, field, localization, and default persistence.
Discipline-aware migration integration specifyweb/backend/workbench/migrations/0007_spdatasetattachment.py, specifyweb/specify/management/commands/*, specifyweb/specify/migration_utils/migration_helpers/*	Migration paths iterate ordered discipline ID/type pairs and call schema update helpers with explicit keyword arguments.
Integration coverage and execution support specifyweb/specify/migration_utils/tests/*, specifyweb/specify/management/commands/tests/schema_config_tests.py, .github/workflows/test.yml, Makefile	Schema tests now inspect persisted records through SchemaReader; checkout fetches full history and MyPy reports tracebacks.

Sequence Diagram(s)

sequenceDiagram
  participant MigrationCommand
  participant SchemaDefaults
  participant SchemaWriter
  participant SchemaReader
  participant Database
  MigrationCommand->>SchemaDefaults: load defaults for discipline_type
  SchemaDefaults-->>MigrationCommand: merged schema defaults
  MigrationCommand->>SchemaWriter: update table or field configuration
  SchemaWriter->>Database: persist containers, fields, and localized strings
  SchemaReader->>Database: read persisted schema configuration
  Database-->>SchemaReader: localized table and field records

Loading

Possibly related issues

• Issue 8307: The migration caller updates directly address schema configuration changes not being applied with discipline-specific defaults.
• Issue 7864: The PR updates QuerySet iteration in several migration helpers, though it does not add .iterator().

Possibly related PRs

• specify/specify7#8039: Directly overlaps the schema-config defaults and deterministic record-selection paths.
• specify/specify7#8223: Shares the schema defaults and migration-helper integration areas.
• specify/specify7#8225: Directly overlaps schema_reader, schema_writer, and their tests.

Suggested reviewers: carolinedenis, grantfitzsimmons, emenslin

🚥 Pre-merge checks | ✅ 3 | ❌ 3

❌ Failed checks (1 warning, 2 inconclusive)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The Makefile and workflow checkout changes are unrelated to schema-default migration fixes and appear out of scope.	Remove or justify the CI/typecheck changes unless they are required for the schema-default migration work.
Automatic Tests	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Testing Instructions	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The migrations and schema helpers now use global schema defaults when creating or updating Schema Config records, matching #8307.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: migrations now consistently use schema defaults.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch v7_12_0_7_schema_config

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[g1rly-c0d3r]

Testing instructions

• Ensure that the Table Labels, Descriptions, and Hidden values are correct for PaleoContext, CollectionObject, Loan, CollectionObjectGroup, TectonicUnit, etc.(any table which is modified in the Schema Config as part of migrations: I can compile a complete list if required).

• Create a Discipline with a type of "fish" and ensure that the field labels and descriptions for the following fields are correct:

  • Ensure existing Schema Config edits/customizations are not lost on startup

---

Looks good, all labels were correct!

[emenslin]

• Ensure that the Table Labels, Descriptions, and Hidden values are correct for PaleoContext, CollectionObject, Loan, CollectionObjectGroup, TectonicUnit, etc.(any table which is modified in the Schema Config as part of migrations: I can compile a complete list if required).

• Create a Discipline with a type of "fish" and ensure that the field labels and descriptions for the following fields are correct:

• Ensure existing Schema Config edits/customizations are not lost on startup

---

Overall looks good but I did run into a few possible problems. I noticed that the Catalog Number format is always set to CatalogNumberString (current, invalid value) no matter what format is selected in guided setup. Although I know that the catalog number format still works, I could see this being very misleading to a user.

Additionally even with a geo discipline selected the Chronostrat and Lithostrat tables are hidden in the schema config by default, but Paleo/Geo Context and Tectonic Unit are visible. All of these should probably be visible for paleo/geo disciplines.

07-22_10.41.mp4

[rijulpoudel]

• On a blank database created through Guided Setup, the tested table captions, descriptions, and hidden settings matched schema_localization_en.json.
• Created a Fish discipline and verified the expected labels and descriptions for the listed fields.
• On the existing database, Schema Config customizations were preserved after restarting Specify.

I found no issues during these checks. Approved.

[melton-jason]

Thank you all for the reviews! 👏

Overall looks good but I did run into a few possible problems. I noticed that the Catalog Number format is always set to CatalogNumberString (current, invalid value) no matter what format is selected in guided setup. Although I know that the catalog number format still works, I could see this being very misleading to a user.

Additionally even with a geo discipline selected the Chronostrat and Lithostrat tables are hidden in the schema config by default, but Paleo/Geo Context and Tectonic Unit are visible. All of these should probably be visible for paleo/geo disciplines.

07-22_10.41.mp4

#8329 (review)

@emenslin, I "fixed" the CollectionObject -> CatalogNumber problem and made PaleContext, Chronostrat, and Lithostrat visible by default in Paleo/Geo Disciplines.

The commit which sets the default catalogNumber (in the Schema Config) to None: c40773f
The commit which unhides PaleoContext, Chronostrat, and Lithostrat in Paleo/Geo disciplines: ebb2a40

The commits are currently on v7_12_0_7_base, and will be included in v7.12.0.7.
Any other modifications to defaults will likely need their own Issue/Feature Request.