diff --git a/docs/openedx_learning/decisions/0002-competency-criteria-model.rst b/docs/openedx_learning/decisions/0002-competency-criteria-model.rst index b41dca757..9ff6cee97 100644 --- a/docs/openedx_learning/decisions/0002-competency-criteria-model.rst +++ b/docs/openedx_learning/decisions/0002-competency-criteria-model.rst @@ -81,7 +81,7 @@ Decision 4. ``course_id``: Nullable foreign key to ``openedx_catalog_courserun.id`` for the course that scopes this criteria tree. 5. ``name``: string 6. ``ordering``: Indicates evaluation sequence number for this criteria group. This defines deterministic evaluation order for siblings during read-time evaluation and event-driven recomputation, and enables short-circuit evaluation. - 7. ``logic_operator``: Either “AND” or “OR” or null. This determines how children are combined at a group node ("AND" or "OR"). + 7. ``logic_operator``: Either “AND” or “OR” or null. This determines how children are combined at a group node ("AND" or "OR"). Deliberately no negation operator: keeping every combinator monotone (advancing a child status can only advance or leave unchanged its parent, never regress it) is what lets :ref:`openedx-learning-adr-0004` record concurrently without coordination. Example: A root group uses "OR" with two child groups. @@ -155,29 +155,68 @@ Decision 3. ``oel_tagging_objecttag(object_id)`` 4. ``CompetencyCriteria(oel_tagging_objecttag_id)`` 5. ``CompetencyCriteria(competency_criteria_group_id)`` - 6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)`` - 7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)`` - 8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)`` - 9. ``CompetencyRuleProfile(competency_taxonomy_id, course_id, organization_id)`` - 10. ``CompetencyMasteryStatuses(status)`` (unique) + 6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)`` (unique; ACTIVE current row) + 7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)`` (unique; ACTIVE current row) + 8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)`` (unique; ACTIVE current row) + 9. ``StudentCompetencyCriteriaStatusHistory(user_id, competency_criteria_id, created)`` (point-in-time reconstruction) + 10. ``StudentCompetencyCriteriaGroupStatusHistory(user_id, competency_criteria_group_id, created)`` + 11. ``StudentCompetencyStatusHistory(user_id, oel_tagging_tag_id, created)`` + 12. ``CompetencyRuleProfile(competency_taxonomy_id, course_id, organization_id)`` + 13. ``CompetencyMasteryStatuses(status)`` (unique) 6. Learner progress status concepts (``StudentCompetency*Status`` database tables) - When a completion event (graded, completed, mastered, etc.) occurs for an object, determine and track the learner's status in earning the competency. To reduce recalculation frequency, store results at each level. + When a completion event (a subsection grade change, delivered as described in + :ref:`openedx-learning-adr-0004`) is evaluated for an object, the learner's status in earning the + competency is determined and tracked. The design: - Relationship to other concepts: + - Stores results at each level (leaf, group, competency) so reads never recompute. + - Splits each level into two tables: + + - ACTIVE: one current row per learner and node, updated in place. Current status is a direct + lookup on this row. + - HISTORY: append-only, one row per genuine status advance. This is the audit and point-in-time + record. - - ``StudentCompetencyCriteriaStatus`` tracks status at ``CompetencyCriterion`` leaf level. - - ``StudentCompetencyCriteriaGroupStatus`` tracks status at ``CompetencyCriteriaGroup`` node level. - - ``StudentCompetencyStatus`` tracks top-level competency demonstration state. - - All learner status rows use a shared lookup table (``CompetencyMasteryStatuses``) so status semantics live in one place and student status tables stay structurally consistent. + - Bounds HISTORY growth: because mastery is monotonic (:ref:`openedx-learning-adr-0005`), advances + per node are capped by the status lattice, so HISTORY grows with learners and nodes, not with + time. - Intended update flow (bottom-up materialization): + These are the large per-learner tables in this model, so they follow the scale posture of + :ref:`openedx-learning-adr-0005` (mirroring edx-platform's persistent grades): - - A learner event updates one ``StudentCompetencyCriteriaStatus`` row. - - Recompute ancestor ``CompetencyCriteriaGroup`` statuses upward to the root. - - At each group, evaluate children in ``ordering`` sequence and short-circuit when the group's result is already determined by its ``logic_operator``. - - Persist only rows whose status changed. + - 64-bit ``BigAutoField`` primary key. + - Narrow rows. + - Per-read-path composite indexes (Decision 5). + - A dedicated database alias reached through a router that defaults to the main database. + - Logical foreign keys to the criteria-definition tables and to the user, declared without + database-level constraints. The tables can therefore live in a different database from the + definitions, with referential integrity and the delete protection of Decision 7 enforced in + application code. + + Relationship to other concepts: + + - ``StudentCompetencyCriteriaStatus`` / ``StudentCompetencyCriteriaStatusHistory`` track status at + the ``CompetencyCriterion`` leaf level. A leaf is atomic, so its status is only ``Demonstrated`` + or ``AttemptedNotDemonstrated``. + - ``StudentCompetencyCriteriaGroupStatus`` / ``StudentCompetencyCriteriaGroupStatusHistory`` track + status at the ``CompetencyCriteriaGroup`` node level. + - ``StudentCompetencyStatus`` / ``StudentCompetencyStatusHistory`` track top-level competency + demonstration state. + - All learner status rows use a shared lookup table (``CompetencyMasteryStatuses``) so status + semantics live in one place and the status tables stay structurally consistent. + + Intended update flow (bottom-up roll-up): + + - A learner event evaluates one leaf; if its status advances, its ACTIVE row is updated in place + and a leaf HISTORY row is appended. + - Recompute ancestor ``CompetencyCriteriaGroup`` statuses upward to the root, reading the stored + child rows directly. + - At each group, evaluate children in ``ordering`` sequence and short-circuit when the group's + result is already determined by its ``logic_operator``. + - Persist only nodes whose status changed: update the ACTIVE row in place and append a HISTORY row. + A banked (``Demonstrated``) status is never regressed, so a downward grade correction advances + nothing and writes no HISTORY row. 1. Add new database table for ``CompetencyMasteryStatuses`` with these columns: @@ -188,34 +227,53 @@ Decision - This table is system-owned lookup data and should be treated as immutable configuration, not user-authored rows. - 2. Add new database table for ``StudentCompetencyCriteriaStatus`` with these columns: - - 1. ``id``: unique primary key - 2. ``competency_criteria_id``: Foreign key to ``CompetencyCriterion.id`` - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table - 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id`` - 5. ``created``: The timestamp at which the student's criterion status was set. - - 3. Add a new database table for ``StudentCompetencyCriteriaGroupStatus`` with these columns: - - 1. ``id``: unique primary key - 2. ``competency_criteria_group_id``: Foreign key to ``CompetencyCriteriaGroup.id`` - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table - 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id`` - 5. ``created``: The timestamp at which the student's criteria-group status was set. - - 4. Add a new database table for ``StudentCompetencyStatus`` with these columns: - - 1. ``id``: unique primary key - 2. ``oel_tagging_tag_id``: Foreign key pointing to Tag id - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table - 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id``. This table should have a constraint to only allow status values of “Demonstrated” and “PartiallyAttempted” since it represents overall competency demonstration state, not in-progress states. - 5. ``created``: The timestamp at which the student's competency status was set. + 2. Add the leaf-level tables ``StudentCompetencyCriteriaStatus`` (ACTIVE) and + ``StudentCompetencyCriteriaStatusHistory`` (HISTORY). Their columns: + + 1. ``id``: 64-bit ``BigAutoField`` primary key + 2. ``competency_criteria_id``: logical foreign key to ``CompetencyCriterion.id`` (no database-level constraint) + 3. ``user_id``: logical foreign key to the learner's user id (no database-level constraint) + 4. ``status_id``: foreign key to ``CompetencyMasteryStatuses.id`` + 5. ``effective_source_timestamp``: UTC timestamp of the source grade change this status was computed from, taken from the event rather than auto-generated. This is source-event time, distinct from ``created`` (persistence time); the out-of-order defense in :ref:`openedx-learning-adr-0004` compares this value, not ``created``. + 6. ``created``: UTC timestamp of when this row was written to the database (``auto_now_add``). On ACTIVE it marks the first advance for this learner and leaf; on HISTORY, when the advance was appended. It diverges from ``effective_source_timestamp`` whenever an event is processed late or arrives out of order. + 7. ``modified``: UTC timestamp of the most recent in-place update (``auto_now``), present on ACTIVE only. ACTIVE rows are updated in place as a learner's status advances; HISTORY rows are append-only and never change after insert, so HISTORY has no ``modified`` column. + + ACTIVE is unique on ``(user_id, competency_criteria_id)``: exactly one current row per learner + and leaf, updated in place. HISTORY carries no uniqueness constraint and is append-only: one row + per genuine advance, with no row written for a suppressed downward correction. + + 3. Add the group-level tables ``StudentCompetencyCriteriaGroupStatus`` (ACTIVE) and + ``StudentCompetencyCriteriaGroupStatusHistory`` (HISTORY). Their columns: + + 1. ``id``: 64-bit ``BigAutoField`` primary key + 2. ``competency_criteria_group_id``: logical foreign key to ``CompetencyCriteriaGroup.id`` (no database-level constraint) + 3. ``user_id``: logical foreign key to the learner's user id (no database-level constraint) + 4. ``status_id``: foreign key to ``CompetencyMasteryStatuses.id`` + 5. ``effective_source_timestamp``: as in the leaf tables + 6. ``created``: as in the leaf tables + 7. ``modified``: as in the leaf tables (ACTIVE only) + + ACTIVE is unique on ``(user_id, competency_criteria_group_id)``: one current row per learner and + group, updated in place. HISTORY is append-only with no uniqueness constraint. + + 4. Add the competency-level tables ``StudentCompetencyStatus`` (ACTIVE) and + ``StudentCompetencyStatusHistory`` (HISTORY). Their columns: + + 1. ``id``: 64-bit ``BigAutoField`` primary key + 2. ``oel_tagging_tag_id``: logical foreign key to the competency ``Tag`` id (no database-level constraint) + 3. ``user_id``: logical foreign key to the learner's user id (no database-level constraint) + 4. ``status_id``: foreign key to ``CompetencyMasteryStatuses.id``. Constrained to “Demonstrated” and “PartiallyAttempted” only, since this represents overall competency demonstration state, not in-progress states. + 5. ``effective_source_timestamp``: as in the leaf tables + 6. ``created``: as in the leaf tables + 7. ``modified``: as in the leaf tables (ACTIVE only) + + ACTIVE is unique on ``(user_id, oel_tagging_tag_id)``: one current row per learner and + competency, updated in place. HISTORY is append-only with no uniqueness constraint. 7. Delete protection boundaries - If no learner status rows exist for a competency definition, hard delete is allowed and cascades through competency metadata tables. - - Once any related learner status exists in ``StudentCompetencyStatus``, ``StudentCompetencyCriteriaGroupStatus``, or ``StudentCompetencyCriteriaStatus``, deletion of associated competency definition rows is blocked. + - Once any related learner status exists (an ACTIVE or HISTORY row at any level), deletion of associated competency definition rows is blocked. Because the learner-status foreign keys carry no database-level constraint (Decision 6), this protection is enforced in application code, not by a database cascade or constraint. - This delete protection applies to ``oel_tagging_taxonomy``, ``CompetencyTaxonomy``, ``oel_tagging_tag``, ``oel_tagging_objecttag``, ``CompetencyCriteriaGroup``, ``CompetencyCriteria``, and ``CompetencyRuleProfile``. - Once any related learner status exists, retiring definitions may be archive-only (hidden from authoring and new associations), not hard delete. @@ -224,6 +282,25 @@ Decision :width: 80% :align: center +.. note:: + + The PNG above is the original overview and is now out of date. It is not regenerated here; apply + the following modifications when reading it. This decision is the source of truth wherever the two + differ. + + 1. Split each learner-status level into two tables: an ACTIVE table (one current row per learner + and node, updated in place) and an append-only HISTORY table (one row per genuine status + advance). The PNG draws a single table per level, so add + ``StudentCompetencyCriteriaStatusHistory``, ``StudentCompetencyCriteriaGroupStatusHistory``, + and ``StudentCompetencyStatusHistory``. + 2. Remove the ``mastery_level_id`` column from ``StudentCompetencyCriteriaStatus``; mastery level + is a future rule type and is not part of this decision. + 3. Add an ``effective_source_timestamp`` column to all six status tables (ACTIVE and HISTORY). + 4. Add a ``modified`` column to the three ACTIVE tables only; the HISTORY tables keep ``created`` + alone, since they are append-only and never updated after insert. + 5. Make each status table's ``id`` a 64-bit ``BigAutoField``, and treat its ``user_id`` and node + references as logical foreign keys with no database-level constraint. + Example ------- diff --git a/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst b/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst index 60f4f5864..9edca17f8 100644 --- a/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst +++ b/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst @@ -39,15 +39,15 @@ For the initial implementation, versioning and traceability of competency achiev 4. Authoring guardrails must warn on potentially impactful edits: - - If a user edits competency criteria definitions or competency object/tag associations after related learner status exists, Studio must display an explicit warning that student statuses have already been set, and these changes will be applied going forward, so existing learner statuses will not be retroactively updated. + - If a user edits competency criteria definitions or competency object/tag associations after related learner status exists, Studio must display an explicit warning that student statuses have already been set. Such edits are monotonic for the learner (:ref:`openedx-learning-adr-0005`): they never lower or revoke an existing status; a structural edit triggers a recompute that can only raise statuses (for example, eased criteria a learner already meets); and a rule or threshold change takes effect only going forward, as new grades arrive. - Applying these changes requires explicit user confirmation. -5. Learner status models/tables are append-only history and do not use ``django-simple-history``: +5. Learner status is stored as an ACTIVE table plus an append-only HISTORY table, using explicit tables rather than ``django-simple-history``: - - For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``, each status change is stored as a new row with ``created`` as the write timestamp. - - Existing learner status rows are not updated in place. - - Current status is determined by the most recent row for a given learner + target entity (ordered by ``created``, with ``id`` as a tie-breaker). - - Older rows represent the learner status history and remain available for audit/tracing. + - For each level (``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``), an ACTIVE table holds one current row per learner and node, updated in place, and a parallel append-only HISTORY table records one row per genuine status advance. The columns are defined in :ref:`openedx-learning-adr-0002`, Decision 6. + - Current status is a direct lookup on the ACTIVE row, not the most recent of many rows. + - A HISTORY row is written only when a status advances. Because mastery is monotonic and advance-only (:ref:`openedx-learning-adr-0005`), the number of advances per node is bounded by the status lattice, so HISTORY grows with learners and nodes, not with time, and still supports point-in-time reconstruction: the status at any past time is the latest advance at or before that time. + - These tables do not use ``django-simple-history``. The advance-only HISTORY is written explicitly by the recorder (:ref:`openedx-learning-adr-0004`) only on a genuine advance, and the recorder writes the ACTIVE and HISTORY tables in bulk. ``django-simple-history`` snapshots on every ``save`` and is oriented to per-row saves, which fits neither the advance-only bounding nor the bulk write path, and it adds history-metadata columns and per-save signal overhead this decision does not want on billion-row tables. It remains the right tool for the low-volume definition tables in Decision 1. Rejected Alternatives @@ -83,3 +83,15 @@ Rejected Alternatives - Cons: - Requires custom tooling to reconstruct past versions - Does not align with existing publishable versioning patterns +5. Append-only learner status only, with current status as the most recent row (the earlier form of this decision, before the ACTIVE/HISTORY split). + - Pros: + - No in-place mutation; a single table per level. + - Cons: + - Current-status reads must resolve the latest of several rows instead of a single-row lookup, on the dashboard hot path. + - There is no in-place current row to anchor per-learner concurrency (:ref:`openedx-learning-adr-0004`). +6. Use ``django-simple-history`` for the learner status tables (as Decision 1 does for definition tables). + - Pros: + - Automatic shadow-table history with no hand-written HISTORY writes, consistent with the definition tables. + - Cons: + - It snapshots on every ``save``, so it cannot express advance-only HISTORY without extra suppression logic, and it does not fit the recorder's bulk write path. + - It adds history-metadata columns and per-save signal overhead that this decision avoids on billion-row tables. diff --git a/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst b/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst new file mode 100644 index 000000000..0b624b918 --- /dev/null +++ b/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst @@ -0,0 +1,225 @@ +.. _openedx-learning-adr-0004: + +4. How should learner competency mastery be recorded concurrently and at scale? +================================================================================ + +Status +------ +Proposed. + +Context +------- +When a learner is graded on a subsection, the platform must evaluate whether that grade +demonstrates any attached competencies and record the learner's mastery. Mastery is recorded at +three levels: the criterion (leaf), the criteria group, and the competency. Per +:ref:`openedx-learning-adr-0002` and :ref:`openedx-learning-adr-0005`, all three levels are +*materialized* (stored), not recomputed on read, so that dashboards and other read surfaces stay +fast. A single grade change therefore writes the changed leaf's status and then re-evaluates and +re-writes the derived rows from that leaf up to the competency root. Per +:ref:`openedx-learning-adr-0005`, each level is stored as an ACTIVE row updated in place, holding +the current status for a learner and node, plus an append-only HISTORY row per genuine status +advance. + +**Monotonicity: competency statuses only ever move forward.** Per +:ref:`openedx-learning-adr-0005`, every node, at every level, advances through a small status +lattice (``AttemptedNotDemonstrated`` to ``PartiallyAttempted`` to ``Demonstrated``) and is never +lowered later. This holds for leaf nodes, group nodes, and top-level competency masteries. + +Two forces shape how recording should happen: + +- **Same-learner correctness.** A grade change writes the changed leaf and then re-derives the + group and competency rows above it. Leaf rows are always correct, since each leaf is a pure + function of its own grade. The derived rows are the hazard: two evaluations for the same learner + that overlap can each read a stale snapshot of the sibling leaf statuses and each write a derived + roll-up computed from an incomplete picture (a *write-skew*). Nothing crashes and no constraint is + violated, but a learner's stored competency status can be silently wrong. + +- **Throughput.** Grading is bursty and spans a very large number of learners, so the recording + path must keep up under peak load. + +The question is how to guarantee same-learner correctness at high throughput, over best-effort +in-process delivery, without making the event bus mandatory. + +Decision +-------- + +**1. Every write is a monotone merge, never a blind overwrite.** A node's status is written as +``status := max(stored status, newly computed status)`` (a single ``GREATEST``-style ``UPDATE``, +atomic at the row for the duration of that one statement, with no application-level lock). Because +the merge takes the higher of the two values, it is commutative, idempotent, and insensitive to +order. This is why out-of-order delivery and re-delivery are harmless without sequence tracking. + + +**2. When a child advances, its parent is recomputed in the same transaction, under a brief row lock on that parent.** +The merge in mechanism 1 makes a single-row write safe, but a *conjunctive* +parent (for example "demonstrated only when all children are demonstrated") is computed by reading +several child rows first, so two overlapping evaluations for one learner could each read a stale +sibling and compute a parent that is too low. To prevent that, recomputing a parent takes a +row-level lock on the parent row (a ``SELECT ... FOR UPDATE``) before reading its children: two +updates that touch the same parent for the same learner take turns, and the second reads the first's +committed children and computes from the complete picture. Locks are taken child-before-parent up +the path to the root, a consistent order, so concurrent updates cannot deadlock. This is an ordinary +single-row lock. + +**3. Out-of-order defense.** A change older than the current leaf's effective source timestamp is +ignored, so a late arrival cannot regress a newer status. The monotone merge already enforces +this for the stored status; the timestamp check avoids writing a spurious HISTORY row for a +stale advance. + +**4. Advance-only; no automatic regression.** Once a status reaches a level it is banked at every +level including the leaf (:ref:`openedx-learning-adr-0005`). A downward grade correction does +not advance the status and writes no HISTORY row, which is what bounds HISTORY by +monotonicity. Reversing a banked status is a separate administrative action, out of scope +here. + +**5. Entry point: record from a subsection-grade-changed signal received in openedx-core.** +edx-platform emits a subsection-granular openedx-event signal; an openedx-core receiver enqueues a +task that does the monotone merge and the upward re-evaluation (see decision 8 for why the receiver +only enqueues). + +**6. Optional batching.** With this ADR, consumer workers can run in parallel, +scaling horizontally. If further performance increase is needed, batching can be introduced: +In that case, a scheduled producer worker on the +edx-platform side polls changed grades, coalesces them, and emits bounded batch events. +Resolving which competencies a subsection feeds is learner-independent, so when batching is used it +is cached and de-duplicated across the batch, and each batch does one bulk read of current ACTIVE +statuses, evaluates in memory, and bulk-writes the changed ACTIVE and HISTORY rows per level. + +**7. One atomic transaction per grade change.** The recording task does the leaf merge and the +upward roll-up (mechanism 2) in one database transaction, so the leaf and every affected ancestor +for a change commit together or not at all. This transaction is the task's own; it is not shared +with edx-platform's grade write (contrast rejected alternative 8), so mastery trails the grade by +the enqueue-and-run delay rather than committing atomically with it. + +**8. Enqueuing a celery task.** The receiver enqueues an async celery task +to do the work to ensure durability. + + +Rejected Alternatives +--------------------- + +1. Serialize all writers with a deployment-wide lock (the previous form of this decision). + + Record under a single deployment-wide lock (or a per-learner lock), so no two workers ever + evaluate the same learner at once and the write-skew cannot occur. This was the earlier design + for competency mastery recording. This alternative is about the *global* serialization lock, not + the brief per-node row lock the chosen design uses in mechanism 2, which is a different and far + cheaper primitive. + + - Pros: + - Correct regardless of delivery order or topology, without depending on monotonicity. + - Self-contained: relies only on the database. + - Cons: + - A single deployment-wide lock forces a single serialized pipeline: recording does not + scale horizontally, which is only adequate while one pipeline keeps up with peak grading. + - A per-learner lock removes that cap but multiplies the lock lifecycle (acquisition, + timeout, stale-lock recovery) across potentially millions of learner keys, and fights the + cross-learner batching that throughput would otherwise rely on. + - Either way it introduces a lock and its failure modes for a guarantee monotonicity already + provides. + - Why rejected: the write-skew a global lock exists to prevent is a consequence of computing + derived rows non-monotonically. Once every write is a monotone merge (mechanism 1) and each + parent recompute is serialized only against concurrent writers of that same node by a brief row + lock (mechanism 2), correctness holds without serializing the whole recorder. The deployment-wide + lock forces a single pipeline and gives up horizontal scale to buy a guarantee that a per-node + row lock already provides at a fraction of the cost. + +2. Allow non-monotonic mastery and repair drift with a reconciliation sweep. + + Let a status move up or down freely with the latest grade, accept that concurrency can leave + derived rows wrong, and run a scheduled job that recomputes each learner from scratch to correct + them. + + - Pros: + - Mastery tracks the current grade exactly, including downward corrections, with no + high-water-mark surprise. + - Cons: + - Gives up order-insensitivity: a non-monotone merge depends on the order writes arrive, so + concurrent writers can corrupt (not merely understate) a derived row, and a brief row lock + no longer suffices to keep them correct. + - Requires an always-on correcting reconciliation process to converge, the kind of standing + cost this decision avoids. + - Why rejected: monotonicity is the linchpin that keeps the write path correct with only a brief + per-node row lock and no global serialization. Trading it away to track downward corrections + reintroduces exactly the coordination problem this decision removes, and the product treats + mastery as banked anyway (:ref:`openedx-learning-adr-0005`). + +3. Overwrite derived rows with the freshly computed value instead of a monotone merge. + + - Why rejected: a blind ``status := computed`` write is a lost update under concurrency, since a + worker computing from a stale sibling snapshot can overwrite a higher value another worker + just wrote. The monotone ``max`` merge is precisely what makes concurrent writes safe; without + it, mechanism 1 does not hold and a lock would be back in scope. + +4. Recompute derived levels on read instead of materializing them. + + - Pros: + - Removes the write-skew hazard entirely: if nothing derived is stored, nothing derived can + drift, and the write path is trivial. + - Cons: + - Reopens :ref:`openedx-learning-adr-0002`, which deliberately materializes derived levels + for dashboard read performance, and moves the cost onto every read. + - Why rejected: out of scope for this decision; the read-performance tradeoff was settled in + :ref:`openedx-learning-adr-0002`. + +5. Let openedx-core obtain grades directly instead of having edx-platform push them. + + Variants: openedx-core polls the persisted subsection-grade and override tables on their indexed + ``modified`` columns; or the subsection-grade model is relocated into openedx-core so edx-platform + imports it; or a swappable base model (in the style of ``AUTH_USER_MODEL``) is defined in + openedx-core and supplied by edx-platform. + + - Why rejected: all three break the layering rule that openedx-core must not depend on + edx-platform. Polling couples the recorder to a private grade schema that changes by migration + without notice and re-implements the override layering it would drift from. Relocating a + mature, deeply woven platform model is a large, risky migration out of proportion to this + feature. The swappable-model pattern is effectively a one-off for the user model, cannot be + retrofitted onto an existing table, and would be first-of-its-kind machinery here. A push from + edx-platform (the versioned event this decision uses, or the synchronous call of rejected + alternative 8) gets the data across the boundary in the correct direction without any of this. + +6. Deliver over a mandatory event bus. + + Route grade events over the Kafka or Redis event bus and consume them in openedx-core as a bus + consumer. + + - Pros: + - The bus is a durable buffer with native batch polling and at-least-once delivery, as a + persistent log rather than the in-process signal plus task-queue retries this decision uses. + - Cons: + - The event bus is not enabled in a stock edx-platform deployment, so requiring it makes + Kafka or Redis mandatory infrastructure for any deployment that wants competencies. + - Why rejected: mandating event-bus infrastructure is a far larger operational imposition than + this feature should force on operators. The chosen design runs over ordinary in-process + delivery and can take advantage of the bus where a deployment already runs one, without + requiring it. + +7. Correctness by keyed partitioning. + + Partition grade-change events by ``user_id`` so every event for a learner is consumed by exactly + one worker, making same-learner events serial by construction with no lock. + + - Why rejected: this was the lock-free alternative worth considering only while correctness + required serialization. Under monotone merge, correctness no longer requires that any two + same-learner events be serialized at all, so partitioning solves a problem this decision no + longer has. It would also couple the recorder to a transport-level routing contract (native on + Kafka, application-level sharding on Redis) that monotonicity makes unnecessary. + +8. Record inside a subsection-grade transaction. + + edx-platform already synchronously, in-process, handles subsection grade changes. + This would wrap a subsection grade change in a transaction that includes competency + mastery updates. + + - Pros: grade and mastery can never diverge; recording is real-time; no new event type is + needed, only a call from code that already runs; and because the leaf and its ancestors + recompute in that same transaction (mechanism 2), the whole subtree is airtight inline with + no follow-up step. + - Cons: mastery work sits on the synchronous grading path, so a slow mastery query or a bug + adds latency to, or rolls back, the grade write (grades are the more critical data); and it + requires the shared database. + - Why rejected: we do not intend for mastery to always be tied to a subsection grade. + We've actually proposed tying mastery to several other things (course completion, unit grade + (a concept that doesn't exist in the platform currently), problem grade, and rubric criterion grade + (also a concept that doesn't exist in the platform currently)), and tying a mastery to a subsection + grade would contradict that. diff --git a/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst b/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst new file mode 100644 index 000000000..b7b62b8ca --- /dev/null +++ b/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst @@ -0,0 +1,352 @@ +.. _openedx-learning-adr-0005: + +5. How should learner competency mastery be stored at scale? +============================================================= + +Status +------ +Proposed. + +Relationship to prior decisions +-------------------------------- +This decision refines two prior ADRs and reverses an earlier draft of this one: + +- It refines :ref:`openedx-learning-adr-0002`'s learner-status model. ADR 0002 materializes + (stores) mastery at all three levels: the criterion (leaf), the criteria group, and the + competency. This decision keeps all three stored, but changes *how*: each level is split into an + ACTIVE table (one current row per learner and node, updated in place) and an append-only HISTORY + table, and the large per-learner leaf tables adopt the large-table techniques edx-platform uses + for persistent grades. +- It supersedes :ref:`openedx-learning-adr-0003`'s rule that the learner-status tables are + append-only with the current status being the most recent row. Those tables move to an ACTIVE + table updated in place plus an append-only HISTORY table. +- It refines :ref:`openedx-learning-adr-0003`'s guardrail that "edits apply going forward; existing + statuses are not retroactively updated," to the monotonic rule described under "Retroactive + criteria changes are monotonic for the learner" below. +- It reverses an earlier draft of this ADR that computed leaves transiently and never stored them. + That design is retained as the first rejected alternative below. + +Context +------- +Per :ref:`openedx-learning-adr-0002`, competency achievement criteria form a boolean tree. An +internal ``CompetencyCriteriaGroup`` node combines child nodes with an ``AND``/``OR`` +``logic_operator``, can be scoped to a course run, and can nest under a parent group. A +``CompetencyCriterion`` leaf is the tree's terminal node: it points at one tag/object association +and a rule (``CompetencyRuleProfile``; only the ``Grade`` rule type is in scope today), and is a +pure function of one subsection grade and that rule. Mastery status is one of ``Demonstrated``, +``PartiallyAttempted``, or ``AttemptedNotDemonstrated``; an absent row means the learner has not +started. A leaf is atomic, so it is only ever ``Demonstrated`` or ``AttemptedNotDemonstrated``; +``PartiallyAttempted`` is a group-level state. + +The learner-facing and instructor-facing progress dashboards need per-assignment detail: for a +given competency they show which individual criteria a learner has and has not demonstrated, not +just the rolled-up group and competency verdicts. That per-criterion view is the primary driver for +storing leaf mastery rather than deriving it on the fly. A durable, append-only record of each +leaf's transitions over time is a secondary benefit, used for audit and tracing. + +Storing every leaf multiplies out at scale. A course can carry on the order of 200 leaf criteria, +so the leaf level is where the row count concentrates: the ACTIVE leaf table (learners x attempted +leaves) reaches the low billions for an Open edX instance with millions of learners. The dominant +multiplier is this per-leaf breadth (roughly 200x per course), not time. Mastery is monotonic (see +"Advance-only banking" below): a node can only advance through the small status lattice +``AttemptedNotDemonstrated`` to ``PartiallyAttempted`` to ``Demonstrated``, at most a handful of +forward steps ever. The append-only HISTORY table records only those genuine advances, so it holds a +small constant number of rows per learner and node, bounded by monotonicity rather than growing +without limit as grades are corrected or re-evaluated over time. HISTORY is therefore about the same +order of size as ACTIVE, not a multiple that grows with a learner's history. This per-leaf scale is +what this decision has to make workable. + +That scale is not, on its own, what makes a relational database struggle. A point lookup against a +billion-row table backed by the right composite index is a logarithmic-time index seek regardless +of the table's size; the dashboard reads this feature performs are exactly such point lookups. What +billions of rows makes painful is schema migrations, backups, and any non-indexed or aggregate +query. The design's job is therefore to keep rows narrow, keep every read on an index, keep the +append-only history off the hot path, and push cross-learner analytics to an analytics store rather +than the relational table. + +openedx-core also cannot read edx-platform's grade tables directly and receives grade changes only +as ``openedx-events`` events (:ref:`openedx-learning-adr-0004`). Leaf mastery is therefore recorded +from the grade carried by the event when it arrives; the relational tables here are the record of +that computation, not a cache that can be rebuilt by re-reading a grade later. + +Decision +-------- +**Store mastery at every level, each split into ACTIVE and HISTORY.** The leaf, group, and +competency levels each keep one ACTIVE row per learner and node, updated in place, so reading a +learner's current status is a direct indexed lookup rather than a scan for the most recent of many +rows. Each level also has a parallel append-only HISTORY table that records one row per genuine +status advance, for audit and point-in-time reconstruction. Because status only advances, the status +at any past time is the latest recorded advance at or before that time, so point-in-time is fully +reconstructable from HISTORY. Because status is monotonic, the number of advances per node is bounded +by the status lattice (a small constant), so HISTORY grows with learners and nodes, not with time, +and stays about the same order of size as ACTIVE. Keeping ACTIVE and HISTORY separate still pays: +ACTIVE is a single in-place current row optimized for the dashboard point lookup and is the row +per-learner concurrency is anchored on (:ref:`openedx-learning-adr-0004`), while HISTORY is +append-only and read only for audit. + +**Leaf mastery is stored per learner and criterion.** The leaf ACTIVE row keys on +``(user_id, competency_criteria_id)`` and carries the derived status plus an effective source +timestamp (the timestamp of the grade change it was computed from; see +:ref:`openedx-learning-adr-0004`). Dashboards read per-criterion status straight from this table. +The leaf HISTORY row is the append-only ledger of those changes. A leaf stores only the derived +status, not the grade value: openedx-core does not persist grades (see the non-goals), so the leaf +is a record of *what was demonstrated*, not a re-evaluable copy of the input. + +**Group and competency roll-ups read stored leaf rows.** Because leaves are now stored, a group +re-evaluates its ``logic_operator`` by reading its child leaf (and child group) ACTIVE rows +directly. The prior draft carried a per-learner "attained-set" on each group row precisely because +leaves were not stored and siblings could not be read; that mechanism is removed. Roll-up is +incremental: when a grade event changes a leaf, the recorder re-evaluates the parent group, and if +the group's status changes, its parent in turn, up to the competency root, writing only the rows +whose status actually changed. Per-group fan-out is small, so reading a group's children is a cheap +indexed range read. + +**Status semantics.** ``Demonstrated`` means the node's logic is satisfied. ``PartiallyAttempted`` +means at least one child is demonstrated but the group's logic is not yet satisfied; it is +meaningful for ``AND`` groups and mixed trees, never for a leaf and never for an ``OR`` group (one +attained child already satisfies an ``OR``). ``AttemptedNotDemonstrated`` means the node has seen at +least one relevant grade event but nothing is yet demonstrated. An absent row means not started. At +the competency level only ``Demonstrated`` and ``PartiallyAttempted`` are surfaced, consistent with +:ref:`openedx-learning-adr-0002`: a ``Demonstrated`` root surfaces as ``Demonstrated``, a +``PartiallyAttempted`` root as ``PartiallyAttempted``, and a root that has only seen unsuccessful +attempts produces no competency-level row at all. + +**The large leaf tables adopt edx-platform's persistent-grade playbook.** The leaf ACTIVE table has +the same shape and cardinality as edx-platform's ``PersistentSubsectionGrade`` (one row per learner +per graded subsection), which runs at production scale in the main database with no separate +database, no partitioning, and no sharding. This decision mirrors the techniques that make that +work, and consciously declines the ones edx-platform did not need: + + - *64-bit primary keys from the start.* The leaf ACTIVE and HISTORY tables use a 64-bit + ``BigAutoField`` primary key, chosen up front, mirroring edx-platform's + ``UnsignedBigIntAutoField`` on ``PersistentSubsectionGrade`` ("primary key will need to be + large for this table"). Changing a primary-key type on a billion-row table later is + prohibitively expensive. + - *Narrow rows; content-hash dedup for any repeated blob.* edx-platform keeps grade rows narrow + by storing only a fixed-width hash of the visible-block set on the hot row and holding the + large JSON once in a ``VisibleBlocks`` dedup table (canonical JSON, SHA-1, base64, + unique-constrained, referenced by hash rather than a surrogate id). Our leaf rows are already + narrow (user, criterion, status, timestamps), so there is no per-row blob to deduplicate + today. The rule adopted here is that if a large, mostly-repeated payload is ever added to + these rows, it must be stored once behind a content hash rather than copied per row. + - *Composite indexes derived from documented read paths.* Indexes are added per named read path, + not by guessing: ``(user_id, competency_criteria_id)`` unique for the recorder's point read + and write; ``(user_id, ...)`` for the learner dashboard; a course-scoped index for + instructor and reporting reads. New read paths add exactly the index they need. + - *In-place current row; history only where audit needs it.* The ACTIVE row is written with an + upsert (``update_or_create`` / bulk upsert). edx-platform keeps no history table on its hot + grade tables and pays for history only on the low-volume override table; this decision + deliberately does keep a leaf HISTORY table, because leaf-transition traceability is an + explicit (if secondary) goal, and isolates it from the ACTIVE path so the hot reads and + writes never touch it. The cost of that choice is called out under accepted tradeoffs. + - *Idempotency in domain logic, not a generic version column.* Correctness under duplicate and + out-of-order delivery comes from two domain rules, not an optimistic-lock column: the + monotonic banking below (a status is set once and only advances, mirroring edx-platform's + write-once ``first_attempted`` and its "only persist a rescore if it is higher") and the + effective-source-timestamp out-of-order defense in :ref:`openedx-learning-adr-0004` + (mirroring edx-platform's check of the source score's timestamp before trusting a queued + recompute). + - *Batch the read and write paths, where batching is used.* When a deployment adopts the + optional batching described in :ref:`openedx-learning-adr-0004`, the recorder reads current + statuses in bulk and writes leaf ACTIVE (bulk upsert), leaf HISTORY (``bulk_create``), and the + rolled-up group/competency rows in a small, fixed number of round-trips per batch, independent + of the number of rows in the batch. This is the same "prefetch per cohort, bulk-write" posture + edx-platform uses. Recording is correct without batching too (:ref:`openedx-learning-adr-0004`'s + monotone merge does not depend on it); batching is a throughput optimization for high-volume + deployments, not a correctness mechanism. + - *Chunk mass recompute and provide a kill switch.* The bulk recompute triggered by a + structural criteria edit is chunked by a configurable batch size and can be halted by an + operational switch, mirroring edx-platform's ``ComputeGradesSetting`` batch size and its + ``DISABLE_REGRADE_ON_POLICY_CHANGE`` switch, so a large edit cannot become an unbounded + recompute storm. + - *A dedicated database alias and router, baked in from the start.* The learner-status tables + (leaf, group, and competency ACTIVE and HISTORY) are assigned to a dedicated Django database + alias through a database router, mirroring edx-platform's courseware-history router + (``StudentModuleHistoryExtended``). The alias defaults to the main database, so a stock + deployment runs everything on one database and gains no new mandatory infrastructure, keeping + faith with :ref:`openedx-learning-adr-0004`'s no-new-mandatory-infrastructure value; a large + deployment points the alias at a separate physical database purely through settings, with no + schema change and no data migration. Building the router now, rather than deferring it, is what + makes that later split a configuration change instead of a migration on a billion-row table. + Foreign keys that would cross the alias boundary (a learner-status row referencing a + criteria-definition row, and the reference to the user) are declared without database-level + constraints, as edx-platform does for ``PersistentSubsectionGrade.user_id``, so the tables can + live in different databases; the delete protection of :ref:`openedx-learning-adr-0002` is + therefore enforced in application code, not by a database constraint. + - *No table partitioning, sharding, or read-replica routing.* These remain consciously rejected: + the edx-platform grades app operates at scale without any of them, and narrow rows plus + indexes plus dedup were sufficient. They can be revisited if a specific need is proven. + +**Advance-only banking, monotonic.** Once a node reaches ``Demonstrated`` its ACTIVE row is retained +("banked"): the recorder never automatically regresses it, not on a later downward grade correction +and not on a criteria change. This applies at every level, including the leaf. A genuine downward +grade correction does not advance the status, so it writes no HISTORY row and leaves the banked +ACTIVE status unchanged; because HISTORY records only advances, it never carries suppressed +regressions. Reversing a banked status is a separate administrative action, out of scope here. +This monotonicity is what makes out-of-order and duplicate delivery safe, since a late or replayed +event can never lower a status, and :ref:`openedx-learning-adr-0004` relies on it. + +**Retroactive criteria changes are monotonic for the learner.** A retroactive edit can newly grant +or preserve mastery, but it never silently revokes it, and it never rewrites a learner's recorded +leaf mastery downward, consistent with :ref:`openedx-learning-adr-0003`'s rule that edits apply +going forward. For STRUCTURAL edits (a leaf added to or removed from a group, an ``AND``/``OR`` +flip, or retagged content), a chunked bulk recompute runs when the edit is published: for each +affected learner it re-evaluates the new group logic against the learner's stored leaf rows and +applies only upward transitions. This recompute is self-contained: it reads the learner's stored +leaf statuses, not the grades, which is why a learner who now meets eased criteria becomes +``Demonstrated`` with no further activity required, while a tightened edit leaves already-banked +learners untouched. A RULE or threshold change (for example, raising a passing bar) cannot be +recomputed from stored state, because whether a leaf was attained depends on the grade, which this +decision deliberately does not store; such a change takes effect only through a re-emission of the +affected grades via the normal recording pipeline, or lazily, the next time the learner produces a +relevant grade event. + +**Non-goals (deferred/out of scope).** openedx-core does not store grades, so re-evaluating leaves +under a changed rule or threshold from stored state is not supported (see above). Administrative +revocation or correction of banked mastery (errata) is deferred to future work; the recorder here +is strictly advance-only. Cross-learner analytics and reporting are served by an analytics store +(ClickHouse/Aspects), not by aggregate queries over these relational tables. Because HISTORY records +only advances, its growth is already bounded by monotonicity (learners x nodes x a small constant), +so no retention or tiering policy is required to keep it workable; one may still be added later, and +the database router above lets the HISTORY tables be relocated to a separate database if a deployment +prefers. Emitting a status-change event for downstream notification is also out of scope here. + +Accepted tradeoffs +------------------ + + - The leaf tables are large by design (billions of rows at the top end), so schema migrations, + backups, and any non-indexed query on them are expensive. This is the same posture + edx-platform accepts for persistent grades; it is mitigated by the 64-bit primary key, narrow + rows, and index-only read paths, but not eliminated. + - Keeping a leaf HISTORY table is a deliberate departure from edx-platform, which keeps no + history on its hot grade tables. Because it records only status advances, it is bounded by + monotonicity to about the same order of size as the ACTIVE leaf table, rather than an unbounded + ledger that grows as grades are re-evaluated. It is justified by the audit and point-in-time + goal, isolated from the hot path, and the dedicated database alias and router above let it be + split onto its own database when a deployment needs to. + - Growth is managed structurally (narrow rows, big keys, indexes) and bounded by monotonicity + (HISTORY records only advances), not by retention or deletion. A retention or tiering policy is + therefore not required to keep the tables workable, and is left as optional future work. + - This decision re-incurs the per-leaf breadth multiplier that the earlier transient-leaf draft + avoided. That cost is accepted in exchange for the per-assignment dashboard detail that + motivates it. + +Rejected Alternatives +--------------------- + +1. Compute leaves transiently, never store them (the earlier draft of this ADR). + + Leaves are derived on the fly from the grade and the rule and never persisted; only group and + competency mastery is stored, in ACTIVE and HISTORY tables, with a per-learner "attained-set" on + each group row so an ``AND`` group can be evaluated without re-reading grades. + + - Pros: + - Removes the ~200x per-leaf breadth multiplier from stored data entirely; the hot store is + hundreds of millions of rows rather than billions. + - No large-table machinery, no separate-database question, cheaper migrations and backups. + - Lower operational burden on small and medium deployers. + - Cons: + - No stored per-criterion status, so the per-assignment dashboard detail that drives this + decision would have to be recomputed on every read, or approximated from group-level + state, rather than read directly. + - No leaf-level history for audit or tracing. + - The attained-set is a bespoke mechanism that exists only because leaves are not stored, and + it must be kept correct under retroactive edits and out-of-order delivery. + - Why rejected: the primary requirement is a live per-criterion dashboard view, which is a direct + read of stored leaf status. Deriving it transiently either moves that cost onto every read or + gives up the detail. Storing leaves also lets group roll-up read sibling rows directly and + removes the attained-set entirely. The storage cost is real but is the same cost edx-platform + already carries for persistent grades, and it is what the feature is for. + +2. Keep everything append-only (no ACTIVE table); current status is the latest row. + + - Pros: + - No in-place mutation; uniform with the original :ref:`openedx-learning-adr-0003` model. + - Cons: + - Current-status reads become "latest advance per node" queries (a group-by or ordered scan) + rather than a single-row point lookup, on the dashboard hot path. + - There is no in-place current row to anchor per-learner concurrency on, which + :ref:`openedx-learning-adr-0004` relies on. + - Why rejected: even with HISTORY bounded by monotonicity, a single in-place ACTIVE row is a + cheaper and simpler dashboard read than resolving the latest advance out of a node's history, + and it is the row per-learner concurrency in :ref:`openedx-learning-adr-0004` is anchored on. + +3. Recompute group and competency status on read instead of storing them. + + - Pros: + - No stored derived state above the leaf. + - Cons: + - Reopens :ref:`openedx-learning-adr-0002`, which materialized these levels for dashboard read + performance, and moves the aggregate cost onto every read. + - Why rejected: the read surface this feature serves is the one that decision protected; leaves + are already stored here, so group and competency roll-ups are cheap incremental writes rather + than repeated read-time aggregation. + +4. Make a separate physical database mandatory, or partition/shard the leaf tables, up front. + + - Pros: + - Isolates the large tables' write, backup, and migration load from the main database for + every deployment. + - Cons: + - Makes separate database infrastructure mandatory for every deployment that enables + competencies, in tension with :ref:`openedx-learning-adr-0004`'s value of adding no new + mandatory infrastructure, and burdens small and medium deployers with a database they may + barely use. + - Partitioning and sharding add operational complexity the edx-platform grades app never + needed at scale. + - Why rejected: the chosen design already bakes in a database router and a dedicated alias, so a + deployment that needs isolation gets it through settings with no migration, while a stock + deployment keeps a single database. Forcing a separate database or a partitioning scheme on + everyone buys nothing the router does not, at a real operational cost. Partitioning and + sharding remain available to revisit if a specific need is proven. + +5. Store a durable per-learner grade-input projection (effective grade per learner and subsection) + and derive leaves on demand. + + - Pros: + - Fully re-evaluable, including for rule/threshold changes. + - Supports leaf-level audit. + - Cons: + - Duplicates grade data into openedx-core with its own PII and retention profile. + - Is a table of the same cardinality as the leaf table, so it does not avoid the scale + question; it adds grade governance on top of it. + - Why rejected: it recreates the leaf-level cardinality and adds a grade-data governance problem, + to buy rule-change recompute that is a documented, accepted limitation of not storing grades. + +6. Store child evaluations on the parent group row instead of a leaf ACTIVE table (an enriched + attained-set). + + Keep the leaf HISTORY table but drop the leaf ACTIVE table. Each group ACTIVE row carries an + array of its direct children's frozen evaluations, one tuple per child of + ``(competency_criteria_id, status, effective_source_timestamp)``. The current per-criterion + status a dashboard shows is read from the parent group's array rather than from a leaf row, and + the leaf HISTORY table remains the append-only audit trail. This is the "attained-set" of the + transient-leaf design (alternative 1) enriched from a set of demonstrated ids to a full per-child + status snapshot, so that it can serve the per-assignment dashboard the bare attained-set could + not. + + - Pros: + - Removes the ~200x leaf multiplier from the hot current-state store: the current-state + tables are group-granular, roughly 7x fewer rows, and a group's whole child set is read in + a single row. + - The write path is slightly leaner: a leaf event appends HISTORY and rewrites the parent's + array in memory, with no leaf ACTIVE upsert and no sibling-row read. + - Preserves the frozen-evaluation, monotonic, and banking properties. + - Cons: + - Reintroduces a denormalized field that must be kept correct under out-of-order delivery and + structural edits, updated by read-modify-write. This is the bespoke-mechanism cost that + storing leaves as first-class rows was chosen to remove. + - Couples a leaf's frozen evaluation to its position in the tree: the eval lives inside the + parent group's row, so a criterion reparented by a later structural edit strands its frozen + eval in the old group's array. A first-class leaf row is keyed by the criterion, + independent of tree position, and is immune to this. + - Gives up an indexed per-criterion query; cross-learner per-leaf reads must scan arrays or go + to the analytics store. + - The dominant table, leaf HISTORY, is unchanged, so the headline migration and backup cost + is not reduced; only the hot store shrinks. + - Why rejected: structural robustness is valued over the hot-store saving. First-class leaf rows + keep a leaf's frozen mastery independent of how the criteria tree is later restructured, and + avoid re-incurring the denormalized-array correctness burden that this decision removed by + storing leaves. The hot-store reduction is real but does not address the largest table, and the + single-row group read it optimizes is served acceptably by an indexed range read of a learner's + leaf rows.