Add automated test coverage for newly added development and CDS Add Model flow#2
Merged
Conversation
added 6 commits
May 23, 2026 11:20
…s, stale baselines - Remove xfail from 8 evaluation detail tests (bugs #6/#7 now passing) - Remove xfail from 4 evaluator sidebar navigation tests (bug #8 fixed) - Remove strict xfails from test_evaluators_management + test_org_selection (XPASS) - Add xfail(strict=False) to 4 tests tracking remaining open bugs (#1, manual mode) - Broaden WIZARD_CANCEL_EVALUATION locator; scope MODAL_CANCEL_BUTTON to [role=dialog] - Use .first on NEW_EVALUATION_BUTTON / cancel clicks to avoid strict mode violations - Add is_hero_visible() / click_hamburger() aliases; wait for footer before counting social links - Add type_evaluation_name() alias on NewEvaluationPage - Fix test_header_is_sticky: use header selector directly with wait - Broaden Text Generation badge check to include uppercase variant - Update API test to accept models returned without org header (contract changed) - Mark bugs #6 and #8 as fixed in docs/app_bugs.md - Delete 6 stale visual baselines; regenerated footer + org_selector baselines
- Replace all `if not X: pytest.skip(...)` guards in e2e tests with `assert X, message` so platform failures surface as real failures rather than hiding as skips - Mark tests for open bugs #10 (cancel button missing) and #11 (manual workspace input panel / min-cases note) as xfail(strict=False) - Add tests/load/ suite covering concurrent mutations, list degradation, parallel Keycloak auth, and wizard init latency; excluded from default run via new `load` marker in pytest.ini - Add dialog auto-accept handler to authenticated_page_fast fixture - Document new bugs #10, #11, #12 in docs/app_bugs.md
…ParakhAPI ca7a0fb) API tests (Phase A+B): - Update QUERY_AUDITS + QUERY_AUDIT_TESTS for paginated wrapper shape (data/totalItemsCount) - Fix test assertions to unpack new wrapper in filtered/empty queries - Add QUERY_AUDITOR_METRICS; TestAuditorMetricsQuery (3 tests) + anon-auth test - Add TestAuditsPagination (4 tests) and TestAuditTestsPagination (1 test) E2E tests (Phase C): - test_deprecated_model.py: 3 tests for DEPRECATED version disabled Start button + toast - test_evaluations.py: TestStatusFilterTabs (6 tests), TestEvaluationsPagination (2 tests) Remove xfail from test_completed_evaluations_are_listed + test_status_badge_colors_are_distinct (Bug #7 fixed) - test_prompt_libraries.py: TestPromptLibrariesPagination (2 tests) - test_ai_maker_dashboard.py: TestAIMakerButtons (4 tests) for Add Org / Add Model dialogs - test_auditor_evaluations.py: TestAuditorDashboardMetrics (2 tests) for auditorMetrics cards - test_new_evaluation_cancel.py: TestCancelNavigation — browser Back no-loop regression test Locator and page object updates: - evaluations_locators.py: STATUS_TAB_* (6), PAGINATION_*, EVAL_COMPLETED_COL - models_locators.py: ADD_A_NEW_MODEL_BUTTON, EDIT_MODEL_BUTTON, START_EVALUATION_BUTTON_DISABLED, DEPRECATED_TOAST - ai_maker_locators.py: ADD_ORGANISATION_BUTTON, EXTERNAL_REDIRECT_DIALOG, EXTERNAL_REDIRECT_CONFIRM - prompt_libraries_locators.py: PAGINATION_CONTAINER, PASTE_TEXT_TOGGLE, UPLOAD_FILES_TOGGLE - base_page.py: fix wait_for_app_ready — give full timeout per stage (was per_stage=timeout//3) - new_evaluation_page.py: remove reload crutches; add Loading models.../modules... curtain waits; increase Loading models wait to 60s and go_to_evaluations_list to wait_for_app_ready(60s) - evaluations_page.py: click_status_tab() helper; go_to_evaluations_list wait_for_app_ready xfail cleanup (Phase D): - Remove xfail from test_back_button_removes_eval_id_from_url (Bug #6 fixed in this route) - Remove xfail from completed eval tests (Bug #7 default tab now ALL)
… hang) API regression introduced by the pagination work: - completed_eval_id fixture (conftest.py) still used the removed audits(status:) arg → switched to the filters wrapper, unpack data/totalItemsCount - FilterSpec must be [FilterSpec!] (non-null elements) — dev schema rejects [FilterSpec]; fixed in conftest.py and tests/data/test_data.py QUERY_AUDITS E2E test fixes (root-caused live via Playwright MCP): - Add Organisation button lives on the AI Maker landing (/dashboard/ai-maker), not the org dashboard (/dashboard/ai-maker/1). Added AIMakerPage.go_to_landing(); pointed the 3 add-org tests at it - test_add_new_model_button: button exists; switched non-waiting count() read to is_visible(timeout=50s) to ride out the session curtain + slow models query - test_auditor_stat_card_labels: page renders all 4 cards but overview load takes ~46s; added a 50s gate on the first stat card before counting (fixes the "found 2 of 4" race) App bug #13 (filed, NOT worked around per decision): frontend GetAudits query with the full field set (metrics+modules+test-counts+dates, limit:100) hangs >30s and never returns on dev for org 1 (981 audits) — evaluations list stuck on "Loading evaluations...". Light field set returns in ~10s; full set times out even at limit:5 → combined per-row resolver cost. The 4 TestStatusFilterTabs tests remain failing as a loud signal until backend is fixed.
… run New test files (covering 22 MCP browser findings + platform behaviour): - tests/e2e/test_breadcrumb.py — breadcrumb label bug (shows 'Dashboard' on sub-pages) - tests/e2e/test_platform_issues.py — meta description typo, 404 UX, mobile a11y, session perf - tests/e2e/test_smoke_critical.py — critical smoke suite targeting <3min runtime - tests/e2e/test_functional.py — status tabs, model cards, prompt libraries, metrics - tests/e2e/test_regression_new_features.py — new UI components (AuditResultsList, ProgressBar, etc.) - tests/api/test_db_health.py — DB connection pool exhaustion detection - tests/api/test_security.py — HTML security headers, unauthenticated session API check - tests/accessibility/test_accessibility_auth.py — axe-core WCAG checks for authenticated routes - tests/load/test_load_graphql.py — concurrent GraphQL read load scenarios - tests/performance/test_performance_auth.py — authenticated route timing + resource size budgets - scripts/generate_pdf_report.py — PDF report generator (fpdf2 + MCP findings) Updated: - locators/evaluations_locators.py — 4 new status tab locators (9-tab StatusFilterTabs) - tests/e2e/test_evaluations.py — 9-tab tests, new status tab coverage - pytest.ini — register 'security' marker - pyproject.toml — author email - CONTRIBUTING.md, README.md — document new suites and run commands
Covers every issue filed from the 2026-06-22 MCP audit session that lacked automated test coverage (546 → 604 tests, +58): New files: - locators/add_model_flow_locators.py - pages/add_model_flow_page.py - tests/e2e/test_add_model_flow.py — CDS-001–005, VALID-001, SEC-004 Extended files: - test_platform_issues.py — SEO-003, UX-015, UX-002, UX-005, UX-006, UX-013, INFRA-004, AUTH-001 - test_accessibility_auth.py — UX-014, UX-009, UX-010, UX-011 - test_security.py — SEC-002 (CSP), SEC-003 (nginx version) - utils/config.py + .env.example — CDS_URL + cds_url() for cross-platform tests All known-broken issues marked xfail; CDS tests skip if CDS_URL unreachable.
saqibmanan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New files
locators/add_model_flow_locators.py— selectors for the cross-platform Add Model flowpages/add_model_flow_page.py— page object for ParakhAI → CivicDataSpace navigationtests/e2e/test_add_model_flow.py— 12 tests: CDS-001 (JS SyntaxError), CDS-002 (ghost model), CDS-003 (misleading autosave), CDS-004 (?tab=registered ignored), CDS-005 (Quill bullet), VALID-001 (wizard step validation), SEC-004 (API key autocomplete)Extended files
test_platform_issues.py— SEO-003, UX-015, UX-002, UX-005 (×2), UX-006, UX-013, INFRA-004, AUTH-001test_accessibility_auth.py— UX-014 (dialog headings doubled), UX-009 (aria-sort), UX-010 (aria-describedby), UX-011 (unlabelled selects)test_security.py— SEC-002 (CSP header), SEC-003 (nginx version disclosure)utils/config.py+.env.example—CDS_URLfield +cds_url()classmethodAll known-broken issues are marked
xfail— they document bugs without failing the suite. CDS tests skip gracefully whenCDS_URLis unreachable.Test plan
pytest --collect-only -q→ 604 collected, 0 errorspytest tests/e2e/test_add_model_flow.py -v— CDS tests skip or xfail (no live CDS session needed for smoke)pytest tests/e2e/test_platform_issues.py tests/api/test_security.py tests/accessibility/test_accessibility_auth.py -v— new gap tests visible