fix(seo): serve real 404, use clean URLs in sitemap, canonicals and links

[nelsonduarte]

Problem

Google Search Console reports 6 pages indexed / 7 not indexed. The audit traced this to two root causes:

1. Global soft-404. docs/404.html did not exist. Cloudflare Pages therefore fell through to index.html with HTTP 200 for any nonexistent path. Every typo, stale link and crawler probe returned a full copy of the home page with a 200, i.e. an infinite surface of duplicates. This is what produced the "alternate page with proper canonical" report.

2. Sitemap and canonicals pointed at redirecting URLs. sitemap.xml and every <link rel="canonical"> / og:url declared .html URLs, which Pages 308-redirects to their clean form. Google classified 4 of them as "page with redirect" and left them out of the index. /privacy was missing from the sitemap entirely.

Fix

• New docs/404.html — same header/footer/typography as the rest of the site (styles.css), <meta name="robots" content="noindex, follow">, no canonical (a self-referential one would be misleading; one pointing at / would make every 404 a duplicate of the home page). Cloudflare Pages serves a root 404.html with a real HTTP 404, which is what eliminates the soft-404. Assets use root-absolute paths so the page renders correctly when served for deep unknown paths like /a/b/c.
• docs/sitemap.xml rewritten to clean URLs + added the missing /privacy. lastmod added. The 404 is deliberately not listed.
• Canonicals + og:url on all five subpages moved to clean URLs. index.html was already correct.
• JSON-LD downloadUrl → https://pdf-apps.com/download (the only .html URL in the block; url was already clean).
• 98 internal hrefs converted from .html to root-absolute clean URLs across the 6 pages, plus the new 404.

Coupled release.yml change (this is the load-bearing part)

release.yml bumps docs/changelog.html via a regex anchored to the footer changelog link, required=True. The anchoring is deliberate: a global v{old} rewrite previously clobbered the topmost historical <span class="version-tag"> heading on every release (that is how v1.13.16 "disappeared").

Rewriting the footer link to /changelog would have made the old anchor stop matching — and because it is required=True, the next release would have failed. Proven, not assumed:

OLD regex matches: False   <- required=True would FAIL the release
NEW regex matches: True

The anchor now accepts both forms and stays anchored (never global):

rf'(<a href="(?:/changelog|changelog\.html)">)v{old_re}(</a>)'

tests/test_release_bump_script.py's fake_repo fixture was updated to the new link shape (this fixture has broken CI before), plus two new regression tests: one asserting the footer link is bumped while the historical heading survives, one asserting the real docs/changelog.html still matches the real workflow regex at the current APP_VERSION.

Validation

Bump dry-run (OLD=1.13.17 → NEW=1.14.2, executed against a full copy of the repo, real files untouched):

• exit code 0 — 18 targets updated, no required=True target left unmatched
• 20 historical changelog headings byte-identical before/after
• footer link bumped on all 6 pages; 4 marketing pages' eyebrow/footer bumped; the only surviving v1.13.17 is the historical heading (correct)

Other checks:

• HTML well-formedness (parser, tag-balance) on all 7 pages — pass
• All internal href/src resolve to existing files; zero .html internal links remain; in-page anchors resolve to real ids
• Sitemap ↔ canonical ↔ og:url coherence verified programmatically
• pytest -q: 478 passed, 2 skipped
• ruff check on the touched Python file: clean
• release.yml parses as valid YAML

Notes

• The 404 footer deliberately carries no v<version> string: 404.html is not a bump target, so it would go stale after every release.
• The remaining "404" and "crawled – not indexed" entries in Search Console are historical/pre-migration and should resolve on the next crawl once the sitemap is resubmitted.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying pdfapps with    Cloudflare Pages

Latest commit:	308e955
Status:	✅  Deploy successful!
Preview URL:	https://99699a48.pdfapps.pages.dev
Branch Preview URL:	https://fix-seo-clean-urls-404.pdfapps.pages.dev

View logs