fix(import): RM-17162 External llms links imported as docs pages

[xavierandueza]

🚥 Resolves RM-17162

🧰 Changes

• BEFORE: External links listed in llms.txt could be imported as docs pages, including live status pages.
• AFTER: llms.txt imports keep canonical docs origins and drop external page links before organization.

🧬 QA & Testing

• node --test src/commands/import.test.js
• npm test
• npm run step:skeleton https://cosmo-docs.wundergraph.com/overview
• Verified no status.wundergraph.com, Platform Status, or WunderGraph GitHub stubs were generated.

[RECORDING/IMAGES]

[xavierandueza]

A couple of maintainability issues before this lands. The behavior is plausible, but the implementation keeps growing the import monolith with another bespoke llms sanitation pass and tests the helper rather than the pipeline boundary.

[xavierandueza]

This adds another full section-pruning pass right next to extractOasJsonUrlsFromParsed and dropAssetItemsFromParsed, with the same mutation/empty-section bookkeeping copied a third time. That is exactly how this 4.5k-line command keeps accumulating spaghetti: every new llms cleanup gets its own bespoke loop and exported test hook.

Can we do the code-judo move here and introduce one small filterParsedItems(parsed, predicate) / dropParsedItems(parsed, predicate) primitive, then express asset/meta and external filtering through it? That deletes the duplicate mechanics, keeps the invariant for pruning empty sections in one place, and makes future llms sanitation changes much harder to implement inconsistently.

[xavierandueza]

(agent response)
Fixed — introduced dropParsedItems so OAS, asset/meta, and external filtering share the same section-pruning mechanics. 51c97cc

[xavierandueza]

This test reaches straight into the private helper, so it does not cover the new boundary that can actually regress: produceOrganizedForSource building the allowed-origin set from the source URL + merged llms source origins before organization.

Can we test the importer/discovery path instead, with a mocked llms file that includes an internal docs page and an external status page? That would verify the wiring, not just the pruning loop, and would let us avoid exporting another internal helper from __test__.

[xavierandueza]

(agent response)
Fixed — replaced the private-helper test with a produceOrganizedForSource importer-boundary test using a mocked llms.txt containing internal docs links and external status/GitHub links. 51c97cc

[xavierandueza]

(agent response)
Addressed both maintainability comments in 51c97cc: shared llms item-pruning mechanics and moved coverage to the importer boundary.

[xavierandueza]

Thermo-nuclear code-quality pass focused on RM-17162 external llms filtering.

[xavierandueza]

This exact-origin allowlist is the right shape for dropping status.wundergraph.com / GitHub / blog links, but it is too blunt for legitimate docs that intentionally span sibling origins. A single canonical llms.txt can reasonably list docs pages on api.example.com, developers.example.com, reference.example.com, or a versioned/docs product subdomain without also linking to a second llms.txt; those pages will now be silently removed because sourceOrigins only tracks origins that hosted an llms file or a redirect target.

Can we make the boundary model explicit instead of equating "not the exact llms origin" with "external"? For example: keep exact origins from fetched/redirected llms files, but also allow same-site docs-shaped origins/paths (or only drop known non-doc external origins after a docs-origin classifier). At minimum this needs a regression test for an intentional multi-origin docs set so we do not fix WunderGraph by making the importer narrower than the llms author intended.

[xavierandueza]

(agent response)
Fixed — replaced exact-origin-only filtering with an explicit llms boundary: fetched/redirected llms origins stay allowed, and same-site docs-shaped origins/paths are preserved while non-doc external origins still drop. Added a multi-origin docs regression test. 6a45308

[xavierandueza]

(agent response)
Fixed — made the llms boundary explicit: exact fetched/redirected llms origins stay allowed, and same-site docs-shaped origins/paths are preserved while status/blog/GitHub-style externals are still dropped. Added a multi-origin docs regression test. 6a45308

[xavierandueza]

One more redirect-specific issue from the review focus.

[xavierandueza]

This allowlists the redirected finalUrl origin, which is good, but it does not actually preserve the canonical origin for relative page links in a redirected llms.txt. fetchLlmsTxt still calls analyzeLlmsTxt(text, llmsUrl, options), so a request to https://docs.example.com/llms.txt that lands on https://canonical-docs.example.com/llms.txt will rewrite (/install.md) to https://docs.example.com/install.md, not the canonical origin.

Can we parse against res.url when the fetch follows a redirect, and add a regression test where a redirected llms file contains relative page links? Otherwise the filter may not drop absolute canonical-origin links, but the importer still fails to preserve the redirected canonical docs origin for the common relative-link case.

[xavierandueza]

(agent response)
Fixed — redirected llms responses are parsed against res.url, and there is now a regression test proving relative links resolve to the canonical redirected origin. 1eca659

[xavierandueza]

(agent response)
Fixed — replaced exact-origin-only filtering with an explicit llms boundary: fetched/redirected llms origins stay allowed, and same-site docs-shaped origins/paths are preserved while non-doc external origins still drop. Added a multi-origin docs regression test. 6a45308

[xavierandueza]

(agent response)
Addressed the follow-up llms boundary/redirect review in 6a45308 and 1eca659: same-site docs origins are preserved, redirected llms files parse relative links against the final URL, and both paths have regression coverage.