fix: don't flag content-rich pages that reference reCAPTCHA as bot-blocked

[ABHA61]

Problem

detectBotBlocker (powering the LLMO "Unlock Opportunities" / bot-blocker card via GET /sites/{siteId}/bot-blocker) reported real customer sites as "blocking Adobe LLM Optimizer from accessing public pages" when they were not.

Root cause: on a 200 OK, the generic branch flagged the page as a challenge if its HTML matched any of CHALLENGE_PATTERNS.general — including the bare /captcha/i and /recaptcha/i. Those match the ubiquitous "This site is protected by reCAPTCHA" disclosure badge / reCAPTCHA form widget, which appears on perfectly normal content pages. Result: { crawlable: false, type: 'unknown', confidence: 0.7, reason: 'Generic challenge patterns detected' }.

Evidence it's a false positive

Confirmed on the affected customer sites by running the shipped function against the live pages and cross-checking the content scraper's own results:

• The live bot-blocker check returned crawlable:false @0.7 ("Generic challenge patterns detected").
• The only matched token was the benign badge (e.g. a recaptcha-text template / "protected by reCAPTCHA" notice) on 100KB+ content homepages.
• The Puppeteer content scraper (which executes JS and actually fetches the pages) accessed the same sites fine — scrapeForbidden:false, thousands of URLs scraped with zero forbidden.

So the heuristic disagreed with the empirical scrape: nothing was actually being blocked.

Fix

Gate the generic-pattern check on an interstitial-shape heuristic (isLikelyInterstitial): on a 200, a challenge pattern only counts as a block when the page's visible text is content-thin. Visible text is measured by character length (not word count), so the check is not biased against languages that do not delimit words with spaces (CJK, Thai, etc.). The challenge pattern is evaluated first, so the content-shape strip only runs when a pattern actually matched.

• ✅ Content-rich page + reCAPTCHA badge → crawlable:true
• ✅ Thin challenge wall (reCAPTCHA / Press-and-Hold / GeeTest / Arkose / etc.) → still crawlable:false
• ✅ CDN-typed blocks (Cloudflare/Imperva/Akamai/Fastly/CloudFront) and 403/error-path detection → unchanged

Validation

• Ran the real detectBotBlocker against 43 onboarded domains, current vs. fixed: it flipped only the known false positives to crawlable:true and changed nothing else — high-confidence akamai @0.99 blocks and non-captcha 0.7 blocks were untouched (no over-correction, no OK→BLOCK).
• Regression tests added: a content-rich page (including a non-Latin / CJK page) that references reCAPTCHA → crawlable:true; a thin captcha interstitial → still crawlable:false. All existing challenge tests (thin HTML) still pass.
• npm test (lint + tests + c8 coverage gate, 100%) passes; bot-blocker-detect.js at 100%.

Notes & limitations

• Scope: only the generic 200 branch is gated. The CDN-specific challenge patterns (Just a moment…, Access Denied…Akamai, etc.) are precise/low-FP and are intentionally left ungated.
• Known limit: this is a non-JS GET + regex. A JS-rendered SPA whose raw homepage shell is content-thin and references reCAPTCHA could still be flagged.

🤖 Generated with Claude Code

[github-actions]

This PR will trigger a patch release when merged.

[tkotthakota-adobe]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[tkotthakota-adobe]

A couple of non-blocking observations from a review pass. Neither is a confirmed defect; sharing for consideration.

[tkotthakota-adobe]

The && isLikelyInterstitial(html) thinness gate is applied to the entire CHALLENGE_PATTERNS.general set, not just the captcha/recaptcha/hcaptcha badges that caused the false positive. general also includes vendor-specific tokens (datadome, geetest, arkose, funcaptcha, press/click hold). Those vendors sometimes render the challenge inside a content-rich templated shell (nav, footer, cookie banner) that can exceed 200 visible chars — in which case a genuinely-blocked page would now flip to crawlable: true (a false negative). Historically this file has fixed FPs by narrowing the specific offending regex (e.g. #1649 tightening press.*hold) rather than adding a broad gate. Consider scoping the thinness gate to only the captcha-family patterns, or adding a regression test proving the vendor patterns still fire on content-rich pages. Not confirmed in practice (43-domain validation covered the FP direction, not this FN direction).

[tkotthakota-adobe]

Minor: this comment states the check "only runs against a site's baseURL, i.e. a homepage," but the same analyzeResponse 200-branch is also reachable via the exported analyzeBotProtection, whose JSDoc says it is "Used by content scraper to analyze Puppeteer results" — i.e. per scraped URL, not just the homepage. No functional bug today (the scraper path only treats specific high-confidence CDN types as blocked), but the safety justification here is scoped to homepages and could mislead a future maintainer. Worth correcting the scope note.