Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 34 additions & 5 deletions docs/adversarial-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,10 +149,38 @@ removed in reverse order after success, failure, cancellation, or timeout.
## Clock control

Playwright browser time supports exact freeze, advance, skew, and resume through
its clock API. The same capability response explicitly marks server process,
scheduler, and database clocks unsupported. A WordPress runtime extension is
required to control those surfaces without faking server behavior in browser
JavaScript.
its clock API. WordPress Playground additionally supports a disposable,
opt-in server clock seam for recipe campaigns. A normalized clock transition is
declared on an adversarial action and applied immediately before that action's
action phase. Setup phases remain grouped before action execution, so a later
transition cannot advance time before an earlier action runs. This preserves
before/after expiry boundaries and is copied with
the action into replay evidence.

```json
"actions": [
{ "type": "assert-before-expiry", "clock": [{ "surface": "scheduler", "operation": "freeze", "time": 1900000000000 }] },
{ "type": "assert-after-expiry", "clock": [{ "surface": "scheduler", "operation": "advance", "milliseconds": 1000 }] }
]
```

The Playground seam exposes `wp_codebox_adversarial_clock_now()`,
`wp_codebox_adversarial_clock_timestamp()`, and
`wp_codebox_adversarial_clock_datetime()` for code paths that explicitly choose
deterministic campaign time. It supplies the scheduler's explicit due-event
timestamp. These WordPress and scheduler surfaces are **emulated**. Native PHP
`time()`, `DateTime`, WordPress `current_time()`/`current_datetime()` consumers
that do not use the seam, and database clock functions remain **unsupported**;
the adapter never claims native interception. Browser clocks remain a separate
Playwright capability; server clock action transitions intentionally exclude the
browser surface rather than claiming mixed-clock coordination.

Clock transition negotiation is backend-specific. The WordPress Playground
adapter is selected only when the active runtime reports that backend; neutral
or other backends fail closed before campaign cases begin. If a case does not
settle within the bounded abort grace after timing out, the campaign is marked
incomplete and no subsequent case is scheduled on that runtime. Runtime teardown
owns any non-cooperative execution that remains alive.

## Browser oracles

Expand Down Expand Up @@ -243,7 +271,8 @@ The following capabilities are intentionally not claimed by this change:
- exact socket framing faults require a lower-level proxy provider;
- server-side WordPress HTTP fault interception requires a WordPress extension
adapter using the generic fault contract;
- PHP/WordPress, cron, and database clock control require a WordPress extension;
- native PHP/WordPress and database clock interception are unsupported; the
WordPress scheduler seam is an explicitly emulated capability;
- WordPress-specific mutation grammars, security policies, and instrumentation
remain extension-owned;
- live vulnerable plugin/theme discovery campaigns require disposable runtime
Expand Down
40 changes: 37 additions & 3 deletions packages/cli/src/adversarial-recipe.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import {
type WorkspaceRecipeStep,
} from "@automattic/wp-codebox-core"
import { stripUndefined } from "@automattic/wp-codebox-core/internals"
import { negotiateWordPressServerClock, wordpressServerClockCleanupAction, wordpressServerClockScheduleAction, type WordPressServerClockNegotiation } from "@automattic/wp-codebox-playground"

import type { InputMountPathMapping } from "./input-mount-paths.js"
import { recipeAdversarialCapabilities } from "./recipe-validation.js"
Expand All @@ -37,6 +38,7 @@ export interface RecipeAdversarialCampaignOutput {
available: string[]
required: string[]
optional: Array<{ id: string; available: boolean }>
clock?: WordPressServerClockNegotiation
}
evidence?: Awaited<ReturnType<typeof writeAdversarialEvidenceBundle>>
}
Expand Down Expand Up @@ -79,6 +81,15 @@ export async function runRecipeAdversarialCampaigns(options: RunRecipeAdversaria
await options.runtime.createCheckpoint({ name: checkpointName, metadata: { campaignId: declaration.id, immutableBaseline: true } })
}
const templates = new Map(declaration.caseTemplates.map((template) => [template.id, template]))
const clockTransitions = declaration.corpus.flatMap(({ actions }) => actions.flatMap((action) => action.clock ?? []))
const runtimeInfo = clockTransitions.length > 0 ? await options.runtime.info() : undefined
if (clockTransitions.length > 0 && runtimeInfo?.backend !== "wordpress-playground") {
throw new Error(`Recipe adversarial campaign ${declaration.id} requires server clock transitions, but runtime backend ${runtimeInfo?.backend ?? "unknown"} does not provide the WordPress Playground clock adapter.`)
}
const clockNegotiation = clockTransitions.length > 0 ? negotiateWordPressServerClock(clockTransitions) : undefined
if (clockNegotiation && !clockNegotiation.supported) {
throw new Error(`Recipe adversarial campaign ${declaration.id} has unsupported clock transitions: ${clockNegotiation.unsupported.map(({ surface, operation, reason }) => `${surface}.${operation} (${reason})`).join(", ")}`)
}
const campaign = adversarialCampaign({
id: declaration.id,
seed: declaration.seed,
Expand All @@ -101,7 +112,17 @@ export async function runRecipeAdversarialCampaigns(options: RunRecipeAdversaria
const campaignOptions = { ...options, executions: campaignExecutions }
const execute = async (plan: AdversarialCasePlan, signal: AbortSignal) => {
if (checkpointName) await options.runtime.restoreCheckpoint!(checkpointName)
return executeRecipeAdversarialCase(declaration, templates, plan, signal, campaignOptions)
try {
return await executeRecipeAdversarialCase(declaration, templates, plan, signal, campaignOptions)
} finally {
if (plan.actions.some((action) => action.clock?.length)) {
const cleanup = wordpressServerClockCleanupAction()
const execution = await options.runtime.execute(cleanup)
campaignExecutions.push(execution)
}
// Checkpoints isolate all runtime state when they are available.
if (checkpointName) await options.runtime.restoreCheckpoint!(checkpointName)
}
}
const result = replay
? await runRecipeAdversarialReplay(campaign, declaration, templates, replay, execute, options.signal)
Expand All @@ -120,6 +141,7 @@ export async function runRecipeAdversarialCampaigns(options: RunRecipeAdversaria
available: capabilities,
required: declaration.requiredCapabilities ?? [],
optional: (declaration.optionalCapabilities ?? []).map((id) => ({ id, available: capabilities.includes(id) })),
...(clockNegotiation ? { clock: clockNegotiation } : {}),
},
})
}
Expand Down Expand Up @@ -272,8 +294,20 @@ function stableAdversarialDiagnosticMessage(message: string, caseId: string): st

function materializeCasePhases(plan: AdversarialCasePlan, templates: Map<string, WorkspaceRecipeAdversarialCampaign["caseTemplates"][number]>): Partial<Record<WorkspaceRecipeFuzzCasePhase, WorkspaceRecipeStep[]>> {
const phases: Partial<Record<WorkspaceRecipeFuzzCasePhase, WorkspaceRecipeStep[]>> = {}
for (const phase of ["setup", "action", "assert", "teardown"] as const) {
phases[phase] = plan.actions.flatMap((action) => (templates.get(action.type)?.phases[phase] ?? []).map((step) => materializeStep(step, plan, action.input)))
for (const action of plan.actions) {
const template = templates.get(action.type)
if (action.clock?.length && (template?.phases.action?.length ?? 0) === 0) {
throw new Error(`Clock transition for adversarial action ${action.type} requires at least one action-phase step.`)
}
for (const phase of ["setup", "action", "assert", "teardown"] as const) {
const steps = (template?.phases[phase] ?? []).map((step) => materializeStep(step, plan, action.input))
if (phase === "action" && action.clock?.length) {
const transition = wordpressServerClockScheduleAction(action.clock)
phases[phase] = [...(phases[phase] ?? []), { command: transition.command, args: transition.args, metadata: transition.metadata }, ...steps]
} else {
phases[phase] = [...(phases[phase] ?? []), ...steps]
}
}
}
return phases
}
Expand Down
2 changes: 1 addition & 1 deletion packages/runtime-core/src/adversarial-browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ export interface AdversarialBrowserOracleResult {
}

export interface ClockControlCapability {
surface: "runtime" | "browser" | "scheduler" | "database"
surface: "runtime" | "wordpress" | "browser" | "scheduler" | "database"
freeze: boolean
advance: boolean
skew: boolean
Expand Down
51 changes: 47 additions & 4 deletions packages/runtime-core/src/adversarial-campaign.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,22 @@ export const DIFFERENTIAL_RESULT_SCHEMA = "wp-codebox/differential-result/v1" as

export type AdversarialMutationKind = "scalar" | "structured" | "binary" | "sequence"
export type AdversarialCaseStatus = "passed" | "failed" | "error" | "timed-out" | "resource-exhausted"
export type AdversarialClockSurface = "runtime" | "wordpress" | "scheduler" | "database"
export type AdversarialClockOperation = "freeze" | "advance" | "skew" | "restore"

/** A portable, normalized clock instruction. Runtime adapters negotiate its fidelity. */
export interface AdversarialClockScheduleEntry {
surface: AdversarialClockSurface
operation: AdversarialClockOperation
time?: number
milliseconds?: number
}

export interface AdversarialAction {
type: string
input?: unknown
/** Transitions applied immediately before this action's first declared phase. */
clock?: AdversarialClockScheduleEntry[]
metadata?: Record<string, unknown>
}

Expand Down Expand Up @@ -158,6 +170,8 @@ export interface AdversarialCampaignRunnerOptions {
signal?: AbortSignal
retainNovelty?: boolean
minimize?: boolean
/** Grace to wait for a cancelled case before terminalizing the campaign. */
abortSettleGraceMs?: number
}

export interface DifferentialCell {
Expand Down Expand Up @@ -230,6 +244,11 @@ export async function runAdversarialCampaign(campaignInput: AdversarialCampaign,
const observation = observations[index] as AdversarialExecutionObservation
executed += 1
if (observation.status === "timed-out") timedOut += 1
if (observation.diagnostics?.some(({ code }) => code === "case-timeout-unsettled")) {
incomplete = true
diagnostics.push({ code: "campaign-timeout-unsettled", message: `Campaign stopped because ${plan.caseId} did not settle after cancellation. The runtime must be torn down before further work.` })
break
}
artifactBytes += (observation.artifacts ?? []).reduce((total, artifact) => total + (artifact.bytes ?? 0), 0)
if (artifactBytes > campaign.budgets.maxArtifactBytes) { incomplete = true; diagnostics.push({ code: "campaign-artifact-budget-exhausted", message: "Campaign stopped before artifact evidence exceeded its byte budget." }); break }

Expand Down Expand Up @@ -301,10 +320,19 @@ async function executeBoundedCase(campaign: AdversarialCampaign, plan: Adversari
options.signal?.addEventListener("abort", abort, { once: true })
let timer: NodeJS.Timeout | undefined
try {
return await Promise.race([
options.execute(plan, controller.signal),
new Promise<AdversarialExecutionObservation>((resolve) => { timer = setTimeout(() => { controller.abort(); resolve({ status: "timed-out", diagnostics: [{ code: "case-time-budget-exhausted", message: `Case exceeded ${campaign.budgets.maxCaseTimeMs}ms.` }] }) }, campaign.budgets.maxCaseTimeMs) }),
const execution = options.execute(plan, controller.signal)
const completed = await Promise.race([
execution.then((observation) => ({ kind: "completed" as const, observation })),
new Promise<{ kind: "timed-out" }>((resolve) => { timer = setTimeout(() => { controller.abort(); resolve({ kind: "timed-out" }) }, campaign.budgets.maxCaseTimeMs) }),
])
if (completed.kind === "completed") return completed.observation
// Do not start another case while a timed-out case can still mutate shared state.
const settled = await Promise.race([
execution.then(() => true, () => true),
new Promise<boolean>((resolve) => setTimeout(() => resolve(false), boundedInteger(options.abortSettleGraceMs, 1_000, 1, 60_000))),
])
if (!settled) return { status: "timed-out", diagnostics: [{ code: "case-timeout-unsettled", message: `Case exceeded ${campaign.budgets.maxCaseTimeMs}ms and did not settle during the abort grace period. Further cases require runtime teardown.` }], metadata: { cancellation: "runtime-teardown-required" } }
return { status: "timed-out", diagnostics: [{ code: "case-time-budget-exhausted", message: `Case exceeded ${campaign.budgets.maxCaseTimeMs}ms and settled after cancellation.` }] }
} catch (error) {
return { status: "error", diagnostics: [{ code: "case-execution-error", message: error instanceof Error ? error.message : String(error) }] }
} finally {
Expand Down Expand Up @@ -391,6 +419,21 @@ function createFinding(campaign: AdversarialCampaign, plan: AdversarialCasePlan,
}
}

export function normalizeAdversarialClockSchedule(entries: readonly AdversarialClockScheduleEntry[]): AdversarialClockScheduleEntry[] {
return entries.map((entry, index) => {
if (!entry || !["runtime", "wordpress", "scheduler", "database"].includes(entry.surface)) throw new Error(`Clock schedule entry ${index} has an invalid surface.`)
if (!entry || !["freeze", "advance", "skew", "restore"].includes(entry.operation)) throw new Error(`Clock schedule entry ${index} has an invalid operation.`)
if ((entry.operation === "freeze" || entry.operation === "skew") && (!Number.isSafeInteger(entry.time) || (entry.time as number) < 0)) throw new Error(`Clock schedule ${entry.operation} entry ${index} requires a non-negative Unix millisecond time.`)
if (entry.operation === "advance" && (!Number.isSafeInteger(entry.milliseconds) || (entry.milliseconds as number) < 0)) throw new Error(`Clock schedule advance entry ${index} requires non-negative milliseconds.`)
if (entry.operation === "restore" && (entry.time !== undefined || entry.milliseconds !== undefined)) throw new Error(`Clock schedule restore entry ${index} cannot include time values.`)
return entry.operation === "advance"
? { surface: entry.surface, operation: entry.operation, milliseconds: entry.milliseconds }
: entry.operation === "freeze" || entry.operation === "skew"
? { surface: entry.surface, operation: entry.operation, time: entry.time }
: { surface: entry.surface, operation: entry.operation }
})
}

function defaultOracleResults(observation: AdversarialExecutionObservation): AdversarialOracleResult[] {
return observation.status === "passed" ? [] : [{ oracleId: "runtime-status", failed: true, code: observation.status, message: observation.diagnostics?.[0]?.message ?? `Runtime status was ${observation.status}.` }]
}
Expand Down Expand Up @@ -418,7 +461,7 @@ function normalizeBudgets(input: Partial<AdversarialResourceBudget> | undefined)
}

function normalizeCorpusEntry(entry: AdversarialCorpusEntry): AdversarialCorpusEntry {
return stripUndefined({ ...entry, actions: entry.actions.map((action) => ({ ...action, input: cloneJsonValue(action.input) })), input: cloneJsonValue(entry.input), signals: entry.signals ? [...new Set(entry.signals)].sort() : undefined })
return stripUndefined({ ...entry, actions: entry.actions.map((action) => stripUndefined({ ...action, input: cloneJsonValue(action.input), clock: action.clock ? normalizeAdversarialClockSchedule(action.clock) : undefined })), input: cloneJsonValue(entry.input), signals: entry.signals ? [...new Set(entry.signals)].sort() : undefined })
}

function mutateActionSequence(actions: AdversarialAction[], seed: string, maximum: number): AdversarialAction[] {
Expand Down
23 changes: 22 additions & 1 deletion packages/runtime-core/src/recipe-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1412,7 +1412,7 @@ export function createWorkspaceRecipeJsonSchema(options: WorkspaceRecipeJsonSche
type: "object",
additionalProperties: false,
required: ["type"],
properties: { type: { type: "string", pattern: "^[A-Za-z0-9][A-Za-z0-9_.-]*$" }, input: {}, metadata: { $ref: "#/$defs/metadata" } },
properties: { type: { type: "string", pattern: "^[A-Za-z0-9][A-Za-z0-9_.-]*$" }, input: {}, clock: { type: "array", minItems: 1, maxItems: 32, items: { $ref: "#/$defs/adversarialClockScheduleEntry" } }, metadata: { $ref: "#/$defs/metadata" } },
},
adversarialCaseTemplate: {
type: "object",
Expand Down Expand Up @@ -1468,6 +1468,27 @@ export function createWorkspaceRecipeJsonSchema(options: WorkspaceRecipeJsonSche
metadata: { $ref: "#/$defs/metadata" },
},
},
adversarialClockScheduleEntry: {
type: "object",
additionalProperties: false,
required: ["surface", "operation"],
allOf: [{
if: { properties: { operation: { enum: ["freeze", "skew"] } }, required: ["operation"] },
then: { required: ["time"] },
}, {
if: { properties: { operation: { const: "advance" } }, required: ["operation"] },
then: { required: ["milliseconds"] },
}, {
if: { properties: { operation: { const: "restore" } }, required: ["operation"] },
then: { not: { anyOf: [{ required: ["time"] }, { required: ["milliseconds"] }] } },
}],
properties: {
surface: { enum: ["runtime", "wordpress", "scheduler", "database"] },
operation: { enum: ["freeze", "advance", "skew", "restore"] },
time: { type: "integer", minimum: 0 },
milliseconds: { type: "integer", minimum: 0 },
},
},
transportFaultModel: {
type: "object",
additionalProperties: false,
Expand Down
2 changes: 1 addition & 1 deletion packages/runtime-core/src/runtime-contracts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -368,7 +368,7 @@ export interface WorkspaceRecipeAdversarialCampaign {
seed: string
corpus: Array<{
id: string
actions: Array<{ type: string; input?: unknown; metadata?: Record<string, unknown> }>
actions: Array<{ type: string; input?: unknown; clock?: import("./adversarial-campaign.js").AdversarialClockScheduleEntry[]; metadata?: Record<string, unknown> }>
input?: unknown
signals?: string[]
metadata?: Record<string, unknown>
Expand Down
1 change: 1 addition & 0 deletions packages/runtime-playground/src/browser-clock-control.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import type { Page } from "playwright"
export const PLAYWRIGHT_CLOCK_CONTROL_CAPABILITIES: ClockControlCapabilities = clockControlCapabilities("playwright", [
{ surface: "browser", freeze: true, advance: true, skew: true, restore: true, fidelity: "exact" },
{ surface: "runtime", freeze: false, advance: false, skew: false, restore: false, fidelity: "unsupported", reason: "Browser clock control does not alter the server process clock." },
{ surface: "wordpress", freeze: false, advance: false, skew: false, restore: false, fidelity: "unsupported", reason: "Browser clock control does not alter WordPress API clock seams." },
{ surface: "scheduler", freeze: false, advance: false, skew: false, restore: false, fidelity: "unsupported", reason: "Server scheduler control requires a runtime extension." },
{ surface: "database", freeze: false, advance: false, skew: false, restore: false, fidelity: "unsupported", reason: "Browser clock control does not alter database time functions." },
])
Expand Down
Loading
Loading