Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,9 @@ ESM TypeScript project (`type: module`). Key layers:


## Rendering and Streaming Contract
- Streaming fragments are transient output only. They MUST NOT be used as internal state, cached for final responses, or promoted into final MCP/JSON/CLI text output.
- Non-streaming runtimes/output modes, including MCP final responses, MUST render only from the final structured result and next-step metadata. If final output needs data, add it to the final result type instead of reading it from fragments.
- Only streaming-capable renderers may observe fragment callbacks, and only to print live progress. Their fragment handling must not affect final structured output or final rendered text.
- Streaming fragments are transient live-progress output only. They may be displayed while a tool is running, but MUST NOT provide final settled MCP/JSON/CLI text.
- Final settled output MUST render from the final structured/domain result and next-step metadata. If final output needs data, add it to the final result type instead of reading it from fragments.
- Streaming-capable renderers may observe fragment callbacks only for live progress. Fragment handling must not affect final structured output or final settled text.

## Test Conventions
- Vitest with colocated `__tests__/` directories using `*.test.ts`
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
- Fixed CLI test summaries showing false-positive compiler errors from xcodebuild NSError dump lines, and added compiler-error snapshot coverage for simulator, device, and macOS build-style flows ([#383](https://github.com/getsentry/XcodeBuildMCP/issues/383)).
- Fixed simulator OSLog helper cleanup so server and daemon startup reconcile same-workspace orphaned log streams without stopping helpers owned by live sessions in other workspaces ([#382](https://github.com/getsentry/XcodeBuildMCP/issues/382)).
- Fixed Weather example test discovery and made CLI test progress visible while tests are running instead of leaving the last build phase displayed.
- Exposed xcresult bundle paths in test result structured output and text output when xcodebuild reports or is given a result bundle path, so agents can inspect test artifacts after simulator, device, and macOS test runs ([#392](https://github.com/getsentry/XcodeBuildMCP/issues/392)).
- Fixed final test summaries to use xcresult top-level test declaration counts when available, avoiding overcounting dynamic-parameter test runs ([#392](https://github.com/getsentry/XcodeBuildMCP/issues/392)).

### Changed

Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,9 @@ Use these sections under `## [Unreleased]`:


## Rendering and Streaming Contract
- Streaming fragments are transient output only. They MUST NOT be used as internal state, cached for final responses, or promoted into final MCP/JSON/CLI text output.
- Non-streaming runtimes/output modes, including MCP final responses, MUST render only from the final structured result and next-step metadata. If final output needs data, add it to the final result type instead of reading it from fragments.
- Only streaming-capable renderers may observe fragment callbacks, and only to print live progress. Their fragment handling must not affect final structured output or final rendered text.
- Streaming fragments are transient live-progress output only. They may be displayed while a tool is running, but MUST NOT provide final settled MCP/JSON/CLI text.
- Final settled output MUST render from the final structured/domain result and next-step metadata. If final output needs data, add it to the final result type instead of reading it from fragments.
- Streaming-capable renderers may observe fragment callbacks only for live progress. Fragment handling must not affect final structured output or final settled text.

## Test Execution Rules
- When running long test suites (snapshot tests, smoke tests), ALWAYS write full output to a log file and read it afterwards. NEVER pipe through `tail` or `grep` directly — that loses output you may need to debug failures.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@
},
"packagePath": {
"type": "string"
},
"xcresultPath": {
"type": "string"
}
},
"required": [],
Expand Down
28 changes: 28 additions & 0 deletions src/mcp/tools/device/__tests__/test_device.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,34 @@ describe('test_device plugin', () => {
expect(result.isError()).toBeFalsy();
});

it('should expose user-provided result bundle paths in test output', async () => {
const mockExecutor = createMockExecutor({
success: true,
output: 'Test Succeeded',
});

const { result } = await runTestDeviceLogic(
{
projectPath: '/path/to/project.xcodeproj',
scheme: 'MyScheme',
deviceId: 'test-device-123',
configuration: 'Debug',
extraArgs: [
'-resultBundlePath',
'/tmp/Stale Device Tests.xcresult',
'-resultBundlePath=/tmp/Device Tests.xcresult',
],
preferXcodebuild: false,
platform: 'iOS',
},
mockExecutor,
mockFs(),
);

expectPendingBuildResponse(result);
expect(result.text()).toContain('Result Bundle: /tmp/Device Tests.xcresult');
});

it('should handle workspace testing successfully', async () => {
const mockExecutor = createMockExecutor({
success: true,
Expand Down
18 changes: 18 additions & 0 deletions src/mcp/tools/swift-package/__tests__/swift_package_test.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,24 @@ describe('swift_package_test plugin', () => {
expect(result.isError()).toBeFalsy();
});

it('should not expose xcresult paths from SwiftPM test output', async () => {
const mockExecutor: CommandExecutor = async (_args, _name, _hideOutput, opts) => {
opts?.onStdout?.('Result bundle written to: /tmp/SwiftPM.xcresult\n');
return createMockCommandResponse({
success: true,
output: 'All tests passed.',
});
};

const { result } = await runSwiftPackageTestLogic(
{ packagePath: '/test/package' },
mockExecutor,
);

expect(result.isError()).toBeFalsy();
expect(result.text()).not.toContain('Result Bundle: /tmp/SwiftPM.xcresult');
});

it('should return error response for test failure', async () => {
const mockExecutor = createMockExecutor({
success: false,
Expand Down
55 changes: 54 additions & 1 deletion src/rendering/__tests__/text-render-parity.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,10 @@ describe('text render parity', () => {
durationMs: 2200,
counts: { passed: 1, failed: 1, skipped: 0 },
},
artifacts: { buildLogPath: '/tmp/Test.log' },
artifacts: {
buildLogPath: '/tmp/Test.log',
xcresultPath: '/tmp/App Tests.xcresult',
},
diagnostics: {
warnings: [],
errors: [],
Expand Down Expand Up @@ -284,9 +287,59 @@ describe('text render parity', () => {
expect(output.match(/Discovered 2 test\(s\):/g)).toHaveLength(1);
expect(output.match(/MCPTestTests\n ✗ testTwo\(\):/g)).toHaveLength(1);
expect(output.match(/1 test failed, 1 passed, 0 skipped/g)).toHaveLength(1);
expect(output).toContain('Result Bundle: /tmp/App Tests.xcresult');
expect(output).toContain('Build Logs: /tmp/Test.log');
});

it('matches cli text and uses structured build summary when streamed build-summary disagrees', () => {
const fixture: TranscriptFixture = {
progressEvents: [
{
kind: 'build-result',
fragment: 'invocation',
operation: 'BUILD',
request: {
scheme: 'MyApp',
projectPath: '/tmp/MyApp.xcodeproj',
configuration: 'Debug',
platform: 'iOS Simulator',
},
},
{
kind: 'build-result',
fragment: 'build-summary',
operation: 'BUILD',
status: 'FAILED',
durationMs: 9900,
},
],
structuredOutput: {
schema: 'xcodebuildmcp.output.build-result',
schemaVersion: '1.0.0',
result: {
kind: 'build-result',
didError: false,
error: null,
summary: { status: 'SUCCEEDED', durationMs: 3200 },
artifacts: { scheme: 'MyApp', buildLogPath: '/tmp/build.log' },
diagnostics: { warnings: [], errors: [] },
},
},
};

const rendered = renderTranscript(
{
items: fixture.progressEvents,
structuredOutput: fixture.structuredOutput,
},
'text',
);

expect(rendered).toBe(captureCliText(fixture));
expect(rendered).toContain('✅ Build succeeded. (⏱️ 3.2s)');
expect(rendered).not.toContain('❌ Build failed. (⏱️ 9.9s)');
});

it('renders next steps in MCP tool-call syntax for MCP runtime text transcripts', () => {
const fixture: TranscriptFixture = {
progressEvents: [],
Expand Down
18 changes: 18 additions & 0 deletions src/snapshot-tests/__tests__/json-normalize.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,24 @@ describe('normalizeStructuredEnvelope', () => {
});
});

it('preserves xcresult paths in test result artifacts', () => {
const envelope: StructuredOutputEnvelope<unknown> = {
schema: 'xcodebuildmcp.output.test-result',
schemaVersion: '1',
didError: false,
error: null,
data: {
summary: { target: 'simulator' },
artifacts: {
buildLogPath: '/tmp/build.log',
xcresultPath: '/tmp/App Tests.xcresult',
Comment thread
sentry-warden[bot] marked this conversation as resolved.
},
},
};

expect(normalizeStructuredEnvelope(envelope)).toEqual(envelope);
});

it('keeps suite-less passed test cases for non-simulator results', () => {
const envelope: StructuredOutputEnvelope<unknown> = {
schema: 'xcodebuildmcp.output.test-result',
Expand Down
1 change: 1 addition & 0 deletions src/types/domain-results.ts
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ export type TestResultArtifacts = AtLeastOne<{
deviceId: string;
buildLogPath: string;
packagePath: string;
xcresultPath: string;
}>;
export interface CoverageSummary extends StatusSummary {
coveragePct?: number;
Expand Down
28 changes: 28 additions & 0 deletions src/utils/__tests__/simulator-test-execution.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ describe('createSimulatorTwoPhaseExecutionPlan', () => {
'/tmp/Calculator.xcresult',
]);
expect(plan.usesExactSelectors).toBe(false);
expect(plan.resultBundlePath).toBe('/tmp/Calculator.xcresult');
});

it('preserves user-supplied selector arguments in both simulator test phases', () => {
Expand Down Expand Up @@ -90,5 +91,32 @@ describe('createSimulatorTwoPhaseExecutionPlan', () => {

expect(plan.buildArgs).toEqual([]);
expect(plan.testArgs).toEqual(['-resultBundlePath', '/tmp/UserProvided.xcresult']);
expect(plan.resultBundlePath).toBe('/tmp/UserProvided.xcresult');
});

it('supports equals-form resultBundlePath arguments', () => {
const plan = createSimulatorTwoPhaseExecutionPlan({
extraArgs: ['-resultBundlePath=/tmp/EqualsProvided.xcresult'],
});

expect(plan.buildArgs).toEqual([]);
expect(plan.testArgs).toEqual(['-resultBundlePath', '/tmp/EqualsProvided.xcresult']);
expect(plan.resultBundlePath).toBe('/tmp/EqualsProvided.xcresult');
});

it('uses the last valid resultBundlePath argument', () => {
const plan = createSimulatorTwoPhaseExecutionPlan({
extraArgs: [
'-resultBundlePath',
'-quiet',
'-resultBundlePath',
'/tmp/First.xcresult',
'-resultBundlePath=/tmp/Last.xcresult',
],
});

expect(plan.buildArgs).toEqual(['-quiet']);
expect(plan.testArgs).toEqual(['-quiet', '-resultBundlePath', '/tmp/Last.xcresult']);
expect(plan.resultBundlePath).toBe('/tmp/Last.xcresult');
});
});
112 changes: 108 additions & 4 deletions src/utils/__tests__/xcodebuild-domain-results.test.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,21 @@
import { describe, expect, it } from 'vitest';
import { createBuildDomainResult } from '../xcodebuild-domain-results.ts';
import { beforeEach, describe, expect, it, vi } from 'vitest';

const mocks = vi.hoisted(() => ({
extractTestSummaryCountsFromXcresult: vi.fn(),
}));

vi.mock('../xcresult-test-failures.ts', () => ({
extractTestSummaryCountsFromXcresult: mocks.extractTestSummaryCountsFromXcresult,
}));

import { createBuildDomainResult, createTestDomainResult } from '../xcodebuild-domain-results.ts';
import { createXcodebuildRunState, type XcodebuildRunState } from '../xcodebuild-run-state.ts';
import type { StartedPipeline, XcodebuildPipeline } from '../xcodebuild-pipeline.ts';

function createStartedPipelineWithState(state: XcodebuildRunState): StartedPipeline {
function createStartedPipelineWithState(
state: XcodebuildRunState,
xcresultPath: string | null = null,
): StartedPipeline {
const pipeline: XcodebuildPipeline = {
onStdout(): void {},
onStderr(): void {},
Expand All @@ -14,14 +26,106 @@ function createStartedPipelineWithState(state: XcodebuildRunState): StartedPipel
highestStageRank() {
return 0;
},
xcresultPath: null,
xcresultPath,
logPath: '/tmp/build.log',
};

return { pipeline, startedAt: Date.now() };
}

describe('xcodebuild-domain-results', () => {
beforeEach(() => {
mocks.extractTestSummaryCountsFromXcresult.mockReturnValue(null);
});

it('includes detected xcresult paths in test result artifacts', () => {
const runState = createXcodebuildRunState({ operation: 'TEST' });

const result = createTestDomainResult({
started: createStartedPipelineWithState(
runState.finalize(true, 1000),
'/tmp/App Tests.xcresult',
),
succeeded: true,
target: 'simulator',
artifacts: { buildLogPath: '/tmp/build.log' },
request: { scheme: 'App' },
});

expect(result.artifacts).toMatchObject({
buildLogPath: '/tmp/build.log',
xcresultPath: '/tmp/App Tests.xcresult',
});
});

it('does not copy parser-detected xcresult paths into SwiftPM test results', () => {
const runState = createXcodebuildRunState({ operation: 'TEST' });

const result = createTestDomainResult({
started: createStartedPipelineWithState(
runState.finalize(true, 1000),
'/tmp/NotFromSwiftPM.xcresult',
),
succeeded: true,
target: 'swift-package',
artifacts: { buildLogPath: '/tmp/build.log' },
request: { target: 'swift-package', packagePath: '/tmp/Package' },
});

expect(result.artifacts).toEqual({ buildLogPath: '/tmp/build.log' });
});

it('preserves provided xcresult paths when the pipeline does not detect one', () => {
const runState = createXcodebuildRunState({ operation: 'TEST' });

const result = createTestDomainResult({
started: createStartedPipelineWithState(runState.finalize(true, 1000)),
succeeded: true,
target: 'macos',
artifacts: {
buildLogPath: '/tmp/build.log',
xcresultPath: '/tmp/User Provided.xcresult',
},
request: { scheme: 'App' },
});

expect(result.artifacts.xcresultPath).toBe('/tmp/User Provided.xcresult');
});

it('uses xcresult top-level declaration counts instead of streamed run counts', () => {
mocks.extractTestSummaryCountsFromXcresult.mockReturnValue({
passed: 16,
failed: 0,
skipped: 0,
});

const runState = createXcodebuildRunState({ operation: 'TEST' });
runState.push({
kind: 'test-result',
fragment: 'test-progress',
operation: 'TEST',
completed: 19,
failed: 0,
skipped: 0,
});

const result = createTestDomainResult({
started: createStartedPipelineWithState(
runState.finalize(true, 1000),
'/tmp/Weather.xcresult',
),
succeeded: true,
target: 'simulator',
artifacts: { buildLogPath: '/tmp/build.log' },
request: { scheme: 'Weather' },
});

expect(mocks.extractTestSummaryCountsFromXcresult).toHaveBeenCalledWith(
'/tmp/Weather.xcresult',
);
expect(result.summary.counts).toEqual({ passed: 16, failed: 0, skipped: 0 });
});

it('does not duplicate fallback lines represented by multi-line parsed errors', () => {
const runState = createXcodebuildRunState({ operation: 'BUILD' });
runState.push({
Expand Down
Loading
Loading