test: add E2E integration tests for the core user journey - #65
Merged
Conversation
The package has full type hints but shipped no PEP 561 marker, so consumers (and AI coding assistants reading type info) saw Any everywhere. decorator.py invests in ParamSpec, TypeVar, and two @overload declarations to preserve tool signatures — all of it was discarded without the marker. Add src/modelfuzz/py.typed to the wheel and sdist, add the Typing :: Typed classifier, and add mypy --strict src/modelfuzz as a CI step so type regressions fail the workflow. Fix the 10 existing mypy --strict errors: - decorator.py: annotate shield_tool, _enforce, _wrap; fix list variance on the default engine; annotate async wrappers - cli.py: annotate _make_client, _probe, _next_attack; tighten HTTP_POST_TOOL to list[dict[str, object]] Closes #56.
Two paths matter to a user: wrapping a tool with @shield_tool and running modelfuzz scan. These tests exercise both end-to-end so development never silently breaks either. Shield journey: - A safe call executes and returns the correct value - A malicious call (containing 'secret') raises ModelFuzzBlockError - The function body never runs when a violation is caught Scanner journey: - Invokes the CLI via typer.testing.CliRunner against a stubbed OpenAI client (no network calls, no API keys) - Asserts the CLI exits 0 and the output contains VULNERABLE - Asserts the stub client was actually used (probe_count > 0) Picked up by the existing pytest config and runs in CI alongside the unit tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two paths matter to a user: wrapping a tool with
@shield_tooland runningmodelfuzz scan. These tests exercise both end-to-end so development never silently breaks either.Test 1 — The Shield Journey
ModelFuzzBlockErrorTest 2 — The Scanner Journey
typer.testing.CliRunneragainst a stubbed OpenAI client (no network calls, no API keys)tool_callVULNERABLEprobe_count > 0)Picked up by the existing pytest configuration — runs in CI alongside the unit tests. 110 tests pass, ruff + mypy --strict clean.