From 6a8b6257e267239789b3e0f095365a6ba3cc9ac7 Mon Sep 17 00:00:00 2001 From: "google-labs-jules[bot]" <161369871+google-labs-jules[bot]@users.noreply.github.com> Date: Wed, 8 Jul 2026 17:19:41 +0000 Subject: [PATCH 1/4] feat: submodule sync and per-test watchdog Implemented two major improvements to devflow: 1. Per-test stall watchdog for `gotest`: - Replaces native per-package cumulative timeout semantics with per-test stall detection. - Injects `-v` to monitor test lifecycle events (`=== RUN`, `=== PAUSE`, `=== CONT`). - Uses a 10x backstop for overall package hangs. - Added `watchdog.go` and `test/watchdog_test.go`. 2. Same-repo submodule handling for `gopush`: - Automatically detects and syncs submodules inside the repository that depend on the parent. - Ensures relative `replace` directives pointing to the local parent. - Bumps parent requirement to the upcoming release tag pre-commit. - Excludes internal submodules from external dependent update flow. - Added `go_selfdep.go` and `test/go_selfdep_test.go`. Updated `docs/GOTEST.md`, `docs/GOPUSH.md`, and `docs/CODEJOB.md` to reflect these changes. Repository-heavy tests are now tagged with `//go:build integration`. Co-authored-by: cdvelop <44058491+cdvelop@users.noreply.github.com> --- docs/CODEJOB.md | 2 +- docs/GOPUSH.md | 11 ++- docs/GOTEST.md | 15 ++-- go_handler.go | 22 ++++- go_mod.go | 129 ++++++++++++++++++++++------ go_selfdep.go | 77 +++++++++++++++++ gotest.go | 127 +++++++++++++++++++++++----- interface.go | 1 + test/go_handler_test.go | 4 + test/go_selfdep_test.go | 103 +++++++++++++++++++++++ test/gonew_test.go | 2 + test/watchdog_test.go | 182 ++++++++++++++++++++++++++++++++++++++++ watchdog.go | 129 ++++++++++++++++++++++++++++ 13 files changed, 744 insertions(+), 60 deletions(-) create mode 100644 go_selfdep.go create mode 100644 test/go_selfdep_test.go create mode 100644 test/watchdog_test.go create mode 100644 watchdog.go diff --git a/docs/CODEJOB.md b/docs/CODEJOB.md index fa3bfc1..7ae2322 100644 --- a/docs/CODEJOB.md +++ b/docs/CODEJOB.md @@ -101,7 +101,7 @@ job := devflow.NewCodeJob(devflow.NewJulesDriver(cfg)) ## Integrated Flow (via gopush) -`codejob` uses `gopush` to sync changes before dispatch and to publish changes when "closing the loop". +`codejob` uses `gopush` to sync changes before dispatch and to publish changes when "closing the loop". It inherits all `gopush` behavior, including **automatic internal submodule syncing** when publishing a release. ## State Check & Cleanup diff --git a/docs/GOPUSH.md b/docs/GOPUSH.md index f75e09b..146c261 100644 --- a/docs/GOPUSH.md +++ b/docs/GOPUSH.md @@ -20,9 +20,14 @@ gopush 'commit message' [tag] 1. Verifies `go.mod` 2. Runs `gotest` (vet, tests, race, coverage, badges) -3. Commits changes with your message -4. Creates/uses tag -5. Intelligent push: Pushes to remote (auto-pulls/rebases if remote is ahead). +3. **Internal submodules sync**: Any submodule inside the repo that depends on the parent module is automatically updated: + - Ensures a relative `replace` points to the local parent. + - Bumps the parent requirement to the next tag. + - Runs `go mod tidy`. + These changes are included in the same release commit. +4. Commits changes with your message +5. Creates/uses tag +6. Intelligent push: Pushes to remote (auto-pulls/rebases if remote is ahead). 6. Automatically installs binaries with version tag (if `cmd/` exists) 7. Finds dependent modules in search path 8. For each dependent (in parallel): diff --git a/docs/GOTEST.md b/docs/GOTEST.md index 3543f46..7d04480 100644 --- a/docs/GOTEST.md +++ b/docs/GOTEST.md @@ -125,22 +125,25 @@ Shows only failed tests with error details, filters out passing tests. Failed ru ## Timeout -By default, each package has a **30-second** timeout. If a test hangs or takes too long, Go's test framework kills the package and `gotest` reports the offending test: +`gotest` uses a **per-test stall watchdog** (default 30s). Unlike Go's native cumulative package timeout, `gotest` kills the process only if a *single* test makes no progress for the specified duration. +If a test stalls: ``` -❌ timeout: TestSlowOperation (exceeded 30s) +❌ timeout: TestSlowOperation stalled >30s (no progress) ``` Override with `-t`: ```bash -gotest -t 120 # 120s per package +gotest -t 120 # 120s per test stall ``` -If you pass `-timeout` directly (Go's native flag), `gotest` respects it and does not inject its own: -```bash -gotest -timeout 2m # Go-native flag, gotest won't override +A **backstop timeout** (10x the watchdog limit) is also injected into `go test -timeout` to catch genuine package-level hangs where the watchdog might be bypassed. If hit, it reports: +``` +❌ timeout: package exceeded 300s total (backstop) ``` +If you pass `-timeout` directly (Go's native flag), `gotest` still injects its watchdog using that value (or its default if not parseable), but respects your explicit package timeout. + ## Notes - Supports all standard `go test` flags diff --git a/go_handler.go b/go_handler.go index 0190ffd..580979e 100644 --- a/go_handler.go +++ b/go_handler.go @@ -199,10 +199,12 @@ func (g *Go) Push(message, tag string, skipTests, skipRace, skipDependents, skip summary = append(summary, "Tests skipped") } - // 3. Execute git push workflow + // 3. Prepare internal submodules and execute git push workflow var pushResult PushResult var err error + modulePath, _ := g.GetModulePath() + if skipTag { if err := g.git.Add(); err != nil { return PushResult{}, fmt.Errorf("git add failed: %w", err) @@ -223,6 +225,22 @@ func (g *Go) Push(message, tag string, skipTests, skipRace, skipDependents, skip pushResult.Summary = "No changes to commit" } } else { + // Hoist tag computation so we can sync internal submodules BEFORE commit + nextTag := tag + if nextTag == "" && g.git != nil { + var err error + nextTag, err = g.git.GenerateNextTag() + if err != nil { + g.log("Warning: could not generate next tag for submodule sync:", err) + } + } + + if nextTag != "" && modulePath != "" { + if err := g.syncInternalSubmodules(modulePath, nextTag); err != nil { + g.log("Warning: failed to sync internal submodules:", err) + } + } + pushResult, err = g.git.Push(message, tag) if err != nil { return PushResult{}, fmt.Errorf("push workflow failed: %w", err) @@ -241,7 +259,7 @@ func (g *Go) Push(message, tag string, skipTests, skipRace, skipDependents, skip } // 5. Get module name - modulePath, err := g.GetModulePath() + modulePath, err = g.GetModulePath() if err != nil { summary = append(summary, fmt.Sprintf("Warning: could not get module path: %v", err)) return PushResult{Summary: strings.Join(summary, ", "), Tag: createdTag}, nil diff --git a/go_mod.go b/go_mod.go index 51a6a2a..390f69a 100644 --- a/go_mod.go +++ b/go_mod.go @@ -93,13 +93,17 @@ func (m *GoModHandler) RemoveReplace(modulePath string) bool { } // Check for the module in replace - if (strings.HasPrefix(trimmed, "replace ") || inReplaceBlock) && strings.Contains(trimmed, modulePath) { - if isLocalReplaceTarget(trimmed) { - newLines = append(newLines, line) - continue + if (strings.HasPrefix(trimmed, "replace ") || inReplaceBlock) { + // Extract module path from line + mod, _ := parseReplaceLine(trimmed, inReplaceBlock) + if mod == modulePath { + if isLocalReplaceTarget(trimmed) { + newLines = append(newLines, line) + continue + } + removed = true + continue // skip this line } - removed = true - continue // skip this line } newLines = append(newLines, line) @@ -114,6 +118,23 @@ func (m *GoModHandler) RemoveReplace(modulePath string) bool { return false } +func parseReplaceLine(line string, inBlock bool) (modPath, targetPath string) { + s := line + if !inBlock { + s = strings.TrimPrefix(s, "replace ") + } + parts := strings.Split(s, "=>") + if len(parts) != 2 { + return "", "" + } + modPath = strings.TrimSpace(parts[0]) + targetPath = strings.TrimSpace(parts[1]) + if idx := strings.Index(targetPath, "//"); idx != -1 { + targetPath = strings.TrimSpace(targetPath[:idx]) + } + return modPath, targetPath +} + // isLocalReplaceTarget reports whether a replace directive line's target // (the part after "=>") is a self-reference to the current directory (e.g. // "=> ./"). Subpackages (tests/, cmd/, etc.) commonly declare their own @@ -162,26 +183,11 @@ func (m *GoModHandler) GetReplacePaths() ([]ReplaceEntry, error) { } if strings.HasPrefix(trimmed, "replace ") || inReplaceBlock { - // Extract part after "replace " if not in block - lineContent := trimmed - if !inReplaceBlock { - lineContent = strings.TrimPrefix(trimmed, "replace ") - } - - // Format is usually: module => path - parts := strings.Split(lineContent, "=>") - if len(parts) != 2 { + modPath, localPath := parseReplaceLine(trimmed, inReplaceBlock) + if modPath == "" || localPath == "" { continue } - modPath := strings.TrimSpace(parts[0]) - localPath := strings.TrimSpace(parts[1]) - - // Clean up comments if any - if idx := strings.Index(localPath, "//"); idx != -1 { - localPath = strings.TrimSpace(localPath[:idx]) - } - // Check if localPath is actually a local path or a versioned module. // Local paths in go.mod MUST start with ./ or ../ or be absolute. isLocal := strings.HasPrefix(localPath, ".") || strings.HasPrefix(localPath, "/") @@ -230,7 +236,8 @@ func (m *GoModHandler) HasOtherReplaces(exceptModule string) bool { } if (strings.HasPrefix(trimmed, "replace ") || inReplaceBlock) && trimmed != "" { - if exceptModule != "" && strings.Contains(trimmed, exceptModule) { + mod, _ := parseReplaceLine(trimmed, inReplaceBlock) + if exceptModule != "" && mod == exceptModule { continue } return true @@ -239,6 +246,65 @@ func (m *GoModHandler) HasOtherReplaces(exceptModule string) bool { return false } +// EnsureReplace ensures that a replace directive exists for the given module path +// pointing to the given local path. Returns true if the file was modified. +func (m *GoModHandler) EnsureReplace(modulePath, localPath string) bool { + // check if loaded + if len(m.Lines) == 0 { + if err := m.load(); err != nil { + } + } + + // Check if already exists + found := false + inReplaceBlock := false + for _, line := range m.Lines { + trimmed := strings.TrimSpace(line) + if strings.HasPrefix(trimmed, "replace (") { + inReplaceBlock = true + continue + } + if inReplaceBlock && trimmed == ")" { + inReplaceBlock = false + continue + } + + if (strings.HasPrefix(trimmed, "replace ") || inReplaceBlock) { + mod, target := parseReplaceLine(trimmed, inReplaceBlock) + if mod == modulePath && filepath.Clean(target) == filepath.Clean(localPath) { + found = true + break + } + } + } + + if found { + return false + } + + // Add it. If there's a replace block, add it there. Otherwise, add a single-line replace. + + // Try to find a place to insert + inserted := false + var newLines []string + for _, line := range m.Lines { + newLines = append(newLines, line) + if !inserted && strings.HasPrefix(strings.TrimSpace(line), "replace (") { + newLines = append(newLines, "\t"+modulePath+" => "+localPath) + inserted = true + } + } + + if !inserted { + // Just append at the end + newLines = append(newLines, "", "replace "+modulePath+" => "+localPath) + } + + m.Lines = newLines + m.Modified = true + return true +} + // Save writes changes back to the file if modified func (m *GoModHandler) Save() error { if !m.Modified { @@ -545,10 +611,13 @@ func (g *Go) UpdateDependents(modulePath, version, searchPath string) error { return nil } -// FindDependentModules searches for modules that have modulePath as dependency +// FindDependentModules searches for modules that have modulePath as dependency. +// It excludes modules located inside the current project's root directory. func (g *Go) FindDependentModules(modulePath, searchPath string) ([]string, error) { var dependents []string + absRoot, _ := filepath.Abs(g.rootDir) + err := filepath.Walk(searchPath, func(path string, info os.FileInfo, err error) error { if err != nil { return nil // Continue despite errors @@ -559,8 +628,16 @@ func (g *Go) FindDependentModules(modulePath, searchPath string) ([]string, erro return nil } + dir := filepath.Dir(path) + absDir, _ := filepath.Abs(dir) + + // Exclude internal submodules + if strings.HasPrefix(absDir, absRoot+string(os.PathSeparator)) || absDir == absRoot { + return nil + } + if g.HasDependency(path, modulePath) { - dependents = append(dependents, filepath.Dir(path)) + dependents = append(dependents, dir) } return nil diff --git a/go_selfdep.go b/go_selfdep.go new file mode 100644 index 0000000..1e2b294 --- /dev/null +++ b/go_selfdep.go @@ -0,0 +1,77 @@ +package devflow + +import ( + "fmt" + "os" + "path/filepath" +) + +// syncInternalSubmodules finds all submodules inside the current repo that depend +// on the parent module, and prepares them for the upcoming release: +// 1. Ensures they have a relative replace pointing to the parent. +// 2. Bumps their requirement of the parent module to the nextTag. +// 3. Runs 'go mod tidy'. +// These changes are made pre-commit so they are included in the release tag. +func (g *Go) syncInternalSubmodules(parentModulePath, nextTag string) error { + absRoot, _ := filepath.Abs(g.rootDir) + + // 1. Find all go.mod files inside the repo (excluding root) + var submods []string + err := filepath.Walk(absRoot, func(path string, info os.FileInfo, err error) error { + if err != nil { + return nil + } + if info.Name() == "go.mod" { + absDir, _ := filepath.Abs(filepath.Dir(path)) + if absDir != absRoot { + submods = append(submods, absDir) + } + } + return nil + }) + if err != nil { + return err + } + + for _, subDir := range submods { + if !g.HasDependency(filepath.Join(subDir, "go.mod"), parentModulePath) { + continue + } + + g.log(fmt.Sprintf("Syncing internal submodule: %s", filepath.Base(subDir))) + + // 1. Ensure relative replace + rel, err := filepath.Rel(subDir, absRoot) + if err != nil { + continue + } + + gomod := NewGoModHandler() + gomod.SetRootDir(subDir) + gomod.EnsureReplace(parentModulePath, rel) + if err := gomod.Save(); err != nil { + return fmt.Errorf("failed to save submodule go.mod: %w", err) + } + + // 2. Bump requirement to nextTag + // Note: We use go get parent@tag. Since we have a replace, this just updates go.mod + // without needing the tag to be published yet. + target := fmt.Sprintf("%s@%s", parentModulePath, nextTag) + if _, err := RunCommandInDir(subDir, "go", "get", target); err != nil { + // If it fails (e.g. tag doesn't exist yet and proxy is used), + // we might need to use a different approach or just ignore if it's not strictly needed + // for the build (because of the replace). + // But gopush's goal is to have the correct version in go.mod. + // Try with GOPROXY=off or similar? + // Actually, 'go get' should work if the replace is there and we are just updating the version string. + g.log(fmt.Sprintf("Warning: go get %s failed in %s: %v", target, subDir, err)) + } + + // 3. Tidy + if _, err := RunCommandInDir(subDir, "go", "mod", "tidy"); err != nil { + return fmt.Errorf("go mod tidy failed in %s: %w", subDir, err) + } + } + + return nil +} diff --git a/gotest.go b/gotest.go index 6c99f20..a949f38 100644 --- a/gotest.go +++ b/gotest.go @@ -171,7 +171,11 @@ func (g *Go) runFullTestSuite(moduleName string, skipRace bool, timeoutSec int, var testErr error var testOutput string - timeoutFlag := fmt.Sprintf("-timeout=%ds", timeoutSec) + // Watchdog and backstop timeout semantincs: + // -t N now means "max N seconds per test stall" + // backstop is 10x larger to catch overall package hangs + timeoutFlag := fmt.Sprintf("-timeout=%ds", timeoutSec*10) + tmpCovDir, _ := os.MkdirTemp("", "gotest-cov") defer os.RemoveAll(tmpCovDir) coverProfilePath := fmt.Sprintf("%s/cover.out", tmpCovDir) @@ -186,10 +190,18 @@ func (g *Go) runFullTestSuite(moduleName string, skipRace bool, timeoutSec int, testArgs = append(testArgs[:1], append([]string{"-race"}, testArgs[1:]...)...) } - // Safety net: context kills process 10s after Go's -timeout should have fired - // This ensures we get the nice panic output from Go if possible - testCtx, testCancel := context.WithTimeout(context.Background(), time.Duration(timeoutSec+10)*time.Second) + // Watchdog setup + testCtx, testCancel := context.WithCancel(context.Background()) defer testCancel() + + var watchdogFired bool + wd := NewWatchdog(time.Duration(timeoutSec)*time.Second, func() { + watchdogFired = true + testCancel() + }) + wd.Start() + defer wd.Stop() + testCmd := GoTestCmdFn(testCtx, g.rootDir, "go", testArgs...) testBuffer := &bytes.Buffer{} @@ -201,6 +213,7 @@ func (g *Go) runFullTestSuite(moduleName string, skipRace bool, timeoutSec int, s := string(p) testBuffer.Write(p) testFilter.Add(s) + wd.Add(s) return len(p), nil }, } @@ -220,31 +233,44 @@ func (g *Go) runFullTestSuite(moduleName string, skipRace bool, timeoutSec int, if runAll { subArgs = append(subArgs[:len(subArgs)-1], "-tags=integration", "./...") } - subCtx, subCancel := context.WithTimeout(context.Background(), time.Duration(timeoutSec+10)*time.Second) + + subCtx, subCancel := context.WithCancel(context.Background()) defer subCancel() + + // Re-initialize watchdog for submodule run + wd = NewWatchdog(time.Duration(timeoutSec)*time.Second, func() { + watchdogFired = true + subCancel() + }) + wd.Start() + subCmd := GoTestCmdFn(subCtx, subDir, "go", subArgs...) subCmd.Stdout = testPipe subCmd.Stderr = testPipe if err := subCmd.Run(); err != nil && testErr == nil { testErr = err } + wd.Stop() } testFilter.Flush() testOutput = testBuffer.String() - // Detect process-level timeout (killed by context) - if testCtx.Err() == context.DeadlineExceeded { - timedOut := FindTimedOutTests(testOutput) - if len(timedOut) > 0 { - for _, name := range timedOut { - addMsg(false, fmt.Sprintf("timeout: %s (exceeded %ds)", name, timeoutSec)) + // Detect process-level timeout (killed by watchdog or backstop) + if testCtx.Err() == context.Canceled && watchdogFired { + culprits := wd.Culprits() + if len(culprits) > 0 { + for _, name := range culprits { + addMsg(false, fmt.Sprintf("timeout: %s stalled >%ds (no progress)", name, timeoutSec)) } } else { - addMsg(false, fmt.Sprintf("timeout: tests exceeded %ds", timeoutSec)) + addMsg(false, fmt.Sprintf("timeout: stall detected (>%ds)", timeoutSec)) } testStatus = "Failed" + } else if testCtx.Err() == context.DeadlineExceeded { + addMsg(false, fmt.Sprintf("timeout: package exceeded %ds total (backstop)", timeoutSec*10)) + testStatus = "Failed" } // Process test results @@ -459,7 +485,27 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i }() // Inject timeout if user didn't already pass -timeout - timeoutFlag := fmt.Sprintf("-timeout=%ds", timeoutSec) + // Use -v for watchdog to work, ConsoleFilter will suppress the noise + if !HasVFlag(customArgs) { + customArgs = append([]string{"-v"}, customArgs...) + } + + // Parse custom timeout if present + for _, arg := range customArgs { + if strings.HasPrefix(arg, "-t=") { + if t, err := strconv.Atoi(strings.TrimPrefix(arg, "-t=")); err == nil { + timeoutSec = t + } + } else if strings.HasPrefix(arg, "-timeout=") { + // Try to parse go's duration format + dStr := strings.TrimPrefix(arg, "-timeout=") + if d, err := time.ParseDuration(dStr); err == nil { + timeoutSec = int(d.Seconds()) + } + } + } + + timeoutFlag := fmt.Sprintf("-timeout=%ds", timeoutSec*10) if !HasTimeoutFlag(customArgs) { customArgs = append(customArgs, timeoutFlag) } @@ -471,8 +517,17 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i } testArgs = append(testArgs, "./...") - customCtx, customCancel := context.WithTimeout(context.Background(), time.Duration(timeoutSec+10)*time.Second) + customCtx, customCancel := context.WithCancel(context.Background()) defer customCancel() + + var watchdogFired bool + wd := NewWatchdog(time.Duration(timeoutSec)*time.Second, func() { + watchdogFired = true + customCancel() + }) + wd.Start() + defer wd.Stop() + testCmd := GoTestCmdFn(customCtx, g.rootDir, "go", testArgs...) testBuffer := &bytes.Buffer{} @@ -483,6 +538,7 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i s := string(p) testBuffer.Write(p) testFilter.Add(s) + wd.Add(s) return len(p), nil }, } @@ -498,14 +554,24 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i subArgs = append(subArgs, "-tags=integration") } subArgs = append(subArgs, "./...") - subCtx, subCancel := context.WithTimeout(context.Background(), time.Duration(timeoutSec+10)*time.Second) + + subCtx, subCancel := context.WithCancel(context.Background()) defer subCancel() + + // Re-initialize watchdog for submodule run + wd = NewWatchdog(time.Duration(timeoutSec)*time.Second, func() { + watchdogFired = true + subCancel() + }) + wd.Start() + subCmd := GoTestCmdFn(subCtx, subDir, "go", subArgs...) subCmd.Stdout = testPipe subCmd.Stderr = testPipe if err := subCmd.Run(); err != nil && testErr == nil { testErr = err } + wd.Stop() } testFilter.Flush() @@ -513,16 +579,20 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i testOutput := testBuffer.String() // Detect process-level timeout - if customCtx.Err() == context.DeadlineExceeded { - timedOut := FindTimedOutTests(testOutput) - if len(timedOut) > 0 { - for _, name := range timedOut { - addMsg(false, fmt.Sprintf("timeout: %s (exceeded %ds)", name, timeoutSec)) + customTestStatus := "Failed" + if customCtx.Err() == context.Canceled && watchdogFired { + culprits := wd.Culprits() + if len(culprits) > 0 { + for _, name := range culprits { + addMsg(false, fmt.Sprintf("timeout: %s stalled >%ds (no progress)", name, timeoutSec)) } } else { - addMsg(false, fmt.Sprintf("timeout: tests exceeded %ds", timeoutSec)) + addMsg(false, fmt.Sprintf("timeout: stall detected (>%ds)", timeoutSec)) } - // Will be caught by EvaluateTestResults as a failure + } else if customCtx.Err() == context.DeadlineExceeded { + addMsg(false, fmt.Sprintf("timeout: package exceeded %ds total (backstop)", timeoutSec*10)) + } else { + customTestStatus = "" // Not a context-triggered failure } // Wait for WASM detection to complete @@ -530,6 +600,9 @@ func (g *Go) runCustomTests(customArgs []string, moduleName string, timeoutSec i // Process stdlib test results (without race detection reporting) testStatus, _, stdTestsRan, msgs := EvaluateTestResults(testErr, testOutput, moduleName, msgs, false) + if customTestStatus != "" { + testStatus = customTestStatus + } // Initialize coveragePercent for custom runs (not calculated for stdlib in fast path usually, but we need it for comparison) coveragePercent := "0" @@ -859,6 +932,16 @@ func ShouldEnableWasm(nativeOut, wasmOut string) bool { return false } +// HasVFlag checks if -v is already present in the args +func HasVFlag(args []string) bool { + for _, arg := range args { + if arg == "-v" || arg == "-test.v" { + return true + } + } + return false +} + // parseGoListFiles converts the output of go list into a map of unique test files func parseGoListFiles(output string) map[string]bool { fileMap := make(map[string]bool) diff --git a/interface.go b/interface.go index a7a1f66..84d0d4e 100644 --- a/interface.go +++ b/interface.go @@ -44,6 +44,7 @@ type GitClient interface { PushWithTags(tag string) (bool, error) PushWithoutTags() (bool, error) HasPendingChanges() (bool, error) + GenerateNextTag() (string, error) } // FolderWatcher defines interface for adding/removing directories to watch diff --git a/test/go_handler_test.go b/test/go_handler_test.go index 1165666..583a249 100644 --- a/test/go_handler_test.go +++ b/test/go_handler_test.go @@ -568,6 +568,10 @@ func (m *MockGitClient) HasPendingChanges() (bool, error) { return true, nil } +func (m *MockGitClient) GenerateNextTag() (string, error) { + return "v0.0.1", nil +} + func TestGoPush_RemoteAccessFailure(t *testing.T) { dir, cleanup := testCreateGoModule("github.com/test/repo") defer cleanup() diff --git a/test/go_selfdep_test.go b/test/go_selfdep_test.go new file mode 100644 index 0000000..308b638 --- /dev/null +++ b/test/go_selfdep_test.go @@ -0,0 +1,103 @@ +package devflow_test + +import ( + "os" + "path/filepath" + "testing" + "strings" + + "github.com/tinywasm/devflow" +) + +// testCreateGoModule duplicated here because I cannot access it from other files easily in this env it seems? +// Actually it is in helpers_test.go, but maybe the test run didn't include it. +func localTestCreateGoModule(moduleName string) (dir string, cleanup func()) { + dir, _ = os.MkdirTemp("", "gitgo-gomod-") + + // Create go.mod + gomod := "module " + moduleName + "\n\ngo 1.20\n" + os.WriteFile(filepath.Join(dir, "go.mod"), []byte(gomod), 0644) + + // Create main.go + main := "package main\n\nfunc main() {}\n" + os.WriteFile(filepath.Join(dir, "main.go"), []byte(main), 0644) + + cleanup = func() { + os.RemoveAll(dir) + } + + return dir, cleanup +} + +func TestInternalSubmoduleHandling(t *testing.T) { + // 1. Create a parent module and an internal submodule + tmpDir, cleanup := localTestCreateGoModule("github.com/parent") + defer cleanup() + + subDir := filepath.Join(tmpDir, "sub") + os.MkdirAll(subDir, 0755) + + subModContent := "module github.com/parent/sub\n\ngo 1.20\n\nrequire github.com/parent v0.0.1\nreplace github.com/parent => ../\n" + os.WriteFile(filepath.Join(subDir, "go.mod"), []byte(subModContent), 0644) + os.WriteFile(filepath.Join(subDir, "main.go"), []byte("package sub\n"), 0644) + + git, _ := devflow.NewGit() + git.SetRootDir(tmpDir) + + gh, _ := devflow.NewGo(git) + gh.SetRootDir(tmpDir) + + // 2. FindDependentModules should EXCLUDE internal submodules + deps, err := gh.FindDependentModules("github.com/parent", tmpDir) + if err != nil { + t.Fatalf("FindDependentModules failed: %v", err) + } + + for _, dep := range deps { + if strings.HasPrefix(dep, tmpDir) { + t.Errorf("Internal submodule %s should be excluded from dependents", dep) + } + } + + // 3. EnsureReplace should add/keep replace parent => ../ + gomod := devflow.NewGoModHandler() + gomod.SetRootDir(subDir) + + // Should keep existing + changed := gomod.EnsureReplace("github.com/parent", "../") + if changed { + t.Error("EnsureReplace should not report change when replace already exists correctly") + } + + // Should add if missing + os.WriteFile(filepath.Join(subDir, "go.mod"), []byte("module github.com/parent/sub\n\ngo 1.20\n\nrequire github.com/parent v0.0.1\n"), 0644) + gomod = devflow.NewGoModHandler() + gomod.SetRootDir(subDir) + changed = gomod.EnsureReplace("github.com/parent", "../") + if !changed { + t.Error("EnsureReplace should report change when adding missing replace") + } + gomod.Save() + + content, _ := os.ReadFile(filepath.Join(subDir, "go.mod")) + if !strings.Contains(string(content), "replace github.com/parent => ../") { + t.Errorf("go.mod missing expected replace. Got:\n%s", string(content)) + } + + // Test with replace block + os.WriteFile(filepath.Join(subDir, "go.mod"), []byte("module github.com/parent/sub\n\ngo 1.20\n\nreplace (\n\tgithub.com/other => ./other\n)\n"), 0644) + gomod = devflow.NewGoModHandler() + gomod.SetRootDir(subDir) + changed = gomod.EnsureReplace("github.com/parent", "../") + if !changed { + t.Error("EnsureReplace should report change when adding to block") + } + gomod.Save() + content, _ = os.ReadFile(filepath.Join(subDir, "go.mod")) + if strings.Contains(string(content), "replace github.com/parent => ../") { + t.Errorf("Should not contain 'replace' keyword inside block, got:\n%s", string(content)) + } + if !strings.Contains(string(content), "github.com/parent => ../") { + t.Errorf("Block should contain module path, got:\n%s", string(content)) + } +} diff --git a/test/gonew_test.go b/test/gonew_test.go index b9f3951..1ac400a 100644 --- a/test/gonew_test.go +++ b/test/gonew_test.go @@ -1,3 +1,5 @@ +//go:build integration + package devflow_test import "github.com/tinywasm/devflow" diff --git a/test/watchdog_test.go b/test/watchdog_test.go new file mode 100644 index 0000000..285a205 --- /dev/null +++ b/test/watchdog_test.go @@ -0,0 +1,182 @@ +package devflow_test + +import ( + "context" + "sync" + "testing" + "time" + + "github.com/tinywasm/devflow" +) + +func TestWatchdogCumulativeNotKilled(t *testing.T) { + // A stream of many fast tests whose total exceeds the limit does NOT trigger a kill + timeout := 100 * time.Millisecond + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + _ = ctx + + var mu sync.Mutex + killed := false + onKill := func() { + mu.Lock() + killed = true + mu.Unlock() + cancel() + } + + w := devflow.NewWatchdog(timeout, onKill) + w.Start() + defer w.Stop() + + // Simulate 5 tests, each taking 30ms. Total 150ms > 100ms timeout. + // But no individual test exceeds 100ms. + for i := 1; i <= 5; i++ { + w.Add(string([]byte("=== RUN TestFast\n"))) + time.Sleep(30 * time.Millisecond) + w.Add(string([]byte("--- PASS: TestFast (0.03s)\n"))) + mu.Lock() + isKilled := killed + mu.Unlock() + if isKilled { + t.Fatalf("Watchdog killed process prematurely at test %d", i) + } + } + + mu.Lock() + isKilled := killed + mu.Unlock() + if isKilled { + t.Error("Watchdog should not have killed the process") + } +} + +func TestWatchdogStallKilled(t *testing.T) { + // A single test with no completion event within the limit DOES trigger + timeout := 50 * time.Millisecond + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + _ = ctx + + var mu sync.Mutex + killed := false + onKill := func() { + mu.Lock() + killed = true + mu.Unlock() + cancel() + } + + w := devflow.NewWatchdog(timeout, onKill) + w.Start() + defer w.Stop() + + w.Add("=== RUN TestStall\n") + + // Wait for watchdog to fire + time.Sleep(150 * time.Millisecond) + + mu.Lock() + isKilled := killed + mu.Unlock() + if !isKilled { + t.Error("Watchdog should have killed the stalled process") + } + + culprits := w.Culprits() + if len(culprits) != 1 || culprits[0] != "TestStall" { + t.Errorf("Expected culprit TestStall, got %v", culprits) + } +} + +func TestWatchdogPauseCont(t *testing.T) { + // PAUSE/CONT sequences do not blame paused tests + timeout := 100 * time.Millisecond + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + _ = ctx + + var mu sync.Mutex + killed := false + onKill := func() { + mu.Lock() + killed = true + mu.Unlock() + cancel() + } + + w := devflow.NewWatchdog(timeout, onKill) + w.Start() + defer w.Stop() + + w.Add("=== RUN TestParallel\n") + w.Add("=== PAUSE TestParallel\n") + + // Stay paused for longer than timeout + time.Sleep(200 * time.Millisecond) + + mu.Lock() + isKilled := killed + mu.Unlock() + if isKilled { + t.Fatal("Watchdog should not kill a paused test") + } + + w.Add("=== CONT TestParallel\n") + + // Now it should be active. Wait a bit less than timeout. + time.Sleep(50 * time.Millisecond) + mu.Lock() + isKilled = killed + mu.Unlock() + if isKilled { + t.Fatal("Watchdog killed test too early after CONT") + } + + // Now wait to exceed timeout + time.Sleep(150 * time.Millisecond) + mu.Lock() + isKilled = killed + mu.Unlock() + if !isKilled { + t.Error("Watchdog should have killed test after it exceeded timeout post-CONT") + } +} + +func TestWatchdogFragmentedLines(t *testing.T) { + timeout := 100 * time.Millisecond + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + _ = ctx + + var mu sync.Mutex + killed := false + onKill := func() { + mu.Lock() + killed = true + mu.Unlock() + cancel() + } + + w := devflow.NewWatchdog(timeout, onKill) + w.Start() + defer w.Stop() + + // Fragmented output + w.Add("=== RUN ") + w.Add(" TestFrag\n") + + time.Sleep(200 * time.Millisecond) + + mu.Lock() + isKilled := killed + mu.Unlock() + if !isKilled { + t.Error("Watchdog should have detected fragmented RUN line") + } + + culprits := w.Culprits() + if len(culprits) != 1 || culprits[0] != "TestFrag" { + t.Errorf("Expected culprit TestFrag, got %v", culprits) + } +} diff --git a/watchdog.go b/watchdog.go new file mode 100644 index 0000000..fc97a41 --- /dev/null +++ b/watchdog.go @@ -0,0 +1,129 @@ +package devflow + +import ( + "regexp" + "strings" + "sync" + "time" +) + +// Watchdog monitors test output for stalled tests. +type Watchdog struct { + timeout time.Duration + onKill func() + running map[string]time.Time + culprits []string + mu sync.Mutex + stop chan struct{} + lineBuf string + runRe *regexp.Regexp + pauseRe *regexp.Regexp + contRe *regexp.Regexp + completeRe *regexp.Regexp + killed bool +} + +// NewWatchdog creates a new watchdog. +func NewWatchdog(timeout time.Duration, onKill func()) *Watchdog { + return &Watchdog{ + timeout: timeout, + onKill: onKill, + running: make(map[string]time.Time), + stop: make(chan struct{}), + runRe: regexp.MustCompile(`=== RUN\s+(\S+)`), + pauseRe: regexp.MustCompile(`=== PAUSE\s+(\S+)`), + contRe: regexp.MustCompile(`=== CONT\s+(\S+)`), + completeRe: regexp.MustCompile(`--- (?:PASS|FAIL|SKIP): (\S+)`), + } +} + +// Start begins the monitoring goroutine. +func (w *Watchdog) Start() { + go func() { + ticker := time.NewTicker(w.timeout / 4) + defer ticker.Stop() + for { + select { + case <-ticker.C: + w.check() + case <-w.stop: + return + } + } + }() +} + +// Stop halts the monitoring goroutine. +func (w *Watchdog) Stop() { + close(w.stop) +} + +// Add appends output to be parsed. +func (w *Watchdog) Add(s string) { + w.mu.Lock() + defer w.mu.Unlock() + + w.lineBuf += s + for { + idx := strings.Index(w.lineBuf, "\n") + if idx == -1 { + break + } + line := w.lineBuf[:idx] + w.lineBuf = w.lineBuf[idx+1:] + w.processLine(line) + } +} + +func (w *Watchdog) processLine(line string) { + if m := w.runRe.FindStringSubmatch(line); m != nil { + w.running[m[1]] = time.Now() + } else if m := w.pauseRe.FindStringSubmatch(line); m != nil { + delete(w.running, m[1]) + } else if m := w.contRe.FindStringSubmatch(line); m != nil { + w.running[m[1]] = time.Now() + } else if m := w.completeRe.FindStringSubmatch(line); m != nil { + delete(w.running, m[1]) + } +} + +func (w *Watchdog) check() { + w.mu.Lock() + onKill := w.onKill + killed := w.killed + timeout := w.timeout + running := make(map[string]time.Time) + for k, v := range w.running { + running[k] = v + } + w.mu.Unlock() + + if killed || onKill == nil { + return + } + + now := time.Now() + for name, start := range running { + if now.Sub(start) > timeout { + w.mu.Lock() + if w.killed { // double-check + w.mu.Unlock() + return + } + w.culprits = append(w.culprits, name) + w.killed = true + w.onKill = nil + w.mu.Unlock() + + onKill() + return + } + } +} + +// Culprits returns the list of tests that were running when the watchdog fired. +func (w *Watchdog) Culprits() []string { + w.mu.Lock() + defer w.mu.Unlock() + return w.culprits +} From 4269710015d9535fa0f0c596d8294793102c2afc Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 8 Jul 2026 14:35:38 -0400 Subject: [PATCH 2/4] deps: update router to v0.1.4 --- docs/{PLAN.md => CHECK_PLAN.md} | 0 docs/GOPUSH_SELFDEP_PLAN.md | 87 --------------------------------- docs/GOTEST_TIMEOUT_PLAN.md | 84 ------------------------------- docs/img/badges.svg | 2 +- go.mod | 4 +- go.sum | 4 +- 6 files changed, 5 insertions(+), 176 deletions(-) rename docs/{PLAN.md => CHECK_PLAN.md} (100%) delete mode 100644 docs/GOPUSH_SELFDEP_PLAN.md delete mode 100644 docs/GOTEST_TIMEOUT_PLAN.md diff --git a/docs/PLAN.md b/docs/CHECK_PLAN.md similarity index 100% rename from docs/PLAN.md rename to docs/CHECK_PLAN.md diff --git a/docs/GOPUSH_SELFDEP_PLAN.md b/docs/GOPUSH_SELFDEP_PLAN.md deleted file mode 100644 index 1b18212..0000000 --- a/docs/GOPUSH_SELFDEP_PLAN.md +++ /dev/null @@ -1,87 +0,0 @@ -# PLAN: gopush Same-Repo Submodule Handling - -Status: **PROPOSED** — sub-plan 2 of [PLAN.md](PLAN.md). Read the orchestrator's Development Rules first; execute after [GOTEST_TIMEOUT_PLAN.md](GOTEST_TIMEOUT_PLAN.md). - -## Problem - -A library can contain submodules with their own `go.mod` that depend on the parent module itself (e.g. `tinywasm/orm` contains `ormcp/`, whose `go.mod` requires `github.com/tinywasm/orm`). When `gopush` (or `codejob`, which publishes through the same `Go.Push`) publishes a new tag: - -1. `UpdateDependents` walks `..` and `FindDependentModules` treats `ormcp/` like any **external** dependent — it cannot tell that the dependent lives inside the repo just published. -2. `UpdateDependentModule` calls `RemoveReplace`, and `isLocalReplaceTarget` only preserves `=> .` — a parent-pointing `replace github.com/tinywasm/orm => ../` is **deleted**. -3. It then runs `go get` + `go mod tidy` + commit + push on the submodule — mutating the repo **after** the tag was created. - -Result: the just-published repo is immediately inconsistent again — every publish re-dirties it, forever. Real evidence (current state of `tinywasm/orm` after publishing v0.9.25): uncommitted changes in `go.mod`, `go.sum`, `ormcp/go.mod`, `ormcp/go.sum`, and `ormcp/go.mod` has **no** `replace` left. - -## Required behavior (same-repo submodules only) - -For a submodule `go.mod` inside the repo being published that requires the parent module, three things must happen — **before** the publish commit, so the tag captures them: - -1. **Version bump:** its `require ` is updated to the tag about to be published. -2. **Replace preserved:** an existing local `replace => ../` (any relative path into the repo) is never removed. -3. **Replace added if missing:** if no such replace exists, add one, so the submodule's tests always build against the local (latest, unpublished) parent code. - -External dependents keep today's behavior exactly (remove stale replace, `go get`, tidy, test, push). - -## Design - -### Detection - -A dependent dir found by `FindDependentModules` is a *same-repo submodule* when it is located inside the published module's root (`strings.HasPrefix(depDir, rootDir+sep)` on absolute cleaned paths). Cross-check: its module path starts with `/`. - -### Flow change in `Go.Push` - -The next tag is already computed before commit (`git_handler.go`: `GenerateNextTag` / `IncrementTag`). New step between tests and the commit/tag: - -``` -tests pass - → nextTag = GenerateNextTag() (existing logic, hoisted) - → for each same-repo submodule requiring parent: - EnsureReplace(parent, relPathToRoot) (add if missing, keep if present) - set require parent → nextTag (edit go.mod line; go mod tidy) - → git add / commit / tag nextTag / push (submodule changes ride the same commit) - → UpdateDependents(...) (now skips same-repo submodules) -``` - -The `replace` makes the `require` version irrelevant for local builds, so pointing at a tag that exists only seconds later is safe; after the push the tag is live and external consumers of the submodule resolve normally. - -### Code changes - -- `go_mod.go` — new `EnsureReplace(modulePath, localPath string) bool` on `GoModHandler` (idempotent; respects single-line and block `replace (…)` syntax). -- New file `go_selfdep.go` — `(g *Go) syncInternalSubmodules(modulePath, nextTag string) error`: find, ensure replace, bump require, tidy. Called from `Push` pre-commit. -- `go_mod.go` `FindDependentModules` (or its caller `UpdateDependents`) — exclude dirs inside `g.rootDir`. -- `isLocalReplaceTarget` unchanged: external dependents' sibling-checkout replaces (`=> ../orm`) must still be removable; internal submodules simply never reach `UpdateDependentModule` anymore. - -## Steps - -### Phase 1 — Reproduce (TDD) - -1. `test/go_selfdep_test.go`, failing first, using temp dirs (pattern of `testCreateGoModule`) with a parent module + `sub/go.mod` requiring it: - - `FindDependentModules`/`UpdateDependents` currently returns/mutates the internal submodule → assert it must be excluded; - - `RemoveReplace` flow deletes `=> ../` → assert preserved via the new path; - - submodule without replace → assert `EnsureReplace` adds `replace parent => ../`; - - submodule require bumped to `nextTag` **before** commit (inspect `go.mod` content at the mocked commit step via `ExecCommand`/`GoTestCmdFn` hooks — no real git/network). - -### Phase 2 — Implement - -2. `EnsureReplace` in `go_mod.go` (+ unit tests for line/block syntax, idempotency). -3. `go_selfdep.go` with `syncInternalSubmodules`; hoist next-tag computation in `Push` so it is known pre-commit; wire the call. -4. Exclusion of same-repo dirs in the dependents scan. - -### Phase 3 — Repair & validate - -5. Repair `tinywasm/orm`: ensure `ormcp/go.mod` gets its `replace github.com/tinywasm/orm => ../` back, then run the new `gopush` on orm — the repo must end **clean**: one commit, tag containing the submodule bump, replace intact, `ormcp` never pushed separately. -6. Regression: publish a module with a genuine external dependent and confirm behavior is unchanged. - -### Phase 4 — Docs & release - -7. Update `docs/GOPUSH.md` (new "Same-repo submodules" section) and `docs/CODEJOB.md` (note that it inherits the behavior via `Publish`). Publish devflow with `gopush`. - -## Test strategy summary - -| Case | Expected | -|---|---| -| `sub/go.mod` requires parent, has `replace => ../` | Replace kept, require bumped to next tag pre-commit, no separate push | -| `sub/go.mod` requires parent, no replace | Replace added, require bumped, no separate push | -| Same repo, submodule does not require parent | Untouched | -| External dependent with stale sibling replace | Today's behavior: replace removed, updated, tested, pushed | -| Publish twice in a row | Second run: "Nothing to push" — repo stays clean (the core bug) | diff --git a/docs/GOTEST_TIMEOUT_PLAN.md b/docs/GOTEST_TIMEOUT_PLAN.md deleted file mode 100644 index c564f57..0000000 --- a/docs/GOTEST_TIMEOUT_PLAN.md +++ /dev/null @@ -1,84 +0,0 @@ -# PLAN: gotest Per-Test Stall Watchdog - -Status: **PROPOSED** — sub-plan 1 of [PLAN.md](PLAN.md). Read the orchestrator's Development Rules first; execute this plan before [GOPUSH_SELFDEP_PLAN.md](GOPUSH_SELFDEP_PLAN.md) (gopush's test gate depends on it). - -## Plan-specific rules - -In addition to the orchestrator's shared Development Rules: - -- New behavior goes in its own small file (`watchdog.go`), not appended to `gotest.go`. -- Only the *semantics* of `-t` are refined (see below); the flag itself keeps working and the change is documented in `docs/GOTEST.md`. - -## Problem - -`gotest` injects `-timeout=s` (default 30s) into `go test`. Go applies that timeout **per test binary (package), cumulatively** — not per test. When a package holds many tests, the budget is spent by the *sum* of all tests, and Go kills whichever test happens to be running when the alarm fires, even if it just started. - -Real failure (devflow itself): - -``` -panic: test timed out after 30s - running tests: - TestGoNewCreateLocalOnly (0s) ← blamed, but only 0s elapsed -... -⚠️ slow: TestGoUpdateModuleFail (4.2s) -Tests failed: ... timeout: TestGoNewCreateLocalOnly (exceeded 30s) ❌ -``` - -`TestGoNewCreateLocalOnly` was not hung — the suite's cumulative time crossed 30s. Raising `-t` "fixes" it until the suite grows again: an infinite arms race. The timeout exists to catch **hung tests fast**, not to cap total suite duration. - -## Design: inactivity watchdog - -Change what `-t N` means: from *"package budget"* to *"maximum time a single test may run without finishing"*. A progressing suite can take as long as it needs; a genuinely stuck test is still killed after N seconds. - -Mechanism (see [diagram](diagrams/GOTEST_WATCHDOG.md)): - -1. `gotest` already streams `go test -v` output through `paramWriter` → `ConsoleFilter`. A new `Watchdog` type taps the same stream. -2. The watchdog parses test lifecycle events: `=== RUN`, `=== PAUSE`, `=== CONT`, `--- PASS/FAIL/SKIP`, `ok \t`, `FAIL\t`. It maintains a set of currently-running tests with their start (or resume) timestamps. -3. A ticker goroutine checks: if any running test has been active longer than `timeoutSec`, cancel the command context (kill the process) and record that test as the culprit. -4. The Go-native `-timeout` is still injected as a **backstop**, at `timeoutSec × 10` (so Go's own panic with stack traces still fires if the watchdog itself is bypassed, e.g. output redirected). -5. Reporting distinguishes the two cases: - - Watchdog kill: `❌ timeout: TestX stalled >30s (no progress)` - - Backstop kill: `❌ timeout: package exceeded 300s total` — plus the slow-test list already produced by `FindSlowestTest`. - -Notes: - -- `runCustomTests` (fast path) does not currently pass `-v`; the watchdog needs lifecycle events, so `-v` is injected there too (`ConsoleFilter` already suppresses the noise). -- Parallel tests: `=== PAUSE` removes a test from the active set; `=== CONT` re-adds it with a fresh timestamp, so paused tests are never blamed. -- The watchdog must be created per command (native, submodules, WASM) — same places `ConsoleFilter` is instantiated today. - -## Steps - -### Phase 1 — Watchdog core (TDD) - -1. `test/watchdog_test.go`: failing tests first — - - a stream of many fast tests whose *total* exceeds the limit does **not** trigger a kill (the original bug); - - a single test with no completion event within the limit **does** trigger, and the culprit name is exact; - - `PAUSE`/`CONT` sequences do not blame paused tests; - - fragmented lines (no trailing `\n`) are handled, same contract as `ConsoleFilter.Add`. - - Use short limits (~50–200ms) and an injectable clock/`onKill` callback — no real processes. -2. `watchdog.go`: implement `Watchdog` (`Add(string)`, `Start(ctx cancel)`, `Stop()`, `Culprits() []string`). - -### Phase 2 — Wiring - -3. In `runFullTestSuite` and `runCustomTests`: feed the watchdog from the existing `paramWriter`, replace injected `-timeout=s` with `-timeout=s` backstop, inject `-v` in the custom path, cancel the command context on watchdog fire. Cover with a test using `GoTestCmdFn` (fake command emitting a scripted event stream). - -### Phase 3 — Reporting - -4. Distinguish `stalled` vs `backstop` messages in the summary; keep `FindTimedOutTests` as fallback for backstop/external kills. Update existing timeout tests. - -### Phase 4 — devflow's own suite hygiene - -5. Tag the heavy tests that do real git/keyring work (`TestGoNewCreateLocalOnly`, gopush/gonew integration-style tests) with `//go:build integration` so the default `gotest` run stays fast; they run via `gotest -all`. This is independent of the watchdog but removes today's noise. - -### Phase 5 — Docs & release - -6. Update `docs/GOTEST.md` "Timeout" section (per-test stall semantics, backstop, `-all`). Link this plan and the diagram from `README.md`. Publish with `gopush`. - -## Test strategy summary - -| Case | Expected | -|---|---| -| 40 tests × 1s each, `-t 30` | Suite passes (total 40s > 30s is fine) | -| One test silent for 31s, `-t 30` | Killed at ~30s, blamed by name, `stalled` message | -| Parallel: paused 60s, active progressing | No kill | -| Backstop reached (watchdog bypassed) | `package exceeded Ns total` + slowest tests listed | diff --git a/docs/img/badges.svg b/docs/img/badges.svg index c5e5017..658db80 100644 --- a/docs/img/badges.svg +++ b/docs/img/badges.svg @@ -51,7 +51,7 @@ text-anchor="middle" font-family="sans-serif" font-size="11" fill="white">Coverage 55.1% + text-anchor="middle" font-family="sans-serif" font-size="11" fill="white">53.6% diff --git a/go.mod b/go.mod index c055831..c3c743f 100644 --- a/go.mod +++ b/go.mod @@ -7,14 +7,14 @@ require github.com/zalando/go-keyring v0.2.6 require ( github.com/tinywasm/gorun v0.0.24 github.com/tinywasm/mcp v0.1.18 + github.com/tinywasm/model v0.0.6 golang.org/x/term v0.40.0 ) require ( github.com/tinywasm/fetch v0.1.24 // indirect github.com/tinywasm/json v0.5.10 // indirect - github.com/tinywasm/model v0.0.6 // indirect - github.com/tinywasm/router v0.1.3 // indirect + github.com/tinywasm/router v0.1.4 // indirect github.com/tinywasm/time v0.5.0 // indirect github.com/tinywasm/unixid v0.2.23 // indirect ) diff --git a/go.sum b/go.sum index 7959b4c..c4fe780 100644 --- a/go.sum +++ b/go.sum @@ -28,8 +28,8 @@ github.com/tinywasm/mcp v0.1.18 h1:duDh83HsxRQiwP1fcclHxZwS5bUTKyTCPTD3VmtUAog= github.com/tinywasm/mcp v0.1.18/go.mod h1:buwfBL++Ljmwxyz9q05J7tFIwzXXRcD6I5hBAtJnuRw= github.com/tinywasm/model v0.0.6 h1:Zh/JvRiBSDOTLLiHmnTXKfxyUrObKGQHNX+gdGyfV4U= github.com/tinywasm/model v0.0.6/go.mod h1:af3eNGmQA/BMVjMqMUXVgL0B8+NMTwlc6h7TpdEggTo= -github.com/tinywasm/router v0.1.3 h1:RMG7+6JaRjU6sYgkCxEWkxNV26fgFDP4tnUovPIfC6o= -github.com/tinywasm/router v0.1.3/go.mod h1:GuEC1pbuKvhxQLJ3yzKuYZj/hUih+ILl3T6NSrbdidw= +github.com/tinywasm/router v0.1.4 h1:kiW3p3wkxs5jcR9Vbb75NsEoOcp7mlxeJ7WfqaY0t9c= +github.com/tinywasm/router v0.1.4/go.mod h1:2GOrA6+KRQq0om5FxtONAUKfiZ9L5XGZAoK/A3hl7FM= github.com/tinywasm/time v0.5.0 h1:M17DNgKrMYfkCK42V6miL9AcwfN/FHghbKOSTXJrkMo= github.com/tinywasm/time v0.5.0/go.mod h1:vxGAkKA40im4ObiTS6hcsJYiIsVVLSz/A63JC/+TwvA= github.com/tinywasm/unixid v0.2.23 h1:Lp/TER0RwbaT7EcHbQNzXj9LOGA2VD8rC8XLYmxJBc0= From e1244bcade6c50b87a6b2b04d2308bde993793e3 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 8 Jul 2026 14:57:14 -0400 Subject: [PATCH 3/4] deps: update mcp to v0.1.19 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index c3c743f..01d7a81 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require github.com/zalando/go-keyring v0.2.6 require ( github.com/tinywasm/gorun v0.0.24 - github.com/tinywasm/mcp v0.1.18 + github.com/tinywasm/mcp v0.1.19 github.com/tinywasm/model v0.0.6 golang.org/x/term v0.40.0 ) diff --git a/go.sum b/go.sum index c4fe780..6584fc6 100644 --- a/go.sum +++ b/go.sum @@ -24,8 +24,8 @@ github.com/tinywasm/gorun v0.0.24 h1:qduMrpcw0nf65nx8Fi2duBWN7pCIwXxePPa9KL0rmWM github.com/tinywasm/gorun v0.0.24/go.mod h1:9pApF2ZoaH/km6kGBVxL5WlStOoLwEZiV1uG4R4qeKo= github.com/tinywasm/json v0.5.10 h1:GBnCQhTx3ZFKfyTgQwv8xcj8X8lZNxqe9xEvjPm+0/0= github.com/tinywasm/json v0.5.10/go.mod h1:KJdG19+pLcAlV6bWGebSt5WJbg2an1IxP1jSnHSVi9k= -github.com/tinywasm/mcp v0.1.18 h1:duDh83HsxRQiwP1fcclHxZwS5bUTKyTCPTD3VmtUAog= -github.com/tinywasm/mcp v0.1.18/go.mod h1:buwfBL++Ljmwxyz9q05J7tFIwzXXRcD6I5hBAtJnuRw= +github.com/tinywasm/mcp v0.1.19 h1:xJ0ugoPEeLwzIPlt+ck64Sxf1Z2fTjrWdGLa16mjdYk= +github.com/tinywasm/mcp v0.1.19/go.mod h1:dh91vRW8almXcKlYPP0LKOMKu9QR3fjyQq90q/KqM/I= github.com/tinywasm/model v0.0.6 h1:Zh/JvRiBSDOTLLiHmnTXKfxyUrObKGQHNX+gdGyfV4U= github.com/tinywasm/model v0.0.6/go.mod h1:af3eNGmQA/BMVjMqMUXVgL0B8+NMTwlc6h7TpdEggTo= github.com/tinywasm/router v0.1.4 h1:kiW3p3wkxs5jcR9Vbb75NsEoOcp7mlxeJ7WfqaY0t9c= From d216f74ce7e442f5bb9b9bd788cda5a73dec807c Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 8 Jul 2026 16:56:17 -0400 Subject: [PATCH 4/4] docs: update SKILL.md for core principles and form-codegen descriptions --- skills/core-principles/SKILL.md | 2 + skills/form-codegen/SKILL.md | 213 ++++++++++++-------------------- 2 files changed, 84 insertions(+), 131 deletions(-) diff --git a/skills/core-principles/SKILL.md b/skills/core-principles/SKILL.md index db09c3a..98bf260 100644 --- a/skills/core-principles/SKILL.md +++ b/skills/core-principles/SKILL.md @@ -20,6 +20,8 @@ description: Go development principles including SRP, dependency injection, fram - **Exit codes as contract:** `0` on success (including help printed and clean shutdown); non-zero on bad/conflicting flags or startup failure. The agent branches on the code, not on prose. Library functions return errors; the thin `main` maps them to exit codes (that mapping is the only logic allowed in `main`). - **Structured results belong to the tool surface:** when an agent needs rich results, expose them through the program's protocol layer (e.g. MCP/JSON-RPC tools), not by parsing free-form stdout. +- **Views belong to the consumer (libraries render no pages):** A reusable library exposes flows, data and contracts — routes (`POST /login`), models (`model.Definition`), callbacks, tokens. HOW anything looks (HTML pages, branding, layout composition) is created by the **consuming app** in its composition root (e.g. `config/`), typically with `tinywasm/form` over the library's generated models plus the app's own CSS. A library that renders its own HTML page is a design smell: split the flow (stays in the library) from the view (moves to the consumer). Published layout skeletons (`tinywasm/layout`) are the one exception — layout structure IS their single responsibility, and even they define only WHERE things go, never WHAT they contain. + - **Framework-less Development:** For Web projects, use only the **Standard Library** (HTML/CSS/JS). No external frameworks or libraries are allowed. - **CSS-First Interactivity:** Minimize JavaScript usage. All UI interactivity (toggles, menus, states) must be implemented using pure CSS whenever possible. - **Minimalist JS:** Use JavaScript only as a last resort for logic that cannot be handled by CSS or the Go backend. diff --git a/skills/form-codegen/SKILL.md b/skills/form-codegen/SKILL.md index d5dc709..b4e7f70 100644 --- a/skills/form-codegen/SKILL.md +++ b/skills/form-codegen/SKILL.md @@ -1,156 +1,107 @@ --- name: form-codegen -description: Unified form validation architecture across tinywasm/fmt, tinywasm/dom, tinywasm/form, tinywasm/json, and tinywasm/orm. Use when working on input widgets, ormc code generation, field validation, widget assignment, or form API design. +description: Model authoring and form generation across tinywasm/model, tinywasm/orm (ormc), tinywasm/form, tinywasm/json, and tinywasm/dom. Use when creating or modifying models/DTOs, input widgets, ormc code generation, field validation, widget assignment, or form API design. --- -# Form + Codegen Unified Architecture +# Models + Forms: typed `model.Definition` → ormc → everything else -## Overview +## The ONE pattern (source of truth) -Form creation and validation uses the same code on frontend (WASM) and backend (server). The `fmt.Widget` interface is the bridge — it carries validation logic AND UI rendering capability. - -**One way to create forms: `form.New(parentID, &struct)` (schema-driven).** No declarative API. - -## Library Roles - -| Library | Responsibility | Knows About | -|---|---|---| -| **fmt** | Field, Permitted, Widget interface, ValidateFields() | Nothing else | -| **dom** | HTML layout elements (Div, H1, Button, Nav...). NO form functions | fmt only | -| **form** | Schema-driven form creation via `form.New()` | fmt, dom, form/input | -| **form/input** | Concrete widgets with validation (Email, Text, etc.) | fmt only | -| **json** | Zero-reflection JSON codec using Fielder | fmt only | -| **orm** | DB mapping + ormc code generator | fmt only (generated code imports form/input) | - -## Key Design Rule - -**dom does NOT have form functions.** No Email(), Form(), Password(), Select(), Textarea(), Input() in dom. These are form concerns with validation — they live in form/input as widgets. dom only provides pure HTML layout elements. - -## Widget Assignment: Go Type Defaults + Explicit Override - -ormc assigns widgets based on Go type defaults. No Resolver, no aliases, no name matching. - -**Only structs with `// ormc:form` or `// ormc:formonly` directive get widgets.** - -| Directive | DB/ORM | Widgets | -|---|---|---| -| _(none)_ | yes | no | -| `// ormc:form` | yes | yes | -| `// ormc:formonly` | no | yes | - -### Defaults by Go type - -| Go type | Default Widget | -|---|---| -| `string` | `input.Text()` | -| `int`, `int64`, etc. | `input.Number()` | -| `float32`, `float64` | `input.Number()` | -| `bool` | `input.Checkbox()` | - -### Tag override - -- `input:"email"` → uses `input.Email()` instead of default -- `input:"textarea,required,min=5"` → override + Permitted modifiers -- `input:"required,min=5"` → keep default widget + modifiers only -- `input:"-"` → excluded from form (no widget) - -## Validation Flow (Same Code Front + Back) - -``` -fmt.ValidateFields(action, model) - → for each field in Schema(): - → field.Validate(value) - → 1. NotNull check (required) - → 2. Widget.Validate(value) — semantic (email format, date format) - → 3. Permitted.Validate(field, value) — characters, length -``` - -Runs identically in: -- **Frontend:** form.Validate() in WASM -- **Backend:** after orm.Insert() or manual validation - -## Example +A model is authored as a **typed `model.Definition` literal**. The var name +MUST end in `Model` — that is how `ormc` discovers it: ```go -// ormc:form -type User struct { - ID string `db:"pk"` // string → input.Text() - Email string `input:"email"` // override → input.Email() - Bio string `input:"textarea"` // override → input.Textarea() - Age int `db:"not_null"` // int → input.Number() - Active bool // bool → input.Checkbox() - SecretKey string `input:"-"` // excluded from form +import ( + "github.com/tinywasm/form/input" + "github.com/tinywasm/model" +) + +var LoginDataModel = model.Definition{ + Name: "login_data", + Fields: model.Fields{ + {Name: "email", Type: model.FieldText, NotNull: true, Widget: input.Email()}, + {Name: "password", Type: model.FieldText, NotNull: true, Widget: input.Password()}, + }, } -``` - -## Custom Inputs (web/input/) - -Projects can define custom widgets in `web/input/` that override stdlib inputs. ormc discovers them via AST scanning. Custom takes priority over stdlib. - -## Cross-Library Execution Order -When making changes that span libraries: - -1. **dom** first (remove form functions — unblocks clean separation) -2. **fmt** if interface changes needed (currently stable) -3. **form/input** (currently complete — Permitted cleanup done) -4. **orm** (pending: widget assignment + tag support) -5. **json** only if codec changes needed (currently stable) - -## ormc Usage in PLAN.md (for agents) +var ProductModel = model.Definition{ + Name: "product", + Fields: model.Fields{ + {Name: "id", Type: model.FieldInt, DB: &model.FieldDB{PK: true, AutoInc: true}}, + {Name: "name", Type: model.FieldText, NotNull: true, Permitted: model.Permitted{Minimum: 2}, Widget: input.Text()}, + {Name: "price", Type: model.FieldFloat, Widget: input.Number()}, + }, +} +``` -`ormc` is a CLI code generator. **Agents must never write `Schema()`, `Pointers()`, `ModelName()`, or `Validate()` by hand** — these are always generated. +- **DB tables**: fields carry `DB: &model.FieldDB{...}` metadata. +- **Form-only DTOs** (login, filters, …): NO `DB` metadata — same pattern, + just no table. +- **UI binding**: `Widget: input.Email()` etc. (typed expression from + `tinywasm/form/input`). A field without `Widget` gets NO input in forms. -### Installation +Then generate: ```bash +go generate ./... # runs ormc (//go:generate ormc); install: go install github.com/tinywasm/orm/cmd/ormc@latest ``` -### What ormc generates (per directive) - -| Directive | Generated methods | -|---|---| -| `// ormc:form` | `ModelName()`, `Schema()` (with widgets), `Pointers()`, `Validate()`, `*List` type | -| `// ormc:formonly` | `Schema()` (with widgets), `Pointers()`, `Validate()`, `*List` type — NO `ModelName()` | -| _(none)_ | `ModelName()`, `Schema()` (no widgets), `Pointers()`, `Validate()`, `*List` type | - -Output file: `model_orm.go` in the same package. **Never edit `model_orm.go` — it is overwritten on each `ormc` run.** - -### How to instruct an agent - -In `PLAN.md`, always say: - -1. Add the struct with the correct directive to `model.go` (or the appropriate model file). -2. Run `ormc` from the module root. -3. Do NOT write `Schema()`, `Pointers()`, `ModelName()`, or `Validate()` manually. +`ormc` parses the Definition literal (including the `Widget:` expressions) +and **generates the concrete Go struct** plus `ModelName()`, `Schema()`, +`Pointers()`, `Validate()`, `EncodeFields()`/`DecodeFields()` (typed codec) +and the `*List` type, into the generated `*_orm.go` file. **Never edit +generated files — they are overwritten on every run.** + +## FORBIDDEN (these are the bugs this skill exists to prevent) + +- ❌ **Struct tags as source of truth** (`input:"email"`, `db:"pk"` on + hand-written structs). The Definition literal is typed; tags are the old, + removed pattern. +- ❌ **Hand-writing the struct or any generated method** + (`Schema`, `Pointers`, `ModelName`, `Validate`, `Encode/DecodeFields`). +- ❌ **Stdlib `encoding/json`** anywhere. Only `tinywasm/json`, which works + exclusively through the generated typed codec — a hand-written DTO without + generated `Encode/DecodeFields` cannot travel. +- ❌ **`form.RegisterInput` as a fix for empty forms.** `form.New` binds + inputs ONLY via `Field.Widget` from the schema — there is no name matching. + If a form renders without fields, the Definition is missing `Widget:` + entries → fix the Definition and regenerate; never patch at the consumer. + +## Library roles + +| Library | Responsibility | Knows about | +|---|---|---| +| **model** | `Definition`, `Field`, `Fielder`, `Widget` iface, `Permitted`, `ValidateFields`, typed codec contracts | nothing else | +| **orm / ormc** | DB mapping + THE code generator (Definition → struct + methods) | model | +| **form** | `form.New(parentID, &Generated{})` — schema-driven form; SSR + reactive render | model, dom, form/input | +| **form/input** | Concrete widgets with validation (`Email`, `Password`, `Phone`, `Text`, `Number`, `Checkbox`, `Textarea`, …) | model | +| **json** | Zero-reflection codec over generated `Encode/DecodeFields` | model | +| **dom** | Pure HTML layout elements + signals. **NO form functions** (no `Input()`, `Form()`, …) | — | -Example instruction in a plan: +## Validation flow (same code front + back) ``` -Add to `modules/foo/model.go`: - - // ormc:form - type Foo struct { - ID int64 - Name string - } - -Then run: - - ormc - -ormc will generate Schema(), Pointers(), ModelName(), Validate() in model_orm.go. -Do NOT write these methods manually. +model.ValidateFields(action, m) + → per field: NotNull → Widget.Validate (semantic) → Permitted (chars/length) ``` -### ID field convention +Runs identically in WASM (`form.Validate()`) and on the server (after decode, +before persistence). + +## How to instruct an agent in a PLAN.md -`ormc` auto-detects `ID` as primary key (auto-increment). No `db:"pk"` tag needed for the standard `ID int64` pattern. Only add `db:` tags to override defaults. +1. Write the `model.Definition` literal(s) — var name ending in `Model`, + every form-visible field with an explicit `Widget:`. +2. Delete any hand-written struct the Definition replaces (ormc generates it; + names would collide). +3. Run `go generate ./...` (ormc). Verify the generated schema carries the + widgets and codecs. +4. Regression test: `form.New(id, &Generated{})` yields exactly the expected + number of inputs — catches a future regeneration that loses widgets. -## Plans Location +## Custom inputs -- dom: `tinywasm/dom/docs/PLAN.md` — pending: remove form functions -- form: `tinywasm/form/docs/PLAN.md` — complete -- orm: `tinywasm/orm/docs/PLAN.md` — pending execution -- Flow diagram: `tinywasm/orm/docs/diagrams/ORMC_FLOW.md` +A project may define custom widgets (same `model.Widget` contract) and use +them in its Definitions like any `input.Xxx()`. They live with the consumer; +stdlib widgets live in `tinywasm/form/input`.