Skip to content

Push selected functions in parallel - #1801

Merged
ChiragAgg5k merged 2 commits into
mainfrom
fix/parallel-function-deployments
Aug 15, 2026
Merged

Push selected functions in parallel#1801
ChiragAgg5k merged 2 commits into
mainfrom
fix/parallel-function-deployments

Conversation

@ChiragAgg5k

@ChiragAgg5k ChiragAgg5k commented Aug 15, 2026

Copy link
Copy Markdown
Member

Closes #1794.

Summary

  • package, upload, and monitor selected functions with bounded concurrency
  • keep worker summaries isolated, then merge them in config order
  • serialize parallel status and build-log writes so terminal output stays intact
  • preserve the existing animated spinner for single-function pushes
before: function A upload + build -> function B upload + build
after:  function A upload + build
        function B upload + build  (in parallel)

Implementation

The function push now sends every selected entry through the parallel worker while keeping sites and single-function pushes on the existing sequential path:

run := deployRun{
    Async:         options.Async,
    Code:          pushCode,
    Activate:      activate,
    WithVariables: options.WithVariables,
    Logs:          options.Logs && !options.Async,
    LabelLogs:     len(entries) > 1,
}
push := func(entry *jsonx.Object, summary *pushSummary) {
    context.pushDeployable(command, resource, entry, run, summary)
}

summary := pushSummary{}
if resource.Name == "function" && len(entries) > 1 {
    command.SetOut(output.Synchronized(command.OutOrStdout()))
    command.SetErr(output.Synchronized(command.ErrOrStderr()))
    summary = pushDeployablesInParallel(entries, push)
} else {
    for _, entry := range entries {
        push(entry, &summary)
    }
}

summary.report(command, resource, options.Async, time.Since(started))

A four-worker pool bounds nested packaging and chunk-upload pressure. Each entry owns a summary, and results are merged in config order after all deployments finish:

const functionPushConcurrency = 4

func pushDeployablesInParallel(
    entries []*jsonx.Object,
    push func(*jsonx.Object, *pushSummary),
) pushSummary {
    results := make([]pushSummary, len(entries))
    jobs := make(chan int)

    workerCount := min(functionPushConcurrency, len(entries))
    var workers sync.WaitGroup
    workers.Add(workerCount)
    for range workerCount {
        go func() {
            defer workers.Done()
            for index := range jobs {
                push(entries[index], &results[index])
            }
        }()
    }

    for index := range entries {
        jobs <- index
    }
    close(jobs)
    workers.Wait()

    summary := pushSummary{}
    for _, result := range results {
        summary.Pushed += result.Pushed
        summary.Deployed += result.Deployed
        summary.Failed = append(summary.Failed, result.Failed...)
    }

    return summary
}

Concurrent output is serialized. Hiding the underlying terminal also makes multi-function pushes use plain progress lines instead of multiple spinners redrawing the same row:

type synchronizedWriter struct {
    writer io.Writer
    mutex  sync.Mutex
}

func (w *synchronizedWriter) Write(contents []byte) (int, error) {
    w.mutex.Lock()
    defer w.mutex.Unlock()

    return w.writer.Write(contents)
}

func Synchronized(writer io.Writer) io.Writer {
    if _, ok := writer.(*synchronizedWriter); ok {
        return writer
    }

    return &synchronizedWriter{writer: writer}
}

Production verification

Built examples/cli/appwrite and pushed two disposable Node functions to Main Project in the SGP production region. The output shows both function pushes and deployment requests in flight together:

ℹ Info: Pushing functions ...
ℹ Info: Pushing function SDK Generator 1794 First ( e2e-1794-first-221334 ) ...
ℹ Info: Pushing function SDK Generator 1794 Second ( e2e-1794-second-221334 ) ...
· GET /v1/functions/e2e-1794-second-221334 404 in 133ms
· GET /v1/functions/e2e-1794-first-221334 404 in 364ms
· POST /v1/functions 201 in 623ms
· POST /v1/functions 201 in 604ms
· GET /v1/proxy/rules 200 in 419ms
· GET /v1/proxy/rules 200 in 402ms
· POST /v1/proxy/rules/function 201 in 575ms
· POST /v1/proxy/rules/function 201 in 656ms
· POST /v1/functions/e2e-1794-first-221334/deployments 202 in 523ms
· POST /v1/functions/e2e-1794-second-221334/deployments 202 in 3.743s
✓ Success: Successfully pushed 2 functions.

The API recorded both deployments in the same second:

first deployment:  2026-08-15T16:43:44.604Z
second deployment: 2026-08-15T16:43:44.853Z
difference:        249 ms

Both deployments reached ready. Both executions completed with HTTP 200 and returned their expected JSON responses:

{"function":"e2e-1794-first-221334","ok":true}
{"function":"e2e-1794-second-221334","ok":true}

The test functions and their proxy rules were deleted afterward.

Test plan

  • php example.php cli
  • cd examples/cli && go mod tidy && go build ./... && go vet ./... && go test ./...
  • cd examples/cli && go test -race ./internal/cmd ./internal/output
  • vendor/bin/phpunit tests/e2e/CLIGo126Test.php
  • composer lint-twig
  • composer refactor:check
  • Production SGP deployment and execution verification described above

@greptile-apps

greptile-apps Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes multi-function pushes concurrent while preserving sequential behavior for sites and single-function pushes.

  • Uses a four-worker bounded pool to package, upload, and monitor selected functions.
  • Maintains per-function summaries and merges results in configuration order.
  • Synchronizes concurrent terminal writes and adds regression coverage for parallelism and its concurrency limit.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously unbounded deployment worker count is now capped at four and covered by a concurrency-limit regression test.

Important Files Changed

Filename Overview
templates/cli/internal/cmd/pushdeploy.go Replaces unbounded function goroutines with a four-worker pool, resolving the previously reported resource-pressure issue while retaining deterministic summary aggregation.
templates/cli/internal/cmd/pushselect_test.go Adds coverage proving selected deployments overlap while no more than four deployment workers run concurrently.
templates/cli/internal/output/message.go Adds a mutex-backed writer wrapper so concurrent status and build-log writes remain intact.

Reviews (2): Last reviewed commit: "address greptile review feedback (greplo..." | Re-trigger Greptile

Comment thread templates/cli/internal/cmd/pushdeploy.go Outdated
@ChiragAgg5k
ChiragAgg5k merged commit b02a0c0 into main Aug 15, 2026
59 checks passed
@ChiragAgg5k
ChiragAgg5k deleted the fix/parallel-function-deployments branch August 15, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report:

1 participant