Skip to content

feat(task): add path-prefix batch cancel, delete and retry#2846

Open
ifloppy wants to merge 2 commits into
OpenListTeam:mainfrom
ifloppy:feat/task-batch-by-path
Open

feat(task): add path-prefix batch cancel, delete and retry#2846
ifloppy wants to merge 2 commits into
OpenListTeam:mainfrom
ifloppy:feat/task-batch-by-path

Conversation

@ifloppy

@ifloppy ifloppy commented Jul 26, 2026

Copy link
Copy Markdown

Summary / 摘要

Adds path-prefix batch operations for tasks, so large numbers of tasks can be cancelled, deleted or retried without selecting them one by one.

为任务增加按路径前缀的批量操作,使大量任务无需逐条勾选即可取消、删除或重试。

Motivation: with tens of thousands of queued tasks, the existing delete_some / cancel_some endpoints require sending every task ID, and clear_done cannot filter by path. Cleaning up all tasks under one directory was impractical.

动机:在有上万条排队任务时,现有 delete_some / cancel_some 需要传入每个任务 ID,而 clear_done 无法按路径过滤,导致按目录清理任务难以实施。

User-visible changes / 用户可感知的变化:

  • New endpoints for every task type (upload, copy, move, offline_download, offline_download_transfer, decompress, decompress_upload):
    • POST /api/task/{type}/delete_by_path
    • POST /api/task/{type}/cancel_by_path
    • POST /api/task/{type}/retry_by_path
  • Request body: {"path": "/prefix"}
  • Response distinguishes tasks selected from operations completed: {"matched": N, "processed": M}
  • A task matches when its source or final destination object path equals the prefix or is under it
  • delete_by_path cancels all matching unfinished tasks first, waits for active execution and cleanup to finish, then removes only tasks confirmed safe to remove
  • cancel_by_path only affects unfinished tasks; retry_by_path only affects failed tasks
  • A global root operation requires an explicit /; ambiguous values such as ., ./, or // are rejected if they resolve to /

Implementation changes / 重要实现变化:

  • Added task.TaskWithPaths interface and task.MatchTaskPath helper

  • Implemented final-object GetSrcPath / GetDstPath values on TaskData, UploadTask, ArchiveContentUploadTask and DownloadTask

  • Added two-phase running-task deletion: cancel all, wait up to the shared deadline for terminal state/cleanup, then remove confirmed stopped tasks

  • Matching reuses utils.IsSubPath, so /a does not match /ab

  • Local temp paths and offline-download URLs are excluded from matching to avoid unintended deletions

  • Non-admin requests are resolved through user.JoinPath, and matching is limited to the requesting user's own tasks

  • Existing endpoints are untouched

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。

  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。

  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

Testing / 测试

Commands run on Windows (go1.26.4):

  • go test ./internal/task/ ./internal/fs/ ./internal/offline_download/tool/ ./server/handles/ ./pkg/utils/ — all pass
  • go vet and gofmt -l on the changed packages — clean
  • go build ./... — succeeds

Automated coverage added:

  • Path matching: prefix, exact match, /a vs /ab, backslash and relative path cleaning, empty path sides
  • Final destination paths: exact normal-upload file and archive-content object paths
  • Root safety: ., ./, //, and repeated separators are rejected when they would resolve to global /; explicit / is accepted
  • State filtering: cancel_by_path skips terminal states, retry_by_path only touches failed tasks
  • Permissions (HTTP level, via httptest): a non-admin cannot affect another user's tasks, and path traversal such as ../../etc/secret cannot reach tasks outside the user's base path
  • Lifecycle: a genuinely running task blocks until cancellation; deletion is asserted to wait until Run exits and delayed OnFailed cleanup finishes, and the task remains inactive after removal
  • Count semantics: concurrent removal after initial matching produces matched=1, processed=0
  • Scale: 10,000 tasks with 3,000 matching the prefix; the matching subset is deleted in a few milliseconds and the rest are left intact

Manual test / 手动测试:

  • Built a linux/amd64 binary with the frontend embedded and ran it on Linux
  • Verified all seven task types expose the three new endpoints
  • Verified an empty path is rejected with HTTP 400 path is required
  • Verified existing undone, done, clear_done and delete_some still respond normally

Note: go test ./... also reports pre-existing failures in drivers/189, drivers/chaoxing, drivers/google_drive, drivers/lanzou, pkg/aria2/rpc and internal/net. These are unrelated to this change; I confirmed they reproduce identically with this branch's changes stashed (non-constant format string vet checks from a newer Go, and tests requiring a local aria2 instance).

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmtgo fmtprettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明):

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

The commits in this PR do not carry a Co-Authored-By trailer. The AI assistance is disclosed here instead. Please let me know if you would prefer the trailer added to the commits, and I will amend them.

此 PR 的提交未附带 Co-Authored-By trailer,AI 辅助情况改在此处声明。如维护者希望在提交中补充该 trailer,请告知,我会修改提交。

- Add `TaskWithPaths` interface and `MatchTaskPath` helper for matching tasks by virtual path prefix
- Expose `GetSrcPath`/`GetDstPath` on transfer, upload, download and archive tasks
- Add `delete_by_path`, `cancel_by_path` and `retry_by_path` endpoints for every task type
- Cancel unfinished tasks before removing them so queued work stops
- Scope matching to the requesting user and their base path
- Exclude local temp and URL sources from path matching
- Add unit and HTTP-level tests covering matching, permissions and large task sets

Signed-off-by: ifloppy <68799904+ifloppy@users.noreply.github.com>
@xrgzs xrgzs added enhancement Module: Task Task, scheduling and other goroutine-based features related labels Jul 26, 2026
@jyxjjj

jyxjjj commented Jul 27, 2026

Copy link
Copy Markdown
Member

I reviewed the path matching, permission boundaries, state filtering, and test coverage. The overall direction looks reasonable, but I have several concerns before this can be merged.

  1. Some task types expose only the destination directory, not the final destination object path.

For normal uploads, the destination path omits the uploaded file name. Archive-content uploads similarly omit the object name. As a result, filtering by an exact destination file path will not match the corresponding task, even though the API describes this as matching the task's source or destination path.

Please either include the final object name in these paths and add tests for exact-file matching, or explicitly define and document that these task types use destination-directory semantics.

  1. The running-task deletion lifecycle is not covered.

delete_by_path cancels an unfinished task and immediately removes it from the manager. The current tests use a manager with execution disabled, so they do not demonstrate that active I/O has stopped before the task disappears or that temporary resources are cleaned up safely.

Please add a test with a genuinely running/blocking task that verifies cancellation is observed, execution stops, cleanup completes, and the task is then removed without continuing in the background.

  1. Ambiguous inputs can normalize to the root path.

Inputs such as . or repeated separators are non-empty but normalize to /. The frontend confirmation currently displays the original input, so an administrator could confirm deletion under . without being clearly told that it matches every task.

Please require an explicit / for root-wide operations, or return/display the normalized path before performing a destructive action.

  1. Please clarify the response count semantics.

The returned count currently represents tasks selected by the initial filter, not necessarily operations that completed successfully. Concurrent task changes can make count differ from the number actually cancelled, removed, or retried. Please either document it as a matched count or return a result that distinguishes matched and successfully processed tasks.

  1. This adds public endpoints, but the related documentation entry is still empty.

Please add or link API documentation covering path normalization, source-or-destination matching, user scoping, root-path behavior, state filtering, and count semantics.

The existing tests cover matching boundaries, ownership, traversal attempts, state filtering, and scale well. Once the lifecycle and path-contract questions above are addressed, the backend and OpenList-Frontend#608 should be reviewed and merged as one coordinated change.

- Match upload and archive-content tasks by their final destination object path
- Require an explicit slash for operations that resolve to the global root
- Wait for running task cancellation and cleanup before removing tasks
- Return separate matched and processed counts for concurrent state changes
- Document normalization, scoping, state filters, lifecycle and response semantics
- Cover exact-file matching, lifecycle cleanup, implicit root rejection and count races

Signed-off-by: ifloppy <68799904+ifloppy@users.noreply.github.com>
@ifloppy

ifloppy commented Jul 27, 2026

Copy link
Copy Markdown
Author

Thank you for the detailed review. I addressed each concern in follow-up commit e8879b3, with the corresponding frontend adjustment in OpenList-Frontend#608 commit ef01089.

  1. Final destination object paths

    • Normal upload paths now include file.GetName().
    • Archive-content non-in-place tasks now include ObjName; in-place directory orchestration keeps destination-directory semantics, while its generated child tasks expose their final object paths.
    • Added exact-file matching tests for both normal uploads and archive-content uploads.
  2. Running-task deletion lifecycle

    • delete_by_path now uses two phases: cancel all matching unfinished tasks first, then wait for active tasks to reach a terminal state before removal.
    • Terminal state is set only after the worker's OnFailed cleanup hook completes, so active tasks are not removed while cleanup is still running.
    • Tasks that do not stop before the shared 30-second deadline remain in the manager and are not counted as processed.
    • Added a test with a genuinely running/blocking task. It verifies cancellation is observed, Run exits, delayed OnFailed cleanup completes, the task is then removed, and execution is not active afterward.
  3. Inputs normalizing to root

    • A global-root operation now requires the explicit input /.
    • Non-empty inputs such as ., ./, //, or repeated separators are rejected with HTTP 400 if their resolved path is /.
    • The frontend also blocks ambiguous administrator inputs before confirmation. Non-admin . remains valid when it resolves to that user's non-root base path.
  4. Response count semantics

    • Replaced count with { matched, processed }.
    • matched is the initial path/ownership/state-filter selection.
    • processed counts state-revalidated operations actually applied; for deletion it counts tasks confirmed absent from the manager after cancellation and cleanup.
    • Added a concurrent-change test that returns matched=1, processed=0 when the selected task disappears before processing.
    • OpenList-Frontend#608 now displays both values.
  5. API documentation

    • Added docs/task-path-batch-api.md, covering normalization, explicit root behavior, source-or-destination matching, final-object semantics, user scoping, state filters, running-task lifecycle, timeout behavior, and matched/processed semantics.
    • Linked it from both related PR descriptions.

Validation performed:

  • go test ./internal/task/ ./internal/fs/ ./internal/offline_download/tool/ ./server/handles/ ./pkg/utils/
  • go vet on all changed Go packages
  • go build ./...
  • frontend pnpm build and Prettier checks

All of the above pass. The follow-up implementation and this response were AI-assisted with Claude, as disclosed in the updated PR AI Disclosure section, and the changes/tests were reviewed and validated before pushing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Module: Task Task, scheduling and other goroutine-based features related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants