DAOS-11218 control: Add --wait flag to dmg pool rebuild commands by mjmac · Pull Request #11600 · daos-stack/daos

mjmac · 2023-03-02T13:41:41Z

Add an opt-in --wait flag to dmg pool exclude/drain/extend/reintegrate.
When supplied, dmg blocks until the rebuild triggered by the operation
reaches the DONE state, polling PoolQuery once per second.

Update a number of ftest sites that used wait_for_rebuild_to_end()
to use the new flag instead, both simplifying the test harness and
ensuring integration test coverage of the new flag.

Signed-off-by: Michael MacDonald github@macdonald.cx

github-actions · 2023-03-02T13:42:05Z

Bug-tracker data:
Ticket title is 'Improve pool reintegrate workflow'
Status is 'In Review'
Labels: 'usability'
https://daosio.atlassian.net/browse/DAOS-11218

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-03T16:13:27Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/1/execution/node/1250/log

daosbuild1 · 2023-03-03T21:20:32Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/1/execution/node/1297/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-04T10:15:27Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/2/execution/node/1031/log

daosbuild1 · 2023-03-06T14:46:26Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/3/execution/node/703/log

daosbuild1 · 2023-03-06T16:22:29Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/3/execution/node/719/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-08T20:34:56Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/4/execution/node/1031/log

daosbuild1 · 2023-03-09T12:02:11Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/4/execution/node/1254/log

daosbuild1 · 2023-03-09T12:34:24Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/4/execution/node/1301/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-10T03:24:07Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/5/execution/node/1297/log

daosbuild1 · 2023-03-10T04:32:26Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/5/execution/node/1250/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-18T04:22:08Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/6/execution/node/1274/log

daosbuild1 · 2023-03-18T04:24:59Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/6/execution/node/1189/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-03-30T07:07:08Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/7/execution/node/1274/log

daosbuild1 · 2023-03-30T07:26:33Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-11600/7/execution/node/1227/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-04-18T09:34:16Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-11600/8/display/redirect

daosbuild3 · 2026-05-31T14:03:12Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-11600/9/execution/node/1263/log

daosbuild3 · 2026-06-01T08:59:18Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/10/testReport/

daosbuild3 · 2026-06-02T15:51:29Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/11/testReport/

daosbuild3 · 2026-06-05T22:32:31Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/12/testReport/

daosbuild3 · 2026-06-12T01:53:02Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/13/testReport/

Add an opt-in --wait flag to dmg pool exclude/drain/extend/reintegrate. When supplied, dmg blocks until the rebuild triggered by the operation reaches the DONE state, polling PoolQuery once per second. Update a number of ftest sites that used wait_for_rebuild_to_end() to use the new flag instead, both simplifying the test harness and ensuring integration test coverage of the new flag. Signed-off-by: Michael MacDonald <github@macdonald.cx>

tanabarr

go changes LGTM but I think @kccain needs to review to see how it fits in with his recent work.

kccain · 2026-06-17T19:28:14Z

go changes LGTM but I think @kccain needs to review to see how it fits in with his recent work.

Thanks @tanabarr . I will need more time to review this in detail and can follow up with any deeper rebuild comments later (if there are any to add). Initial quick questions and feedback:

What is the intended exit-code contract for the wait behavior after a successful initial command (reintegrate, drain, extend, exclude)? Should command success be based only on detecting rebuild state done, or should it also consider the final rebuild error value from pool query?
Should wait stop for both successful and failed rebuild completions, or only for successful completion? I'm most interested in drain, extend, and reintegrate, where failed rebuilds ultimately end in done with a nonzero rebuild error.
Is there a unit test for the case where rebuild reaches done but reports a nonzero rebuild error?
What user-facing output is intended while waiting? Nothing until completion, periodic progress/state output, or final-state-only output?
In ftest coverage for this option, are checks based on machine-readable JSON output or human-readable text parsing? Tom did some work recently to improve the human-readable output to reflect states such as "failing" and "failed" for when a nonzero rebuild error number is seen.

DAOS-18347 added a new DerivedState to the rebuild response struct. Use this to determine the exit status of a rebuild rather than the simple State field. Signed-off-by: Michael MacDonald <github@macdonald.cx>

mjmac · 2026-06-22T15:40:31Z

Hi @kccain, thanks for the feedback.

Initial quick questions and feedback:

What is the intended exit-code contract for the wait behavior after a successful initial command (reintegrate, drain, extend, exclude)? Should command success be based only on detecting rebuild state done, or should it also consider the final rebuild error value from pool query?

I've updated the PR to use DerivedState as the controlling state. I hadn't seen that change earlier when I revived this patch. So now we'll only get exit 0 on a fully successful operation, not failed or stopped.

Should wait stop for both successful and failed rebuild completions, or only for successful completion? I'm most interested in drain, extend, and reintegrate, where failed rebuilds ultimately end in done with a nonzero rebuild error.

I would think it should stop for both, in order to notify the operator if the operation didn't complete successfully for any reason, no?

Is there a unit test for the case where rebuild reaches done but reports a nonzero rebuild error?

Yes, added with the rework.

What user-facing output is intended while waiting? Nothing until completion, periodic progress/state output, or final-state-only output?

No output until completion. There may be an opportunity to add progress output for long running commands later, but it should be done as part of an overall feature design rather than piecemeal per command. There is a design somewhere for adding tracking support to long running commands that would dovetail well with UX/UI updates.

In ftest coverage for this option, are checks based on machine-readable JSON output or human-readable text parsing? Tom did some work recently to improve the human-readable output to reflect states such as "failing" and "failed" for when a nonzero rebuild error number is seen.

The ftest stuff is mostly JSON based in order to avoid fragile text parsing. There should already be unit tests for the pretty printers that produce human-readable output.

daosbuild3 · 2026-06-23T02:51:07Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/15/testReport/

daosbuild3 · 2026-06-27T16:11:48Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-11600/16/execution/node/1569/log

daosbuild3 · 2026-07-03T00:37:15Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-11600/17/execution/node/1796/log

daosbuild3 · 2026-07-03T03:00:25Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-11600/17/testReport/

daosbuild3 · 2026-07-06T09:52:47Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-11600/18/execution/node/1653/log

mjmac requested a review from a team as a code owner March 2, 2023 13:41

daosbuild1 reviewed Mar 2, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 05241a4 to c0c3dda Compare March 3, 2023 22:10

daosbuild1 reviewed Mar 3, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from c0c3dda to 15ce0d7 Compare March 8, 2023 14:10

daosbuild1 reviewed Mar 8, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 15ce0d7 to 251ca17 Compare March 9, 2023 13:28

daosbuild1 reviewed Mar 9, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 251ca17 to 7a08327 Compare March 17, 2023 13:09

daosbuild1 reviewed Mar 17, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 7a08327 to 9985c11 Compare March 28, 2023 19:10

daosbuild1 reviewed Mar 28, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 9985c11 to 6345c0b Compare April 17, 2023 21:23

daosbuild1 reviewed Apr 17, 2023

View reviewed changes

mjmac force-pushed the mjmac/DAOS-11218 branch from 0a36fa8 to d86f3c4 Compare May 31, 2026 14:01

mjmac force-pushed the mjmac/DAOS-11218 branch from d86f3c4 to 34714c8 Compare June 2, 2026 14:47

mjmac requested review from kjacque and tanabarr June 2, 2026 15:25

mjmac force-pushed the mjmac/DAOS-11218 branch from 34714c8 to 263be86 Compare June 4, 2026 14:46

mjmac force-pushed the mjmac/DAOS-11218 branch from 263be86 to ff20304 Compare June 11, 2026 14:24

mjmac force-pushed the mjmac/DAOS-11218 branch from ff20304 to cf92bd6 Compare June 15, 2026 17:45

mjmac marked this pull request as ready for review June 16, 2026 21:13

mjmac requested review from a team as code owners June 16, 2026 21:13

tanabarr previously approved these changes Jun 17, 2026

View reviewed changes

Use DerivedState in PoolRebuildResp to determine status

b844b76

DAOS-18347 added a new DerivedState to the rebuild response struct. Use this to determine the exit status of a rebuild rather than the simple State field. Signed-off-by: Michael MacDonald <github@macdonald.cx>

mjmac dismissed tanabarr’s stale review via b844b76 June 22, 2026 15:28

Merge branch 'master' into mjmac/DAOS-11218

acf2eee

Merge branch 'master' into mjmac/DAOS-11218

8bfb101

Merge branch 'master' into mjmac/DAOS-11218

7703f06

Uh oh!

Conversation

mjmac commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 3, 2023

Uh oh!

daosbuild1 commented Mar 3, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 4, 2023

Uh oh!

daosbuild1 commented Mar 6, 2023

Uh oh!

daosbuild1 commented Mar 6, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 8, 2023

Uh oh!

daosbuild1 commented Mar 9, 2023

Uh oh!

daosbuild1 commented Mar 9, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 10, 2023

Uh oh!

daosbuild1 commented Mar 10, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 18, 2023

Uh oh!

daosbuild1 commented Mar 18, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Mar 30, 2023

Uh oh!

daosbuild1 commented Mar 30, 2023

Uh oh!

daosbuild1 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild1 commented Apr 18, 2023

Uh oh!

daosbuild3 commented May 31, 2026

Uh oh!

daosbuild3 commented Jun 1, 2026

Uh oh!

daosbuild3 commented Jun 2, 2026

Uh oh!

daosbuild3 commented Jun 5, 2026

Uh oh!

daosbuild3 commented Jun 12, 2026

Uh oh!

tanabarr left a comment

Choose a reason for hiding this comment

Uh oh!

kccain commented Jun 17, 2026

Uh oh!

mjmac commented Jun 22, 2026

Uh oh!

daosbuild3 commented Jun 23, 2026

Uh oh!

daosbuild3 commented Jun 27, 2026

Uh oh!

mjmac commented Mar 2, 2023 •

edited

Loading

github-actions Bot commented Mar 2, 2023 •

edited

Loading