Add sync_isilon_allocations management command#486
Merged
Conversation
- New coldfront/plugins/isilon/management/commands/sync_isilon_allocations.py: reconciles Isilon/PowerScale Directory SmartQuotas (hard-limited) against ColdFront Project/Allocation state, per-resource or via --resource flag - Add utils.py helpers: IsilonDirectoryQuota wraps the raw SDK quota object (path/cf_path/hard_limit_bytes/usage_bytes/has_hard_limit), plus is_isilon_path_ignored, get_directory_group, find_matching_pending_allocation, update_allocation_quota_and_usage, sync_allocation_for_quota, and sync_isilon_resource_allocations for the create/activate/update reconciliation - Add ISILON_PATH_IGNORE setting so specific unlimited-quota paths can be excluded from the "no hard limit" warning - Add 12 TestCase-based unit tests mocking IsilonConnection, covering the quota-with-no-limit, missing-project, existing-allocation-update, pending-request-activation, new-allocation-creation, and quota-unchanged/usage-refresh paths Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- find_matching_pending_allocation now also matches requests attached to resource.parent_resource (e.g. a bos-isilon quota should match a pending request against "Tier 1"), since storage requests are commonly made against the tier rather than the specific cluster a quota lands on - sync_allocation_for_quota repoints the activated allocation's resources from the tier to the specific resource once matched, so later syncs find it via the specific-resource lookup used elsewhere - Add test covering a Tier-1-attached pending request being activated and repointed to the specific cluster resource Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sync_isilon_resource_allocations was passing a None group_name straight into missing_projects when get_directory_group's ACL lookup couldn't resolve an owner (e.g. an orphaned/unmapped GID), which then crashed sorted(set(...)) in the management command since None isn't orderable against str - Treat an unresolved group name as its own report bucket (report['unresolved_group'], keyed by path) distinct from missing_projects (which now only ever holds known group names with no matching Project), and skip the project lookup entirely in that case - Add a regression test for a quota whose ACL group name is None Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- get_directory_group can return a domain-qualified name (e.g. 'RC\poisson_lab') rather than the bare group name ColdFront Project titles use, causing valid groups to be reported as missing_projects. Strip any 'DOMAIN\' prefix before returning the name. - Add tests for the prefixed, unprefixed, and unresolved (None) cases on get_directory_group directly, plus an end-to-end sync test confirming a domain-qualified name still matches its Project. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tions - sync_allocation_for_quota now writes a RequiresPayment AllocationAttribute mirroring resource.requires_payment whenever it creates a new Allocation or activates a pending allocation request - Activation uses update_or_create since a pending request may already carry a RequiresPayment value from when it was originally submitted, which should be corrected to match the resource the quota actually landed on - Existing (already-synced) allocations are untouched, since the ask was scoped to created/activated allocations only - Add coverage for both branches, including overwriting a stale value on activation and a resource with requires_payment=True Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sync_isilon_resource_allocations now tracks every quota path seen on a resource (hard-limited or not, resolved group or not - presence on the volume is what matters) and, after processing, deactivates any Active allocation on that resource whose recorded path isn't among them - deactivate_missing_allocations() sets status to Inactive, matching the existing precedent in pull_vast_quotas.py - Allocations with no recorded Subdirectory path are left alone, since absence from the volume isn't a meaningful signal for them - Add coverage for: missing-path deactivation, an allocation whose quota is still present (even without a hard limit) staying Active, and a pathless allocation being left alone Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sync_isilon_allocationsmanagement command that reconciles Isilon/PowerScale Directory SmartQuotas (hard-limited) against ColdFront Project/Allocation state — updates existing allocations, activates matching pending allocation requests, or creates new allocations, since nothing previously noticed a quota with no ColdFront-side recordIsilonDirectoryQuotawrapper class inisilon/utils.pyaround the raw SmartQuotas API object (path/cf_path/hard_limit_bytes/usage_bytes/has_hard_limit), avoiding redundant re-derivation of the same fields at multiple call sitesISILON_PATH_IGNOREsetting (opt-in, defaults to empty list) so specific unlimited-quota paths can be excluded from the "no hard limit" warningAllocationare only rewritten when actually changed, to avoid noisyHistoricalRecordschurn on every sync run, while usage is always refreshedTest plan
manage.py test coldfront.plugins.isilon— 12/12 pass (covers: no-hard-limit warning + ignore-list skip, group-with-no-matching-Project reporting, existing-allocation update without duplication, pending-request activation, new-allocation creation, quota-unchanged/usage-refresh behavior, andIsilonDirectoryQuotafield/cf_pathderivation)manage.py check— cleancoldfront.core.allocation+coldfront.core.project+coldfront.core.resource+coldfront.plugins.isilon(143 tests) — passcoldfront.plugins.api,coldfront.plugins.fasrc) are pre-existing onmaster(missingPLUGIN_API/ATT_VERIFYconfig in this environment, unrelated to this change) via stash-and-rerun comparison--resource <name>before relying on this in production