feat: Add opt in persistent storage for the aggregator#251
Merged
aa1ex merged 8 commits intoJul 10, 2026
Conversation
Add a persistence block to the shared aggregator spec. When enabled the aggregator will run as a StatefulSet with a persistent volume per replica so disk buffers survive pod restart and reschedule. Includes the API type, defaults, the persistenceEnabled helper, and regenerated deepcopy and CRD manifests.
Add the StatefulSet workload builder and branch the reconcile so the aggregator is rendered as a StatefulSet with a volume claim template per replica when persistence is enabled, otherwise it stays a Deployment. The data volume now comes from the volume claim template in persistent mode, so the hostPath data volume is only added for the Deployment path. The StatefulSet references a headless governing service by name.
Assigning the whole desired spec on every reconcile fails once a StatefulSet uses volume claim templates, since Kubernetes rejects changes to volumeClaimTemplates, serviceName, selector, and podManagementPolicy. Keep those fields from the live object on update and only assign the mutable ones. Replicas are preserved when the desired count is nil so the operator does not fight the autoscaler, and an immutable change is logged as a warning since it needs the aggregator to be recreated.
Add a headless governing service in persistent mode so the StatefulSet has stable per replica DNS, and point the HPA at the StatefulSet instead of the Deployment when persistent. Both aggregator controllers now own StatefulSets and have the RBAC to manage them, so the operator reconciles on StatefulSet changes and can create them.
Add builder unit tests for the StatefulSet path and an e2e test that deploys a persistence enabled aggregator and checks it becomes a StatefulSet with a bound per replica volume and a headless service. Document the persistence field, the Deployment versus StatefulSet rule, and why volume resizing is left manual. Sync the helm chart CRDs and add a changelog entry.
Extract the shared pod template so the Deployment and StatefulSet builders no longer duplicate it. Delete the opposite workload when persistence is toggled, so switching modes does not leave a Deployment and a StatefulSet serving side by side. Reject a raw volumeClaimTemplates escape hatch that has no claim named data, since the container mounts that volume, and make the name a single constant. Align the resize docs with the log behavior.
Contributor
Author
|
E2E failure looks like an intermittent dependency fetch failure, unrelated. Might just need a rerun? |
stigglor
changed the title
aa1ex
reviewed
Jul 9, 2026
aa1ex
left a comment
aa1ex
left a comment
Contributor
There was a problem hiding this comment.
Really nice work - clean, well documented and well tested, and it does exactly what it says. Left one small optional note inline on deleteObsoleteWorkload.
deleteObsoleteWorkload issued a DELETE for the opposite workload kind on every reconcile, which just returned NotFound in steady state. Read from the Owns() cache first and only delete when the workload exists. Add tests covering the delete and the skip paths.
aa1ex
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
persistenceoption toVectorAggregatorandClusterVectorAggregator. When you enable it, the operator renders the aggregator as a StatefulSet with a persistent volume per replica instead of the default Deployment, so the Vectordata_dirgets durable, stable per replica storage.The reason is that Vector's disk buffer lives under
data_dirand only survives a restart if that directory is persistent and tied to the same replica. Today the aggregator is a Deployment on a node local hostPath, so a rescheduled pod comes back with a foreign or empty directory. This is the same split Vector's own Helm chart makes, where the durable Aggregator role is a StatefulSet with a per replica PVC.This gives you the storage only. Disk buffering still has to be enabled per sink with
buffer.type = disk, the operator does not set it for you.Closes #250