[KYLIN-6087] Support built-in auto scheduled segment build with model-level config and scheduler#2347
Open
Mrhs121 wants to merge 3 commits into
Open
[KYLIN-6087] Support built-in auto scheduled segment build with model-level config and scheduler#2347Mrhs121 wants to merge 3 commits into
Mrhs121 wants to merge 3 commits into
Conversation
…el config and scheduler
…el config and scheduler
Contributor
Author
|
retest this please |
Contributor
Author
|
@jlfsdtc PTAL, Thanks |
Contributor
|
Please add "dev design" and create an Apache issue at the same time. @Mrhs121 |
Mrhs121
changed the title
fishcus
reviewed
Jul 23, 2026
| } | ||
| val parts = key.split("/", 2); | ||
| if (parts.length != 2) { | ||
| return; |
Contributor
There was a problem hiding this comment.
add error log, identify the abnormal key
fishcus
reviewed
Jul 23, 2026
| @Getter | ||
| private final AtomicInteger schedulerModelCount = new AtomicInteger(0); | ||
|
|
||
| @Scheduled(cron = "*/30 * * * * ?") |
Contributor
There was a problem hiding this comment.
Why is it necessary to schedule this operation every 30 minutes? or this crontab expression should be configurable.
fishcus
reviewed
Jul 23, 2026
| val projectName = project.getName(); | ||
| val modelManager = NDataModelManager.getInstance(KylinConfig.readSystemKylinConfig(), projectName); | ||
| for (NDataModel model : modelManager.listAllModels()) { | ||
| val segmentConfig = model.getSegmentConfig(); |
Contributor
There was a problem hiding this comment.
skip the offline model or broken model？
fishcus
reviewed
Jul 23, 2026
|
|
||
| private void startCron(String key, String cron) { | ||
| stopCron(key); | ||
| checkSchedulerThreadPoolSize(); |
Contributor
There was a problem hiding this comment.
check every time of startCron or every time of schedule？
fishcus
reviewed
Jul 23, 2026
| if (autoSegmentBuild == null || !autoSegmentBuild.isEnabled()) { | ||
| return; | ||
| } | ||
| if (model.isStreaming() || model.isMultiPartitionModel() |
Contributor
There was a problem hiding this comment.
model‘s status has been checked when walking through the model list in function reconcile，combine these into one.
fishcus
reviewed
Jul 23, 2026
| JobTypeEnum[] buildJobTypes = JobTypeEnum.getJobTypeByCategory(JobTypeEnum.Category.BUILD) | ||
| .toArray(new JobTypeEnum[0]); | ||
| List<AbstractExecutable> jobs = executableManager.listExecByModelAndStatus(modelId, | ||
| ExecutableState::isProgressing, buildJobTypes); |
Contributor
There was a problem hiding this comment.
hasRunningBuildJob of the same segment or any segment？
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.