process issue/pr comment collection as batches#425
Draft
MoralCode wants to merge 5 commits into
Draft
Conversation
this will help with a merge conflict later Signed-off-by: Adrian Edwards <adredwar@redhat.com>
Signed-off-by: Adrian Edwards <adredwar@redhat.com>
Signed-off-by: Adrian Edwards <adredwar@redhat.com>
Signed-off-by: Adrian Edwards <adredwar@redhat.com>
This helps adapt between the generator nature of the heavy API requests from the endpoint, and the list-focused processing function that does all sorts of additional queries and things. we cant do it inside the processing function because of all the extra queries - too risky Signed-off-by: Adrian Edwards <adredwar@redhat.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| from collectoss.tasks.init.celery_app import celery_app as celery | ||
| from collectoss.tasks.init.celery_app import CoreRepoCollectionTask | ||
| from collectoss.application.db.data_parse import * | ||
| from collectoss.tasks.github.util.github_data_access import GithubDataAccess, UrlNotFoundException |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused UrlNotFoundException imported from collectoss.tasks.github.util.github_data_access (unused-import)
| from collectoss.tasks.util.worker_util import batched, remove_duplicate_dicts | ||
| from collectoss.tasks.github.util.util import get_owner_repo | ||
| from collectoss.application.db.models import PullRequest, Message, Issue, PullRequestMessageRef, IssueMessageRef, Contributor, Repo, CollectionStatus | ||
| from collectoss.application.db import get_engine, get_session |
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused get_engine imported from collectoss.application.db (unused-import)
| @@ -2,7 +2,7 @@ | |||
| import pytest | |||
| import sqlalchemy as s | |||
There was a problem hiding this comment.
[pylint] reported by reviewdog 🐶
W0611: Unused sqlalchemy imported as s (unused-import)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR reuses the same code that we use for bulk-fetching comments for repos with few issues/PRs and applies it to larger repos.
This PR fixes #419
Notes for Reviewers
Just throwing it together, havent extensively tested yet.
might need to make sure it works for larger repos.
Might also want to block this until we have full-collection testing in place to verify that we dont hit some kind of pagination limit or something in the github API
Signed commits
Generative AI disclosure
Please select one option:
If AI tools were used, please provide details below:
- What tools were used? Sonnet 4.6 Medium
- How were these tools used? identifying the issue, writing the
batchedfunction and its unit tests- Did you review these outputs before submitting this PR? yes