Skip to content

Skip comments that return a 410 Gone error from github as well. #332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MoralCode wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Skip comments that return a 410 Gone error from github as well. #332

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion collectoss/tasks/github/messages.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from collectoss.tasks.init.celery_app import celery_app as celery
from collectoss.tasks.init.celery_app import CoreRepoCollectionTask
from collectoss.application.db.data_parse import *
from collectoss.tasks.github.util.github_data_access import GithubDataAccess, UrlNotFoundException
from collectoss.tasks.github.util.github_data_access import GithubDataAccess, UrlNotFoundException, ResourceGoneException
from collectoss.tasks.github.util.github_task_session import GithubTaskManifest
from collectoss.tasks.util.worker_util import remove_duplicate_dicts
from collectoss.tasks.github.util.util import get_owner_repo
Expand Down Expand Up @@ -127,6 +127,9 @@ def process_large_issue_and_pr_message_collection(repo_id, repo_git: str, logger
except UrlNotFoundException:
logger.info(f"{task_name}: PR or issue comment url of {comment_url} returned 404. Skipping.")
skipped_urls += 1
except ResourceGoneException:
logger.info(f"{task_name}: PR or issue comment url of {comment_url} returned 410. Skipping.")
skipped_urls += 1

if len(all_data) >= message_batch_size:
process_messages(all_data, task_name, repo_id, logger, db_session)
Expand Down
Loading