Handle moved repositories with no redirect (404) when looking for commits in `get_login_with_commit_hash` phase of contributor resolution

[cdolfi]

there are UrlNotFoundExceptions during the commit look up in the get_login_with_commit_hash function. I think some basic error handling would solve this issue
/

Details

Stack Trace

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 248, in insert_facade_contributors
    process_commit_metadata(logger, key_auth, batch, repo_id, platform_id, tool_source, tool_version, data_source)
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 66, in process_commit_metadata
    login = get_login_with_commit_hash(logger, auth, contributor, repo_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py", line 371, in get_login_with_commit_hash
    login_json = github_data_access.get_resource(url)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 108, in get_resource
    response = self.make_request_with_retries(url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 164, in make_request_with_retries
    return self.__make_request_with_retries(url, method, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 330, in wrapped_f
    return self(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 467, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 186, in __make_request_with_retries
    result = self.make_request(url, method, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 134, in make_request
    raise UrlNotFoundException(f"Could not find {url}")
collectoss.tasks.github.util.github_data_access.UrlNotFoundException: Could not find https://api.github.com/repos/TheAlaskanPhysicist/simulation-physics-droplet/commits/427e12127328230ebb6a931029590ec4bf597d52

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 439, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 2d71d7c2-aa42-49f7-a59b-8a1b42bfe05d raised UrlNotFoundException('Could not find https://api.github.com/repos/TheAlaskanPhysicist/simulation-physics-droplet/commits/427e12127328230ebb6a931029590ec4bf597d52')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 470, in trace_task
    I, R, state, retval = on_error(task_request, exc)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 381, in on_error
    R = I.handle_error_state(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 175, in handle_error_state
    return {
           ^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 227, in handle_failure
    task.backend.mark_as_failure(
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 172, in mark_as_failure
    self.on_chord_part_return(request, state, exc)
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 552, in on_chord_part_return
    return self.chord_error_from_stack(callback, exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 304, in chord_error_from_stack
    return backend.fail_from_current_stack(callback.id, exc=exc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 311, in fail_from_current_stack
    self.mark_as_failure(task_id, exc, exception_info.traceback)
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 167, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 526, in store_result
    self._store_result(task_id, result, state, traceback,
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 975, in _store_result
    current_meta = self._get_task_meta_for(task_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 997, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 871, in get_key_for_task
    raise ValueError(f'task_id must not be empty. Got {task_id} instead.')
ValueError: task_id must not be empty. Got None instead

[cdolfi]

This exception also happening in collect_pull_requests

Stack Trace

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/pull_requests/tasks.py", line 49, in collect_pull_requests
    for pr in retrieve_all_pr_data(repo_git, logger, manifest.key_auth, core_data_last_collected):
  File "/collectoss/collectoss/tasks/github/pull_requests/tasks.py", line 82, in retrieve_all_pr_data
    num_pages = github_data_access.get_resource_page_count(url)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 91, in get_resource_page_count
    response = self.make_request_with_retries(url, method="HEAD")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 164, in make_request_with_retries
    return self.__make_request_with_retries(url, method, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 330, in wrapped_f
    return self(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 467, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 186, in __make_request_with_retries
    result = self.make_request(url, method, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 134, in make_request
    raise UrlNotFoundException(f"Could not find {url}")
collectoss.tasks.github.util.github_data_access.UrlNotFoundException: Could not find https://api.github.com/repos/ritstudentgovernment/pawprints/pulls?state=all&direction=desc&sort=updated

[MoralCode]

https://github.com/TheAlaskanPhysicist/simulation-physics-droplet repo doesnt exist (404)

https://github.com/ritstudentgovernment/pawprints/ turned off pull requests (off topic for this issue and better for #229)

definitely agree we need better error handling wherever we call external APIs tho

[MoralCode]

Since I'm bringing forward some github API refactoring I'm doing in #301 (to unify URLs to eventually allow the domain to be swapped out), I'm likely going to be touching just about every place we use the github API anyway.

my plan is to use that opportunity to also double check the error handling with regard to 404's on unexpected things (like repos being deleted mid-collection).

My main question is going to be:

What is the correct response CollectOSS should have for these failed API calls? I imagine it may somewhat depend on the thing being fetched, but do we have any generally applicable ways of doing this?

So far my thoughts are either to short circuit collection or to somehow either let the task fail/retry, mark it as deleted/ignored, or something like that

[cdolfi]

There needs to be a bit of investigation into some the logs to see the cause of the 404s. does the commit/file/pr etc actually not exist or is it a redirect/using an old url issue

[MoralCode]

from the ~3-4 variants of the stack trace I have seen, they are all legitimate failures - i.e. repos not existing/404ing in github web UI, Issues or PRs being disabled (and also 404ing) etc

[MoralCode]

hmm ok seems like i misspoke, this is a different issue,

heres Another stack trace for a commit the API doesnt think exists, but the webUI does:

https://api.github.com/repos/halucinator/halucinator_avatar2/commits/37fa74d1ff93becd33cc0536691ff5de4e4cfc5c

halucinator/avatar2@37fa74d

im going to say this is due to githubs four 7's of uptime/reliability tbh

Stack Trace

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 248, in insert_facade_contributors
    process_commit_metadata(logger, key_auth, batch, repo_id, platform_id, tool_source, tool_version, data_source)
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 66, in process_commit_metadata
    login = get_login_with_commit_hash(logger, auth, contributor, repo_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py", line 371, in get_login_with_commit_hash
    login_json = github_data_access.get_resource(url)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 108, in get_resource
    response = self.make_request_with_retries(url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 164, in make_request_with_retries
    return self.__make_request_with_retries(url, method, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 330, in wrapped_f
    return self(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 467, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 186, in __make_request_with_retries
    result = self.make_request(url, method, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 134, in make_request
    raise UrlNotFoundException(f"Could not find {url}")
collectoss.tasks.github.util.github_data_access.UrlNotFoundException: Could not find https://api.github.com/repos/halucinator/halucinator_avatar2/commits/37fa74d1ff93becd33cc0536691ff5de4e4cfc5c

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 439, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency 8d33b5a4-d6a2-485e-99dd-c74846ce526f raised UrlNotFoundException('Could not find https://api.github.com/repos/halucinator/halucinator_avatar2/commits/37fa74d1ff93becd33cc0536691ff5de4e4cfc5c')

During handling of the above exception, another exception occurred:

edit, ok looks like this is a case where the repo URL (repo name) has changed

[MoralCode]

The original stack trace with which this issue was reported was also a move, new URL: https://github.com/TheAlaskanPhysicist/RIT-Physics-Research-2022

[MoralCode]

Another example on github.com/psf/psf-bylaws (renamed to https://github.com/psf/bylaws)

Stack Trace

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 248, in insert_facade_contributors
    process_commit_metadata(logger, key_auth, batch, repo_id, platform_id, tool_source, tool_version, data_source)
  File "/collectoss/collectoss/tasks/github/facade_github/tasks.py", line 66, in process_commit_metadata
    login = get_login_with_commit_hash(logger, auth, contributor, repo_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py", line 371, in get_login_with_commit_hash
    login_json = github_data_access.get_resource(url)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 108, in get_resource
    response = self.make_request_with_retries(url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 164, in make_request_with_retries
    return self.__make_request_with_retries(url, method, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 330, in wrapped_f
    return self(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 467, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/collectoss/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 186, in __make_request_with_retries
    result = self.make_request(url, method, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/collectoss/tasks/github/util/github_data_access.py", line 134, in make_request
    raise UrlNotFoundException(f"Could not find {url}")
collectoss.tasks.github.util.github_data_access.UrlNotFoundException: Could not find https://api.github.com/repos/psf/psf-bylaws/commits/2043a258b632202de5213da1c34766932c9377f1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in on_chord_part_return
    resl = [unpack(tup, decode) for tup in resl]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 533, in <listcomp>
    resl = [unpack(tup, decode) for tup in resl]
            ^^^^^^^^^^^^^^^^^^^
  File "/collectoss/.venv/lib/python3.11/site-packages/celery/backends/redis.py", line 439, in _unpack_chord_result
    raise ChordError(f'Dependency {tid} raised {retval!r}')
celery.exceptions.ChordError: Dependency adaf3375-bf16-422e-a748-0bfcc3c06824 raised UrlNotFoundException('Could not find https://api.github.com/repos/psf/psf-bylaws/commits/2043a258b632202de5213da1c34766932c9377f1')

During handling of the above exception, another exception occurred:

[MoralCode]

in terms of our instance specifically: psf/bylaws, TheAlaskanPhysicist/RIT-Physics-Research-2022 both have updated git URLs in our repo table. their old urls do not appear in the historical_repo_urls table, and the repo_name in the repo table is still set to the old name.

I wonder if these URLs are being assembled from a bad source/overly duplicated column in the DB?

[cdolfi]

@MoralCode hmmm, I wonder how flower then has the new name?

[MoralCode]

Flower might be pulling from repo_git column