Skip to content

refactor: fetcher のページネーションを paginateGraphQL ヘルパーに統一する#269

Merged
coji merged 1 commit intomainfrom
refactor/fetcher-pagination
Mar 31, 2026
Merged

refactor: fetcher のページネーションを paginateGraphQL ヘルパーに統一する#269
coji merged 1 commit intomainfrom
refactor/fetcher-pagination

Conversation

@coji
Copy link
Copy Markdown
Owner

@coji coji commented Mar 31, 2026

Summary

  • 6箇所のページネーションループを paginateGraphQL 共通ヘルパーに統合(-78行)
  • pullrequestsWithDetails 内の PR 基本フィールド変換を shapePullRequestNode に統一
  • codegen 由来の型(CommitNode, ReviewNode 等)でノード型を導出し、手書き型を削減

paginateGraphQL は2つのパターンを統一:

  • handleGraphQLError 付き (minPageSize 指定): pullrequests, pullrequestList, pullrequestsWithDetails
  • シンプル: commits, reviews, tags

Closes #266

Test plan

  • pnpm validate が通る(41 test files, 294 tests)
  • JTC org で crawl --repo falcon9 --pr 8945 が動作する

🤖 Generated with Claude Code

Summary by CodeRabbit

リリースノート

  • 内部改善
    • GitHubデータ取得処理を大幅に整理・統一し、ページング処理の安定性と再試行(ページサイズ調整含む)を向上しました。
    • コメント、コミット、レビュー、タグなどのデータ整形を統一して処理の一貫性と保守性を改善しました。
    • 性能と信頼性が向上しており、エンドユーザー向けの挙動に目立った変更はありません。

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 31, 2026

Warning

Rate limit exceeded

@coji has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 11 minutes and 36 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 11 minutes and 36 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fbc5d1af-7830-4cb7-a876-c259a2366a5c

📥 Commits

Reviewing files that changed from the base of the PR and between 83b7be7 and 76855a8.

📒 Files selected for processing (1)
  • batch/github/fetcher.ts
📝 Walkthrough

Walkthrough

batch/github/fetcher.ts の複数の GraphQL ページネーションループを汎用ヘルパー paginateGraphQL に統合し、shapeCommentNode / shapeCommitNode / shapeReviewNode 等のノード整形関数を導入、主要フェッチ関数をこれらでリファクタしました。

Changes

Cohort / File(s) Summary
ページネーション共通化
batch/github/fetcher.ts
ジェネリックな cursor ベースページネーションヘルパー paginateGraphQL を追加。ページサイズ削減・リトライ・早期終了(shouldStop)を一元化し、既存の while/cursor ループを置換。
ノード整形ヘルパー
batch/github/fetcher.ts
shapeCommentNodeshapeCommitNodeshapeReviewNode を追加し、GraphQL 生ノードを型付けされた整形オブジェクトに変換(無効なノードは null を返す)。
フェッチメソッドのリファクタ
batch/github/fetcher.ts
pullrequestspullrequestListcommitsreviewstagspullrequestsWithDetailscomments(issue comments 部分)などを paginateGraphQL とノード整形ヘルパーで再実装。いくつかの関数は async を外して Promise を直接返す形に変更。
レビュー/スレッドコメント処理
batch/github/fetcher.ts
レビュースレッドのコメント収集はスレッド単位のループを維持しつつ、各コメントノードの整形に shapeCommentNode を使用するよう変更。

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Possibly related PRs

Poem

ふわふわとバグを跳ね除けて、
カーソルたちをひとつに束ねたよ🐇
ページめくる手間も軽やかに、
整形して届けるデータの歌、
走るはフェッチャー、ぴょんと完成。

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning PR は Issue #266 の目的(1. GraphQL ページネーションヘルパーの抽出、2. shapePullRequestNode の再利用)をほぼ実装していますが、3.GraphQL fragment 共通化と 4.型の codegen 導出は実装されていません。 Issue #266 の目的 3(GraphQL fragment による PR フィールド共通化)と目的 4(shapePullRequestNode の型を codegen から導出)の実装を追加するか、リンク済み Issue を更新して実装範囲を明確化してください。
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed PRのタイトルは変更内容の主要な目的(fetcher.tsのページネーションを paginateGraphQL ヘルパーに統一する)を明確に説明しており、差分内容と一致しています。
Out of Scope Changes check ✅ Passed 変更はすべて Issue #266 のスコープ内(ページネーション共通化、shapePullRequestNode の再利用)です。GraphQL query や型定義への変更は確認されません。

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/fetcher-pagination

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coji coji force-pushed the refactor/fetcher-pagination branch 2 times, most recently from babb623 to 5b71104 Compare March 31, 2026 07:48
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@batch/github/fetcher.ts`:
- Around line 1209-1218: The paginateGraphQL call that invokes
graphqlWithTimeout mixes two paginated connections by repeatedly passing
reviewThreadsCursor: null when fetching issueComments (and vice versa), and also
hardcodes reviewThreads.comments to first: 100 without requesting
pageInfo/endCursor, causing repeated re-fetch of page 1 and silent truncation of
long threads; fix by splitting the concerns into two separate paginated queries
(one for issue comments and one for reviewThreads) or use GraphQL `@include` to
omit the unused connection per loop, and implement cursor-based pagination for
reviewThreads.comments (request pageInfo/endCursor and use that cursor rather
than fixed first:100); update calls to paginateGraphQL/graphqlWithTimeout to
stop passing the other side’s cursor as null and ensure both connections
paginate to completion.
- Around line 819-825: In the minPageSize check (the block referencing
minPageSize, logger.warn, label, pageSizeRef.value and cursor) do not use break;
instead throw an Error that includes the label and cursor (and optionally
current pageSize/pageSizeRef.value and any partial nodes count) so callers can
detect an incomplete crawl; replace the break with a thrown error from the same
location and ensure the thrown message clearly identifies label and cursor (and
include symbols like nodes if available) so the caller can handle the failure.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1ca35886-8d21-4961-b2b3-4cef184a9356

📥 Commits

Reviewing files that changed from the base of the PR and between 9ba26b2 and 5b71104.

📒 Files selected for processing (1)
  • batch/github/fetcher.ts

@coji coji force-pushed the refactor/fetcher-pagination branch from 5b71104 to 83b7be7 Compare March 31, 2026 08:28
6箇所のページネーションループを共通ヘルパーに統合。
pullrequestsWithDetails の PR 基本フィールド変換を shapePullRequestNode に統一。

- paginateGraphQL<TResult, TNode, TItem>: カーソルベースページネーション共通ヘルパー
  - handleGraphQLError 付きパターン(minPageSize 指定時)
  - シンプルパターン(minPageSize なし)
  - shouldStop による早期打ち切り
- pullrequests, pullrequestList, commits, reviews, tags を paginateGraphQL で書き直し
- pullrequestsWithDetails 内の PR 変換を shapePullRequestNode に統一
- codegen 由来の型(CommitNode, ReviewNode 等)でノード型を導出

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coji coji force-pushed the refactor/fetcher-pagination branch from 83b7be7 to 76855a8 Compare March 31, 2026 08:35
@coji coji merged commit febd5a8 into main Mar 31, 2026
6 checks passed
@coji coji deleted the refactor/fetcher-pagination branch March 31, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fetcher.ts の内部リファクタ: ページネーション共通化・GraphQL fragment・型導出

1 participant