Skip to content

Refactor CSFD API helpers and improve error handling #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bartholomej wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Refactor CSFD API helpers and improve error handling #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 48 additions & 18 deletions src/fetchers/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,26 +11,56 @@ const defaultHeaders = {
'User-Agent': USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)],
};

export class CsfdError extends Error {
constructor(message: string, public status?: number) {
super(message);
this.name = 'CsfdError';
}
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
Copy link

@coderabbitai coderabbitai bot Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Duplicate sleep — and the copy in global.helper.ts is broken.

A sleep function also exists in src/helpers/global.helper.ts (line 58), but that version omits the ms argument to setTimeout, making it resolve immediately regardless of the delay value. Consider removing the duplicate in global.helper.ts or fixing it and importing from a single location to avoid divergence.

#!/bin/bash
# Verify the buggy sleep in global.helper.ts
rg -n 'setTimeout' --type=ts -C2
🤖 Prompt for AI Agents 
In `@src/fetchers/index.ts` at line 21, There are two implementations of sleep:
the correct one in the sleep constant in src/fetchers/index.ts and a broken
duplicate in src/helpers/global.helper.ts that calls setTimeout without the ms
argument; remove the duplicate and centralize the utility by keeping a single
export (e.g., export const sleep = (ms: number) => new Promise(resolve =>
setTimeout(resolve, ms)) in one file) then update callers to import that sleep
function (replace local definitions/usages in fetchers and global.helper with an
import), or alternatively fix the broken implementation in global.helper.ts to
accept and pass the ms parameter and export it for reuse so all modules import
the same symbol.


export const fetchPage = async (url: string, optionsRequest?: RequestInit): Promise<string> => {
try {
const mergedHeaders = new Headers(defaultHeaders);
if (optionsRequest?.headers) {
const reqHeaders = new Headers(optionsRequest.headers);
reqHeaders.forEach((value, key) => mergedHeaders.set(key, value));
}
const { headers: _, ...restOptions } = optionsRequest || {};
const maxRetries = 3;
let lastError: Error | null = null;

const response = await fetchSafe(url, { credentials: 'omit', ...restOptions, headers: mergedHeaders });
if (response.status >= 400 && response.status < 600) {
throw new Error(`node-csfd-api: Bad response ${response.status} for url: ${url}`);
}
return await response.text();
} catch (e: unknown) {
if (e instanceof Error) {
console.error(e.message);
} else {
console.error(String(e));
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const mergedHeaders = new Headers(defaultHeaders);
if (optionsRequest?.headers) {
const reqHeaders = new Headers(optionsRequest.headers);
reqHeaders.forEach((value, key) => mergedHeaders.set(key, value));
}
const { headers: _, ...restOptions } = optionsRequest || {};

const response = await fetchSafe(url, { credentials: 'omit', ...restOptions, headers: mergedHeaders });

if (response.status >= 400 && response.status < 600) {
// Do not retry on 4xx errors (client errors)
if (response.status < 500) {
throw new CsfdError(`node-csfd-api: Bad response ${response.status} for url: ${url}`, response.status);
}
// Throw to trigger catch and retry for 5xx
throw new CsfdError(`node-csfd-api: Server error ${response.status} for url: ${url}`, response.status);
}
return await response.text();
} catch (e: unknown) {
const error = e instanceof Error ? e : new Error(String(e));
lastError = error;

// Don't retry if it is a 4xx error (which we threw as CsfdError)
if (error instanceof CsfdError && error.status && error.status >= 400 && error.status < 500) {
throw error;
}

if (attempt === maxRetries) {
throw error;
}

// Exponential backoff: 1s, 2s...
await sleep(1000 * Math.pow(2, attempt - 1));
}
return 'Error';
}

throw lastError || new Error('Unknown error');
};
70 changes: 28 additions & 42 deletions src/helpers/global.helper.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,60 +1,47 @@
import { CSFDColorRating } from '../dto/global';
import { CSFDColors } from '../dto/user-ratings';

export const parseIdFromUrl = (url: string): number => {
if (url) {
const idSlug = url?.split('/')[2];
const id = idSlug?.split('-')[0];
return +id || null;
} else {
return null;
export const parseIdFromUrl = (url: string | undefined | null): number | null => {
if (!url) return null;
// Match first sequence of digits
const match = url.match(/(\d+)/);
if (match && match[1]) {
return +match[1];
}
return null;
};

const COLOR_MAP: Record<string, CSFDColorRating> = {
'lightgrey': 'unknown',
'page-lightgrey': 'unknown',
'red': 'good',
'page-red': 'good',
'blue': 'average',
'page-blue': 'average',
'grey': 'bad',
'page-grey': 'bad'
};

export const getColor = (cls: string): CSFDColorRating => {
switch (cls) {
case 'page-lightgrey':
return 'unknown';
case 'page-red':
return 'good';
case 'page-blue':
return 'average';
case 'page-grey':
return 'bad';
default:
return 'unknown';
}
return COLOR_MAP[cls] || 'unknown';
};

export const parseColor = (quality: CSFDColors): CSFDColorRating => {
switch (quality) {
case 'lightgrey':
return 'unknown';
case 'red':
return 'good';
case 'blue':
return 'average';
case 'grey':
return 'bad';
default:
return 'unknown';
}
return COLOR_MAP[quality] || 'unknown';
};

export const addProtocol = (url: string): string => {
return url.startsWith('//') ? 'https:' + url : url;
};

export const getDuration = (matches: any[]) => {
// Internal helper
const getDuration = (matches: RegExpMatchArray | null) => {
if (!matches) {
return { hours: 0, minutes: 0 };
}
return {
sign: matches[1] === undefined ? '+' : '-',
years: matches[2] === undefined ? 0 : matches[2],
months: matches[3] === undefined ? 0 : matches[3],
weeks: matches[4] === undefined ? 0 : matches[4],
days: matches[5] === undefined ? 0 : matches[5],
hours: matches[6] === undefined ? 0 : matches[6],
minutes: matches[7] === undefined ? 0 : matches[7],
seconds: matches[8] === undefined ? 0 : matches[8]
hours: matches[6] ? +matches[6] : 0,
minutes: matches[7] ? +matches[7] : 0
};
};

Expand All @@ -63,10 +50,9 @@ export const parseISO8601Duration = (iso: string): number => {
/(-)?P(?:([.,\d]+)Y)?(?:([.,\d]+)M)?(?:([.,\d]+)W)?(?:([.,\d]+)D)?T(?:([.,\d]+)H)?(?:([.,\d]+)M)?(?:([.,\d]+)S)?/;

const matches = iso.match(iso8601DurationRegex);

const duration = getDuration(matches);

return +duration.minutes;
return duration.hours * 60 + duration.minutes;
};

// Sleep in loop
Expand Down
Loading