FreeProxy scraper

FreeProxy is a Python tool designed to discover, extract, and verify free proxies at scale from GitHub repositories. It is capable of collecting up to +100,000 proxies and verifying them at extreme speeds.

Features

• Smart Discovery: Automatically scans recent GitHub repositories for updated proxy lists.
• Anti-Blocking Rotation: Implements a truly random, non-repeating rotation system to bypass GitHub API rate limits.
• Ultra-Fast Verification: Uses multithreading (+50 parallel workers) to test thousands of proxies in minutes.
• Data Cleaning: Automatically extracts IP:PORT format from messy text files.
• Efficient Management: Manages proxy lists in-memory to prevent unnecessary disk I/O and overhead.

Installation

1. Clone the repository:

   git clone https://github.com/0xlibfree/FreeProxy.git
   cd FreeProxy

2. Install the required dependencies:

   pip install requests

How tu use?

Simply run the main script:

python main.py

The process works in two phases:

1. Discovery Phase: The script searches for GitHub repositories created or updated in the last 24 hours and downloads every .txt file containing potential proxies. It stops once the PROXIES_LIMIT is reached.
2. Verification Phase: It loads the collected list and performs mass-testing using 100 parallel threads. Validated proxies are saved to proxies.txt.

Configuration

You can adjust the following parameters inside main.py:

• PROXIES_LIMIT: Maximum number of proxies to collect before starting the test phase (default: 100,000).
• max_workers: Within the ThreadPoolExecutor block, you can adjust the number of parallel threads (default: 100).