3 lines (3 loc) · 187 Bytes

http - RoBot

A parallel web crawler using master-slaves processes to parse links from an initial database file ~ queue.db It only support GET, HTTP/1.1 but without using keepalive yet,