Whether you are to build next google, or just another clever SaaS tool. You often need access to WWW index - and that demands crawling. Renting the servers that crawl and later process data can be expensive (citation needed).
This repository acts as a 'thought experiment' where we discuss
- How expensive can crawling and processing be (for a search engine, for a vertical search engine, for a typical SaaS tool)
- How many consumers can this solution attract. Would there be enough customers who would want to use this 'blockchain based crawler' that A: crawls the internet and B: processes the data on request
- How affordable can such solution actually be in comparison to regular cloud providers or putting up your own clusters
- What engineering challenges lie ahead in establishing a marketplace on blockchain where miners can contribute crawled data (where they are penalized for contributing false data).
- and more..?