Implement robots.txt download priority system #4
Description
Differentiate URLs sent from the crawl API and others sent by users (when its query is not found). In this case, the priority is to download the robots file of the URL sent by the user.
We can have multiple levels of priority to allow the development of more use cases and/or business requirements.