Check for robots meta tag while parsing a page

[ScottMansfield]

If nofollow then don't enqueue pages to fetch.

if noindex then don't index the current page.

default to index, follow, meaning index the current page and send link to the parse stage to process.

Need to comma-separate list and parse the list if it exists. Whitespace will be ignored.

[ScottMansfield]

The nofollow also implies no HEAD requests to check for Content-Type