Cache improvements

[Marmelatze]

decide which cache entries to use based on status code. Also feed cached url to fuzz queue to not miss a previously seen URL.

---

The program was tested solely for our own use cases, which might differ from yours.

Florian Pfitzer florian.pfitzer@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH
Provider Information

Licensed under GPL 2.0

[percepo]

Thanks!
Regarding the plugin, we may have a design decision to make
Excerpt from the httpqueue:
cached = self.cache.get_object_from_object_cache(fuzz_result)
# If the request is cached, put it in the queue to be processes by plugins and return.
# This does not make additional requests, but it does allow plugins to process the cached request.
if cached:
cached.plugins_res.clear()
self.result_queue.put((self.base_result_priority, cached, False))
return

Confusing modes of operandi and default behavior:
Currently, objects from file cache are reprocessed if they would have been re-requested in the current runtime. If they wouldnt be re-requested, they also wont be reprocessed either
If the suggested cache plugin is added, this behavior is extended by forcing requeuing all of the endpoints parsed from the cachedir, if the plugin is activated
But this might be a more intuitive behavior to begin with: We could have that behavior integrated into the core of the --cache-file option. Is there a scenario in which we want the first scenario? If not, I would suggest we move this feature from plugin to core. Otherwise, we can take the plugin as an addition