Add support for page metadata and clear cache

[azharzaman1]

Added support for metadata in crawl results via option includeMetadata?: boolean, supports existing cache implementation.
Added support for cache clearing via option clearCache?: boolean.

export interface CrawlOptions {
  ...
  // Whether to include metadata in the crawl results
  includeMetadata?: boolean;
  // Whether to clear the cache directory before crawling
  clearCache?: boolean;
}