Deep research

[ManavSarkar]

After scrapping a web, analyze the content, and then prepare a list of queries for subsequent web search and scrap. Can make the knowledge detailed.

Suggest an approach for getting assigned.

[ParagGhatage]

@ManavSarkar ,

Here's the approach -

DeepResearch will use DuckDuckGo search engine to search related content.
and then answer from that context.

Please assign this issue to me.

[ParagGhatage]

@ManavSarkar ,
It will extract keywords from scrapped data of article and search according to those words to get real world knowledge from the internet.

[ManavSarkar]

@ParagGhatage okay give it a try

[ParagGhatage]

@ParagGhatage okay give it a try

I will make it the best feature.

[PranshuLakhotia]

@ManavSarkar i was also thinking of using The AutoGen framework because it has a a very high success rate in DOM parsing using bs4 and also and it also employs contrastive learning to produce 3-5 optimized search queries per iteration.