Feature request: adding links to documentation websites/other data sources

[abhi12299]

What?

Similar to Cursor AI's feature, we can add the ability for the user to be able to add links to any websites and allowing the LLM to reference them during the conversation.

How?

1. When the user adds link to a website, scrape it using the standard fetch API (this may not be allowed on many websites, but we can implement a workaround by using a paid service like smartproxy's web scraper)
2. Parse the HTML and create small text chunks. This is the most important step - a good chunking algorithm will lead to better output
3. Create embeddings for chunks (using the anthropic API) and save them locally in the melty directory. This can be done using sqlite and sqlite-vec. We can have a single sqlite database for all embeddings as they will be shared across conversation
4. During the conversation, if the user includes the website, perform a similarity search and retrieve the top 5-10 chunks based on the available context window
5. Pass the relevant context to the LLM.

Additionally, we can even display the references on the basis of which the LLM responded. This can be done by:

1. Once the LLM finishes responding, we take the response and create an embedding (using anthropic API)
2. Perform similarity search between the embedding created above and the embeddings of the chunks we sent
3. Show the top 3 chunks as references. Rather than displaying as simple text, we can include metadata such as the URL, title and favicon of the website during the chunking process (in the previous list's step 2) and show a nicely formatted reference like Google search where we include the metadata and show the relevant chunk's text below it. Something like this:

---

This is a typical RAG workflow, however, it is a feature which needs a lot of time to implement and get right. I would love to start working on this. This issue is meant to be a technical discussion on the implementation details.

[abhi12299]

@jacksondc @cbh123 your thoughts on this? is this something you'd want to see in melty?

[jacksondc]

Documentation would be awesome. I think we should start out by letting the user add an entire webpage into the context, since that will be relatively easy.

Down the road I would want to try giving Claude a search tool. I'm not opposed to trying RAG as well, but my hunch is that will be less useful.

@cbh123 what's the Mac app you use for documentation? Do you know what the data source is there?

[abhi12299]

Including an entire webpage would lead to context exhaustion relatively quickly.. also i noticed that currently the entire conversation is sent over to Claude, which means longer conversations are at a risk of context exhaustion especially after including images.. this is also something to work on

[pahulgogna]

hey guys, I would love to contribute in this project. I just recently made a package called "SearchURL" that lets you search through webpages for certain keywords, as of now it does not have semantic search, but I was looking into it and could not find anthropic API (as it was mentioned in the issue) that converts chunks to text embeddings, in the anthropic docs they recommend Voyage AI . If you would like me to continue with that or some other plan, then please let me know.
Just to clarify the package is written in python.