-
Notifications
You must be signed in to change notification settings - Fork 19
Readme update #129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
FabianMaumeApify
wants to merge
4
commits into
apify:master
Choose a base branch
from
FabianMaumeApify:master
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+39
−57
Open
Readme update #129
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -2,11 +2,10 @@ | |||||
|
|
||||||
| [](https://apify.com/apify/rag-web-browser) | ||||||
|
|
||||||
| This Actor provides web browsing functionality for AI agents and LLM applications, | ||||||
| Give your LLM application up-to-date context from the web: query Google Search, crawl the top results, and get clean Markdown back in one call. The extracted text can then be injected into prompts and retrieval augmented generation (RAG) pipelines, to provide your LLM application with up-to-date context from the web. | ||||||
|
|
||||||
| The RAG Web Browser provides web browsing functionality for AI agents and LLM applications, | ||||||
| similar to the [web browsing](https://openai.com/index/introducing-chatgpt-search/) feature in ChatGPT. | ||||||
| It accepts a search phrase or a URL, queries Google Search, then crawls web pages from the top search results, cleans the HTML, converts it to text or Markdown, | ||||||
| and returns it back for processing by the LLM application. | ||||||
| The extracted text can then be injected into prompts and retrieval augmented generation (RAG) pipelines, to provide your LLM application with up-to-date context from the web. | ||||||
|
|
||||||
| ## Main features | ||||||
|
|
||||||
|
|
@@ -19,6 +18,7 @@ The extracted text can then be injected into prompts and retrieval augmented gen | |||||
| - 🪗 **Collapsed sections are expanded** in Browser mode, so their content is not missing from the output | ||||||
| - 🔌 Supports **OpenAPI and MCP** for easy integration | ||||||
| - 🪟 It's **open source**, so you can review and modify it | ||||||
| - 🧹**HTML to Markdown conversion** | ||||||
|
|
||||||
| ## Example | ||||||
|
|
||||||
|
|
@@ -50,6 +50,12 @@ For a search query like `fast web browser in RAG pipelines`, the Actor will retu | |||||
| ] | ||||||
| ``` | ||||||
|
|
||||||
| **This Actor supports Google search parameters.** You can use: | ||||||
| - "site:apify.com keyword" to search results only on the Apify website | ||||||
| - "after:2026-01-31 keyword" to get results from after the 1st of January 2026 | ||||||
| - "before:2026-01-31 keyword" to get results from before the 1st of January 2026 | ||||||
|
|
||||||
|
|
||||||
| If you enter a specific URL such as `https://openai.com/index/introducing-chatgpt-search/`, the Actor will extract | ||||||
| the web page content directly like this: | ||||||
|
|
||||||
|
|
@@ -96,6 +102,18 @@ This mode is useful for testing and evaluation, but might be too slow for produc | |||||
| because it takes some time to start the Actor's Docker container and a web browser. | ||||||
| Also, one Actor run can only handle one query, which isn't efficient. | ||||||
|
|
||||||
| #### Normal run output | ||||||
| When running the Actor in normal mode, all results are saved in a dataset with the following format: | ||||||
| ```json | ||||||
| [ | ||||||
| { | ||||||
| "metadata.url": "https://github.com/apify/rag-web-browser", | ||||||
| "metadata.title": "GitHub - apify/actor-rag-web-browser: RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web. · GitHub", | ||||||
| "searchResult.resultType": "ORGANIC", | ||||||
| "markdown": "GitHub - apify/actor-rag-web-browser: RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines ..." | ||||||
| } | ||||||
| ] | ||||||
| ``` | ||||||
| ### Standby web server | ||||||
|
|
||||||
| The Actor also supports the [**Standby mode**](https://docs.apify.com/actors/running/standby), | ||||||
|
|
@@ -124,12 +142,17 @@ The `/search` GET HTTP endpoint accepts all the input parameters [described on t | |||||
|
|
||||||
| RAG Web Browser has been designed for easy integration with LLM applications, GPTs, OpenAI Assistants, and RAG pipelines using function calling. | ||||||
|
|
||||||
| ### Use Apify MCP server | ||||||
| You can connect to the MCP server using clients like Claude Desktop and LibreChat, or even build your own. The RAG Web Browser is the default Actor for the MCP server, so you will access it right after installing the MCP server. | ||||||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| Simply follow the tutorial to set up [Apify MCP server](https://blog.apify.com/how-to-use-mcp/). | ||||||
|
|
||||||
| ### OpenAPI schema | ||||||
|
|
||||||
| Here you can find the [OpenAPI 3.1.0 schema](https://apify.com/apify/rag-web-browser/endpoints) | ||||||
| for the Standby web server. Note that the OpenAPI definition contains | ||||||
| all available query parameters, but only `query` is required. | ||||||
| You can remove all the others parameters from the definition if their default value is right for your application, | ||||||
| You can remove all the other parameters from the definition if their default value is right for your application, | ||||||
| in order to reduce the number of LLM tokens necessary and to reduce the risk of hallucinations in function calling. | ||||||
|
|
||||||
| ### OpenAI Assistants | ||||||
|
|
@@ -154,56 +177,10 @@ You can easily add the RAG Web Browser to your GPTs by creating a custom action. | |||||
|
|
||||||
| Learn more about [adding custom actions to your GPTs with Apify Actors](https://blog.apify.com/add-custom-actions-to-your-gpts/) on Apify Blog. | ||||||
|
|
||||||
| ### Anthropic: Model Context Protocol (MCP) Server | ||||||
|
|
||||||
| The RAG Web Browser Actor can also be used as an [MCP server](https://github.com/modelcontextprotocol) and integrated with AI applications and agents, such as Claude Desktop. | ||||||
| For example, in Claude Desktop, you can configure the MCP server in its settings to perform web searches and extract content. | ||||||
| Alternatively, you can develop a custom MCP client to interact with the RAG Web Browser Actor. | ||||||
|
|
||||||
| In the Standby mode, the Actor runs an HTTP server that supports the MCP protocol via SSE (Server-Sent Events). | ||||||
|
|
||||||
| 1. Initiate SSE connection: | ||||||
| ```shell | ||||||
| curl https://rag-web-browser.apify.actor/sse?token=<APIFY_API_TOKEN> | ||||||
| ``` | ||||||
| On connection, you'll receive a `sessionId`: | ||||||
| ```text | ||||||
| event: endpoint | ||||||
| data: /message?sessionId=5b2 | ||||||
| ``` | ||||||
|
|
||||||
| 1. Send a message to the server by making a POST request with the `sessionId`, `APIFY-API-TOKEN` and your query: | ||||||
| ```shell | ||||||
| curl -X POST "https://rag-web-browser.apify.actor/message?session_id=5b2&token=<APIFY-API-TOKEN>" -H "Content-Type: application/json" -d '{ | ||||||
| "jsonrpc": "2.0", | ||||||
| "id": 1, | ||||||
| "method": "tools/call", | ||||||
| "params": { | ||||||
| "arguments": { "query": "recent news about LLMs", "maxResults": 1 }, | ||||||
| "name": "rag-web-browser" | ||||||
| } | ||||||
| }' | ||||||
| ``` | ||||||
| For the POST request, the server will respond with: | ||||||
| ```text | ||||||
| Accepted | ||||||
| ``` | ||||||
|
|
||||||
| 1. Receive a response at the initiated SSE connection: | ||||||
| The server invoked `Actor` and its tool using the provided query and sent the response back to the client via SSE. | ||||||
|
|
||||||
| ```text | ||||||
| event: message | ||||||
| data: {"result":{"content":[{"type":"text","text":"[{\"searchResult\":{\"title\":\"Language models recent news\",\"description\":\"Amazon Launches New Generation of LLM Foundation Model...\"}} | ||||||
| ``` | ||||||
|
|
||||||
| You can try the MCP server using the [MCP Tester Client](https://apify.com/jiri.spilka/tester-mcp-client) available on Apify. In the MCP client, simply enter the URL `https://rag-web-browser.apify.actor/sse` in the Actor input field and click **Run** and interact with server in a UI. | ||||||
| To learn more about MCP servers, check out the blog post [What is Anthropic's Model Context Protocol](https://blog.apify.com/what-is-model-context-protocol/). | ||||||
|
|
||||||
| ## ⏳ Performance optimization | ||||||
|
|
||||||
| To get the most value from RAG Web Browsers in your LLM applications, | ||||||
| always use the Actor via the [Standby web server](#standby-web-server) as described above, | ||||||
| When using RAG Web Browser in production, run it via the [Standby web server](#standby-web-server) as described above, | ||||||
| and see the tips in the following sections. | ||||||
|
|
||||||
| ### Scraping tool | ||||||
|
|
@@ -216,6 +193,11 @@ The **most critical performance decision** is selecting the appropriate scraping | |||||
|
|
||||||
| This single parameter choice can significantly impact both response times and content quality, so select based on your target websites' characteristics. | ||||||
|
|
||||||
| ### Result count | ||||||
| When you use the RAG Web Browser to perform a web search, limit the results to 10 using the `maxResults` parameter. | ||||||
|
|
||||||
| If you request more than 10 results, the Actor will need to go through several pages of Google search results, which increases the response time. | ||||||
|
|
||||||
| ### Request timeout | ||||||
|
|
||||||
| Many user-facing RAG applications impose a time limit on external functions to provide a good user experience. | ||||||
|
|
@@ -276,7 +258,7 @@ When running the RAG Web Browser in Standby web server, the Actor can process a | |||||
| This number is determined by the following [Standby mode](https://docs.apify.com/actors/running/standby) settings: | ||||||
|
|
||||||
| - **Max requests per run** and **Desired requests per run** - Determine how many requests can be sent by the system to one Actor run. | ||||||
| - **Memory** - Determines how much memory and CPU resources the Actor run has available, and this how many web pages it can open and process in parallel. | ||||||
| - **Memory** - Determines how much memory and CPU resources the Actor run has available, and thus how many web pages it can open and process in parallel. | ||||||
|
|
||||||
| Additionally, the Actor manages its internal pool of web browsers to handle the requests. | ||||||
| If the Actor memory or CPU is at capacity, the pool automatically scales down, and requests | ||||||
|
|
@@ -303,7 +285,7 @@ Results were averaged for the three queries. | |||||
| | 8 | 1 | 16 | | ||||||
| | 8 | 3 | 17 | | ||||||
|
|
||||||
| Please note the these results are only indicative and may vary based on the search term, target websites, and network latency. | ||||||
| Please note that these results are only indicative and may vary based on the search term, target websites, and network latency. | ||||||
|
|
||||||
| ## 💰 Pricing | ||||||
|
|
||||||
|
|
@@ -317,13 +299,13 @@ The Actor uses [Google Search](https://www.google.com/) in the United States wit | |||||
| and so queries like "_best nearby restaurants_" will return search results from the US. | ||||||
|
|
||||||
| If you need other regions or languages, or have some other feedback, | ||||||
| please [submit an issue](https://console.apify.com/actors/3ox4R101TgZz67sLr/issues) in Apify Console to let us know. | ||||||
| please [submit an issue](https://console.apify.com/actors/3ox4R101TgZz67sLr/issues) in Apify Console. | ||||||
|
|
||||||
|
|
||||||
| ## 👷🏼 Development | ||||||
|
|
||||||
| The RAG Web Browser Actor has open source available on [GitHub](https://github.com/apify/rag-web-browser), | ||||||
| so that you can modify and develop it yourself. Here are the steps how to run it locally on your computer. | ||||||
| The RAG Web Browser Actor is open source, available on [GitHub](https://github.com/apify/rag-web-browser), | ||||||
| so that you can modify and develop it yourself. Here are the steps to run it locally on your computer. | ||||||
|
|
||||||
| Download the source code: | ||||||
|
|
||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we mention a couple more popular clients?