# searxNcrawl, a SearXNG MCP server with crawling By Das Digitale Momentum, published 2026-11-03 searxNcrawl is an open-source MCP server that turns your SearXNG instance into a `search` tool for AI agents and adds `crawl` and `crawl_site` for clean Markdown. This page lists the tools and shows the setup for each MCP client. ## What the SearXNG MCP server does - `search`: queries your SearXNG instance and, opt-in, You.com in parallel and returns one merged, deduplicated list with the source of each result. Filters for language, time range, safe search and paging; SearXNG categories and engines. With `include_content=true`, each result also carries the Markdown of its page. - `crawl`: one or many URLs to Markdown or JSON, with repeated blocks removed and links optionally stripped. - `crawl_site`: follows links from a start URL within depth, page and time limits, tuned for documentation. ## Why search and crawling belong in one server [SearXNG](https://docs.searxng.org/) is a metasearch engine: it forwards a query to many search engines, merges their answers and tracks nobody. For an agent that is a good start, but a result list holds only titles and snippets of a few dozen words. To answer a question, the agent usually needs the page behind the link, and a page in HTML carries navigation, cookie banners and scripts that cost tokens without adding facts. searxNcrawl closes that gap. The server that searches also renders pages in Chromium through [Crawl4AI](https://github.com/unclecode/crawl4ai) and [Playwright](https://playwright.dev/) and returns clean Markdown with repeated blocks removed. A typical agent flow is one `search` with `include_content=true`, or a `search` followed by `crawl` on the most relevant results; for documentation, `crawl_site` reads a whole section within the limits you set. Crawling also works without any search provider. An agent that already knows the address, for example of a documentation page or an issue, calls `crawl` directly. ## What a search returns Through MCP, `search` returns structured results with the fields `title`, `url`, `content`, `publishedDate` and `providers`, the list of providers that found the result. With `include_content=true`, each result also carries `markdown`, the crawled page, or `crawl_error` when the page could not be read; the results arrive either way. The CLI prints the same results as Markdown, which is the quickest way to check a SearXNG instance: ``` SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5 ``` ``` # Search: model context protocol _Found 3 results_ ## 1. <address of the first result> **Sources:** searxng <snippet of the first result> ``` The default fields keep each result small, which keeps the context of the agent short. `SEARCH_RESULT_FIELDS` selects other fields when a workflow needs them, for example only `url` and `title` for a quick overview. ## Search providers: SearXNG and You.com SearXNG is the default provider and needs no key. With your own You.com API key in `YDC_API_KEY`, searxNcrawl queries both providers in parallel and merges them into one list: duplicate addresses are joined conservatively, the results alternate between the providers, and results found by both come first. Each result names its providers, and if one provider fails the other still answers. You.com queries are processed in the US and return results for the first 10 pages only; the `providers` parameter picks a provider for a single call ([changelog](https://github.com/DasDigitaleMomentum/searxNcrawl/blob/main/CHANGELOG.md)). Results that only You.com found carry youcom in `providers`, so an agent can tell where a fact came from and weigh it accordingly. ## Before you start - A SearXNG instance with JSON output enabled. The [guide to the SearXNG JSON API](https://www.searxncrawl.io/searxng-json-api/) does it in three steps. - [uv](https://docs.astral.sh/uv/), which runs searxNcrawl with `uvx` without a separate install. - Room for Chromium: the first crawl downloads Playwright's Chromium, about 550 MB. uv downloads the package on the first run and caches it, so later starts are quick. To give a whole team the same release, start it as `uvx searxncrawl@0.33.1` in the MCP configuration. ## Set up every agent in one step The setup command finds Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and an optional You.com API key, shows each change and backs up every file before it writes. `uvx searxncrawl setup --remove` undoes it. ``` uvx searxncrawl setup ``` The command edits the configuration files of the clients it finds and stores your SearXNG address there. Run it again after an update or to change the address or the key: it updates its own entries instead of adding duplicates. ## Claude Code ``` claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl ``` This registers searxNcrawl as an MCP server named searxncrawl and passes the SearXNG address as an environment variable. Add `--env YDC_API_KEY=...` the same way to enable You.com as a second provider. ## Codex ``` codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl ``` Like Claude Code, Codex passes the SearXNG address as an environment variable to the server it starts; add `--env YDC_API_KEY=...` to enable You.com as a second provider as well. ## VS Code ``` code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}' ``` ## Cursor, Claude Desktop and other clients Clients that read an `mcpServers` file take this block; replace the address with the one of your SearXNG instance. ``` { "mcpServers": { "crawler": { "command": "uvx", "args": ["searxncrawl"], "env": { "SEARXNG_URL": "http://your-searxng:8888" } } } } ``` Claude Desktop can also install searxNcrawl from the bundle `searxncrawl-.mcpb` on the GitHub releases page; the bundle asks for the SearXNG address, optional basic authentication and extra headers in a form. ## LM Studio LM Studio connects MCP servers since version 0.3.17 and, as its [MCP documentation](https://lmstudio.ai/docs/app/mcp) says, follows the `mcp.json` notation of Cursor, so the `mcpServers` block above works unchanged. Open the Program tab in the right sidebar, choose Install and then Edit mcp.json, and paste the block. LM Studio starts `uvx` itself, so uv has to be installed on the same machine. ## OpenCode OpenCode reads MCP servers from the `mcp` key of its `opencode.json`. A local server has the type local, its command as an array and its variables under `environment` ([OpenCode MCP documentation](https://opencode.ai/docs/mcp-servers/)): ``` { "$schema": "https://opencode.ai/config.json", "mcp": { "searxncrawl": { "type": "local", "command": ["uvx", "searxncrawl"], "environment": { "SEARXNG_URL": "http://your-searxng:8888" }, "enabled": true } } } ``` ## OpenClaw ``` openclaw mcp add searxncrawl --command uvx --arg searxncrawl --env SEARXNG_URL=http://your-searxng:8888 ``` OpenClaw keeps MCP servers in its own registry and passes them to the agent runs it manages; `openclaw mcp add` saves the command, its argument and the variable ([OpenClaw MCP documentation](https://docs.openclaw.ai/cli/mcp/registry)). OpenClaw also has a SearXNG provider for its own `web_search` tool ([SearXNG search in OpenClaw](https://docs.openclaw.ai/tools/searxng-search)). That provider only searches; searxNcrawl adds `crawl` and `crawl_site` for the pages behind the results. ## Crawl pages behind a login Some documentation and dashboards are only visible after a login. Capture the session once with `crawl-capture`, from a login window or from a browser that is already signed in; it saves a Playwright storage state file. Pass that file as `storage_state` to `crawl` or `crawl_site`, and the crawler sees the pages as you do. Treat the file like a password, since it holds the cookies of your session. ## Remote clients over HTTP Remote and browser-based clients connect over HTTP; the server listens on port 8000 and serves the MCP endpoint at `/mcp`. For shared access, put an authenticated TLS proxy in front of it and set `FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true` to enable the Host and Origin checks. ``` SEARXNG_URL=http://localhost:8888 uvx searxncrawl --transport http ``` Choose STDIO when the agent runs on the same machine: the client starts searxNcrawl as a child process and nothing listens on the network. Choose HTTP when several clients share one server or the client runs elsewhere, and keep `SEARXNG_URL` and any keys on that server. Host and Origin allowlists alone neither enable the checks nor authenticate callers, so the proxy in front stays necessary. ## Open WebUI Open WebUI supports MCP natively since v0.6.31, over Streamable HTTP only ([Open WebUI MCP documentation](https://docs.openwebui.com/features/extensibility/mcp)). Start searxNcrawl with `--transport http` as above, open Settings, Admin, Integrations and add a connection under External Tool Servers: - Type: MCP (Streamable HTTP), not OpenAPI - URL: `http://localhost:8000/mcp`, or `http://host.docker.internal:8000/mcp` when Open WebUI runs in Docker - Auth: None on your own machine; a token when a proxy in front of searxNcrawl asks for one For servers that speak only STDIO, Open WebUI points to its proxy [mcpo](https://github.com/open-webui/mcpo). searxNcrawl speaks HTTP itself, so it needs no proxy for Open WebUI. ## Troubleshooting - SearXNG answers 403: JSON output is off. The [guide to the SearXNG JSON API](https://www.searxncrawl.io/searxng-json-api/) turns it on. - No search provider configured: since 0.32 there is no default address, so set `SEARXNG_URL`, `YDC_API_KEY` or both; `setup` warns when neither is set. - 429 from a provider: searxNcrawl retries once when `Retry-After` asks for at most 5 seconds; otherwise the error says when to try again. - You.com answers 401 or 403: 401 means a missing or wrong key, 403 a key without access to the search API. - Results from some SearXNG engines are missing: the engine rate-limited or challenged your instance; the other engines still answer. - The first crawl takes long: Chromium downloads once, about 550 MB. Where browsers are provisioned separately, set `PLAYWRIGHT_AUTO_INSTALL=false`. ## Configuration - SEARXNG_URL: Address of your SearXNG instance, for example `http://localhost:8888` - SEARXNG_USERNAME, SEARXNG_PASSWORD: Basic authentication for SearXNG, both or neither - SEARXNG_EXTRA_HEADERS: Extra headers for SearXNG requests; values can be read from files - YDC_API_KEY: Your You.com API key, to add You.com as a second provider (opt-in) - PLAYWRIGHT_AUTO_INSTALL: false skips the automatic Chromium download These options describe searxNcrawl 0.33.1; the [MCP server documentation on GitHub](https://github.com/DasDigitaleMomentum/searxNcrawl#readme) lists every parameter. Keep secrets out of shared configuration files. `SEARXNG_EXTRA_HEADERS` accepts references such as `@~/.config/searxncrawl/cf_secret`, which searxNcrawl reads from that file at start. Invalid headers switch SearXNG off without printing their values, and You.com stays available. ## How we tested this page We ran the CLI search above on 2026-11-04 with searxNcrawl 0.33.1 against SearXNG 2026.9.15 in Docker, set up as in the [JSON API guide](https://www.searxncrawl.io/searxng-json-api/). searxNcrawl asked SearXNG for `format=json` with language, safe search and page set, and printed three results attributed to searxng. The configurations for LM Studio, OpenCode, OpenClaw and Open WebUI follow the documentation linked in their sections as we read it on 2026-11-04; we did not run those four clients. We reviewed these facts on 2026-11-04 against the linked sources. Our [editorial and corrections policy](https://www.searxncrawl.io/about/#editorial) says how we keep them current and fix errors. ## Frequently asked questions ### What is a SearXNG MCP server? An MCP server that lets AI agents query a SearXNG metasearch instance with a tool call. searxNcrawl is one; it also crawls the pages it finds, and whole sites, to Markdown. ### Does a SearXNG MCP server need an API key? Not with searxNcrawl: searching through your own SearXNG instance needs no API key and has no usage fees. Only the optional You.com provider needs your own You.com API key. ### Can the SearXNG MCP server read the pages it finds? Yes. `search` with `include_content=true` adds the Markdown of every result page to the results, and `crawl` reads any URL on its own. ### Can I use a SearXNG instance behind a login or a proxy? Yes. Set `SEARXNG_USERNAME` and `SEARXNG_PASSWORD` for basic authentication, or `SEARXNG_EXTRA_HEADERS` for other headers; values can be read from files so secrets stay out of the MCP configuration. Updated: 2026-11-04 --- Source: <https://www.searxncrawl.io/searxng-mcp-server/>