# An MCP server to search and crawl the web for your AI agents > Run the open-source MCP server for web search and Markdown crawling yourself, or join the waitlist for searxNcrawl Cloud. searxNcrawl is an open-source MCP server that gives AI agents web search and clean Markdown crawling; a managed Cloud version is planned. MCP server · Web search · Web crawling · Markdown for LLMs · Open source (MIT) · Developed in Germany ## Open source: Run it yourself Free, MIT licence. - MCP server over STDIO or HTTP - Search via SearXNG and/or You.com - Crawl pages and whole sites to Markdown - Crawl behind logins with a captured session ## Cloud: Let us run it Planned, waitlist open. - Drop-in replacement for the Firecrawl API - One hosted MCP endpoint with an API key - Premium search and stealth crawling - Batch crawls, monitoring and extraction ```sh uvx searxncrawl ``` ## From query to Markdown in one call searxNcrawl searches the web through your own SearXNG instance and, if you add your own API key, the You.com search API, queried in parallel and merged into one result list. It crawls pages into clean, deduplicated Markdown with Crawl4AI and Playwright. It runs on your own hardware, needs no API key with SearXNG and is published on [GitHub](https://github.com/DasDigitaleMomentum/searxNcrawl) and [PyPI](https://pypi.org/project/searxncrawl/). - Web search: Your own SearXNG instance and, opt-in with your own API key, You.com, queried in parallel. Results are merged, deduplicated and attributed to their provider, filtered by language and time range. - Page crawl: One or many URLs to Markdown or JSON. Repeated blocks are removed, links can be stripped. - Site crawl: Crawls of whole sites with a depth limit and a page limit, tuned for documentation. - Authenticated crawls: Capture a login session once with `crawl-capture`, from a login window or a running browser, then crawl behind the login. - STDIO and HTTP: Plug into local MCP clients, or expose an HTTP endpoint with host and CORS guards. - CLI and Python API: `crawl` and `search` in your shell, `crawl_page_async` and `crawl_site` in your code. ## How to run searxNcrawl Three steps from an empty machine to an agent that searches and crawls. 1. Pick a search provider: Run your own SearXNG instance with JSON output enabled, add a You.com API key, or both. Crawling works without either. 2. Register it with your agents: Run `uvx searxncrawl setup`. It finds Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and an optional You.com API key, and backs up every file before it changes it. 3. Search and crawl: Your agent calls `search`, `crawl` or `crawl_site`. Chromium installs itself on the first crawl. Other clients take the MCP configuration below. ### uvx ``` # Register it with your coding agents uvx searxncrawl setup # or run the MCP server yourself (STDIO) export SEARXNG_URL=http://localhost:8888 uvx searxncrawl # or serve it over HTTP uvx searxncrawl --transport http --port 9555 ``` ### CLI ``` # Crawl a page crawl https://docs.example.com # Crawl a site with limits crawl https://docs.example.com --site --max-depth 2 --max-pages 10 -o docs/ # Search the configured providers search "python tutorials" --max-results 5 ``` ### Python ``` from crawler import crawl_page_async, crawl_site # One page doc = await crawl_page_async("https://docs.example.com/intro") print(doc.markdown) # A whole site result = crawl_site("https://docs.example.com", max_depth=2, max_pages=10) for doc in result.documents: print(doc.status, doc.final_url) ``` ### Docker ``` # MCP server with Chromium included cp .env.example .env # set SEARXNG_URL, YDC_API_KEY or both docker compose up --build # MCP endpoint http://localhost:9555/mcp ``` ## MCP configuration One configuration block: add it to your MCP client and point it at your SearXNG instance. ```json { "mcpServers": { "crawler": { "command": "uvx", "args": ["searxncrawl"], "env": { "SEARXNG_URL": "http://your-searxng:8888" } } } } ``` ### MCP clients It runs where your agents run: STDIO for local clients, HTTP for remote ones. Claude Code, VS Code, Zed, opencode, Any STDIO or HTTP MCP client. ## searxNcrawl Cloud (Planned) The open-source tools as a managed service, plus the parts that are hard to run yourself: a Firecrawl-compatible API, premium search and crawling, job queues, webhooks, monitoring and extraction. - Drop-in for the Firecrawl API: A Firecrawl-compatible REST API for scrape, crawl, map and search. Point an existing Firecrawl SDK or integration at searxNcrawl by changing the base URL. Planned for the open-source server as well. - Hosted MCP endpoint: One HTTPS address and an API key. Works with every MCP client, with no SearXNG, browser or server to run. - Search with page content: One call returns the search results together with the Markdown of each result page. Builds on the search and crawl tools. - Premium search: Several search sources, including commercial search APIs, queried in parallel, merged, deduplicated and labelled per source, with a fallback when one source fails. Builds on multi-provider search, released in the open-source project with v0.32. - Premium crawling: A hardened browser with realistic fingerprints and rotating residential proxies, for sites that block crawlers from data centres. - Batch crawls with webhooks: Queue many URLs or whole sites and receive a signed webhook when the results are ready. - Crawls behind logins: Login sessions stored encrypted for your team, so agents crawl authenticated pages without session files on laptops. Builds on crawl-capture and session files. - Change monitoring: Watch pages on a schedule, compare the Markdown and get notified when the content changes. - Structured extraction: Send a JSON Schema with a URL and receive typed JSON instead of Markdown. Planned feature set. Scope and timing can change before launch; people on the waitlist hear first. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it. ## Open source or Cloud The same tools and the same output format. The difference is who runs the infrastructure. | Capability | Open source | Cloud | | --- | --- | --- | | Firecrawl compatibility: Firecrawl-compatible REST API | Not included: Planned for open source too | Planned: Change the base URL | | Getting started: Setup | Available: `uvx searxncrawl setup` and SearXNG or You.com | Planned: An API key, nothing to install | | Getting started: MCP tools | Available: search, crawl, crawl_site | Planned: The same tools | | Getting started: Who runs it | Available: You: server, SearXNG, browser | Planned: We do | | Search: Search sources | Available: Your SearXNG and/or You.com, merged | Planned: Several sources, merged | | Search: Fallback when a source fails | Partly: The other provider still answers | Planned: Automatic | | Search: Commercial search APIs | Available: You.com, with your own API key | Planned: Included in premium search | | Search: Search with page content | Partly: Two calls: search, then crawl | Planned: One call | | Crawling: Pages and sites to Markdown | Available: Depth and page limits | Planned: The same output | | Crawling: Crawls behind logins | Available: Local session file from crawl-capture | Planned: Encrypted sessions per team | | Crawling: Batch crawls | Partly: Synchronous, on your machine | Planned: Queued, with signed webhooks | | Crawling: Sites that block crawlers | Not included: Standard Chromium from your network | Planned: Stealth browser, residential proxies | | Automation: Change monitoring | Not included: Not included | Planned: Scheduled, with notifications | | Automation: Structured extraction | Not included: Not included | Planned: JSON Schema to typed JSON | | Support: Help | Available: GitHub issues | Planned: E-mail support | ## Join the waitlist Leave your e-mail address and we will notify you when early access to searxNcrawl Cloud opens. Join the waitlist: - One e-mail when access opens - Revocable at any time - No third parties ## Frequently asked questions ### What is searxNcrawl? searxNcrawl is an open-source MCP server that gives AI agents web search and clean Markdown crawling; a managed Cloud version is planned. It searches through your own SearXNG instance and, opt-in, the You.com API, merges the results, turns web pages into Markdown without repeated blocks, is MIT licensed and is maintained by Das Digitale Momentum. ### How do I install searxNcrawl? Run `uvx searxncrawl setup`. It registers the MCP server with Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and an optional You.com API key, shows each change before applying it and backs up every file it touches. To start the server by hand, run `uvx searxncrawl` with `SEARXNG_URL` and/or `YDC_API_KEY` set. Chromium is downloaded on the first crawl. Docker, pip and uv installs are described in the [installation guide on GitHub](https://github.com/DasDigitaleMomentum/searxNcrawl/blob/main/docs/usage/installation.md). ### Can searxNcrawl replace Firecrawl? That is the plan: a Firecrawl-compatible REST API for scrape, crawl, map and search is planned for the open-source server and for searxNcrawl Cloud, so existing Firecrawl SDKs work by changing the base URL. Today searxNcrawl offers the MCP tools search, crawl and crawl_site, a CLI and a Python API. ### Which MCP clients work with searxNcrawl? Any MCP client. searxNcrawl speaks the STDIO transport for local clients such as Claude Code, VS Code, Zed and opencode, and the HTTP transport for remote and browser-based clients. ### Do I need an API key? Not with SearXNG. The open-source version needs no API key and has no usage fees when it searches through your own SearXNG instance with JSON output enabled. You.com search is opt-in: it needs your own You.com API key (`YDC_API_KEY`), and those queries go to You.com in the US. ### What is searxNcrawl Cloud? searxNcrawl Cloud is a planned managed version: a hosted MCP endpoint with search and page content in one call, search across several sources, batch crawls with webhooks, crawls behind logins, change monitoring and structured extraction. The [waitlist](https://www.searxncrawl.io/#waitlist) is open. ### Will the open-source version stay free? Yes. The open-source searxNcrawl stays MIT licensed and free to run yourself. The Cloud version adds managed infrastructure and builds on the open-source project. ## Built by DDM searxNcrawl is developed and maintained by DDM, Das Digitale Momentum GmbH & Co. KG, a software company in Much near Cologne. It succeeds the earlier `searxng-mcp` project. Read [who builds searxNcrawl](https://www.searxncrawl.io/about/) or [contact the team](https://www.searxncrawl.io/contact/). - [searxNcrawl at Das Digitale Momentum](https://www.das-digitale-momentum.de/en/open-source/#searxncrawl) - [All open-source projects of Das Digitale Momentum](https://github.com/DasDigitaleMomentum) searxNcrawl is an open-source project of Das Digitale Momentum GmbH & Co. KG, Much, Germany. Updated: 2026-10-15 --- Source: