searxNcrawl, a web crawler MCP server with Crawl4AI

By Das Digitale Momentum, published November 4, 2026

searxNcrawl is an open-source MCP server that gives AI agents a web crawler: crawl turns pages into clean Markdown, crawl_site reads a documentation section within limits, and both run on your machine with Crawl4AI and Playwright. This page shows both tools in a real run and compares them with the MCP endpoint of Crawl4AI.

What the crawler tools do

• crawl: one or many URLs, rendered in Chromium and returned as Markdown or JSON. Sections that repeat within a page are removed, links can be kept, removed or listed on their own, and a saved browser session opens pages behind a login.
• crawl_site: starts at one URL and follows its links depth first, within a depth, a page limit and a timeout, on the same domain unless include_subdomains is set. Each page comes back as its own Markdown section.
• search with include_content=true: finds pages through SearXNG and/or You.com and crawls each result in the same call, as the SearXNG MCP server page describes.

All three run in one Python process on your machine. The crawler underneath is Crawl4AI, an open-source library that drives Chromium through Playwright. searxNcrawl wraps it in MCP tools with limits and defaults for documentation, and it needs no account and no API key. Crawling works without any search provider: an agent that already knows the address of a page calls crawl directly.

Why an agent needs a crawler

Many MCP clients ship a fetch tool that downloads a page and converts its HTML. That works for static pages, but a page that builds its content with JavaScript arrives empty or half filled, and the HTML of a typical documentation page is mostly navigation, sidebars, cookie banners and scripts. Every one of those lines costs context and tokens without adding a fact the agent needs.

A crawler built for language models renders the page in a real browser first, waits until the content is there and then converts what a reader sees into Markdown: headings stay headings, code stays code, and tables and links survive. searxNcrawl then removes sections that appear twice within the same page, such as a menu rendered at the top and again at the bottom. It does not compare pages with each other, so every page of a site crawl keeps its full text.

The result is text an agent can quote and reason about. Agents use it to read the documentation of a library before they write code against it, to check a changelog or an issue, or to collect the pages a search found into one answer with sources.

Crawl one page from the shell

The CLI is the quickest way to see what an agent will get. This command crawls the search API page of the SearXNG documentation and writes it to a file:

uvx --from searxncrawl crawl https://docs.searxng.org/dev/search_api.html -o page.md

In our run the page answered 200, the crawl took 8.3 seconds including the browser start, and the file held 3,221 characters of Markdown: the endpoints, the parameters, the curl examples and the warning that a format missing from settings.yml answers 403. With --json, the same command returns the Markdown together with the status code, the title, the five links of the page as references and the counts of the deduplication.

On this page the exact deduplication removed 23 of 46 sections, and searxNcrawl logged a warning because that rate is above its threshold of 45 percent. The warning does not restore anything; it tells you to look at such a page yourself when every detail matters, or to crawl it again with --dedup-mode off.

Crawl a documentation section

For a whole section, add --site with a depth and a page limit. This run starts at the administrator documentation of SearXNG and stops after five pages:

uvx --from searxncrawl crawl https://docs.searxng.org/admin/ \
  --site --max-depth 1 --max-pages 5 -o docs/

The crawl finished in 14.7 seconds with five pages read and none failed, and the output directory held one Markdown file per page. --max-depth counts link hops from the start page, --max-pages caps the total, and --include-subdomains widens the scope from the domain of the start URL to its subdomains.

Choose the limits from the question, not from the size of the site. Depth 1 with a handful of pages reads an overview and the pages it links; depth 2 with the default of 25 pages covers a typical documentation chapter. The crawl returns its pages when it has finished, and if the overall timeout of 120 seconds runs out first, it returns no pages and a crawl_timeout error, so keep site crawls small and run several of them rather than one large one.

Call the tools through MCP

An agent calls the same crawler as MCP tools. To see exactly what it receives, we called both tools with the command-line mode of the MCP Inspector, which starts searxNcrawl over STDIO the way an MCP client does:

npx @modelcontextprotocol/inspector --cli \
  -e SEARXNG_URL=http://localhost:8888 uvx searxncrawl \
  --method tools/call --tool-name crawl \
  --tool-arg 'urls=["https://docs.searxng.org/dev/search_api.html"]'

# https://docs.searxng.org/dev/search_api.html
_Crawled: 2026-11-04 09:11:13 UTC_

## Search API¶
SearXNG supports querying via a simple HTTP API. ...

crawl returned 3,304 characters without an error: the Markdown of the page under a heading with its address and the time of the crawl. The same call for crawl_site read the five pages of the shell run and returned them as one result of 16,637 characters, each page under its own address and separated by a rule:

npx @modelcontextprotocol/inspector --cli \
  -e SEARXNG_URL=http://localhost:8888 uvx searxncrawl \
  --method tools/call --tool-name crawl_site \
  --tool-arg url=https://docs.searxng.org/admin/ --tool-arg max_depth=1 --tool-arg max_pages=5

urls

crawl: the list of addresses to read

url, max_depth, max_pages

crawl_site: the start address and the limits

output_format

Markdown by default, or JSON with metadata and references

remove_links

Drops the links from the Markdown to save tokens

dedup_mode

exact by default; off keeps every section

storage_state

A saved browser session for pages behind a login

timeout

Seconds per URL for crawl (default 15), for the whole site for crawl_site (default 120)

Add the crawler to your MCP client

Run uvx searxncrawl setup. It registers searxNcrawl with Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and an optional You.com API key, and backs up every file it changes. Clients that read an mcpServers file take this block; without a SearXNG address, the crawler tools still work:

{
  "mcpServers": {
    "crawler": {
      "command": "uvx",
      "args": ["searxncrawl"],
      "env": { "SEARXNG_URL": "http://your-searxng:8888" }
    }
  }
}

The SearXNG MCP server page shows the setup for each client, including LM Studio, OpenCode, OpenClaw and Open WebUI, and the HTTP transport for clients on other machines. The first crawl downloads the Chromium of Playwright, about 550 MB, once.

searxNcrawl and the Crawl4AI MCP endpoint

Crawl4AI ships its own MCP endpoint in its Docker server. Both use the same crawling library, so the difference lies in how they run and what they add around it (Crawl4AI Docker guide):

Crawl4AI facts as published in its Docker guide, checked on 2026-11-04; searxNcrawl facts describe version 0.33.1 and its crawling documentation. Choose the Crawl4AI server when an agent needs screenshots, PDFs or JavaScript on a page; choose searxNcrawl when it needs search and readable pages from one local server. Both can be registered side by side.

When a crawler is the wrong tool

• Sites that forbid automated access in their terms or robots.txt: check before you crawl them at scale.
• Sites behind bot protection: searxNcrawl crawls with a standard Chromium from your network and does not try to evade such protection.
• Structured data such as prices or tables in a fixed schema: a crawler returns Markdown, so extraction stays a step for the agent or another tool.
• Crawls of thousands of pages: crawl_site is built for a documentation section, not for indexing a domain.

Crawling from your own network has a second side. It reaches intranet pages and documentation behind a VPN that no hosted service can see, but sites that block unknown addresses block your machine as well. Keep crawls polite: small limits, a few pages at a time, and only the pages the question needs.

How we tested this page

We ran every command on this page on 2026-11-04 with searxNcrawl 0.33.1 and Crawl4AI 0.9.5, the MCP calls with the MCP Inspector 2.13.0. The numbers above are from those runs; they vary with the network and the page. The behaviour of deduplication and site crawls follows the site crawling documentation of searxNcrawl. Crawl4AI is a project of its maintainers; searxNcrawl is not affiliated with it.

We reviewed these facts on 2026-11-04 against the linked sources. Our editorial and corrections policy says how we keep them current and fix errors.

Frequently asked questions