Skip to content
42crawl.fyi

Open source alternatives to 42crawl.fyi

200 open source apps like 42crawl.fyi, closest match first.

Top picks

  1. L

    LibreCrawl

    Launch

    LibreCrawl is a free, open-source website crawler for SEO analysis and site audits, positioned as an alternative to paid tools such as Screaming Frog. It runs as a Python Flask web application that several users can use at once, each with a

    FreeOpen source

  2. F

    FreeCrawl SEO Tool

    Launch

    A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, 172 SEO issue checks, and zero native dependencies.

    FreeOpen source

  3. C

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-

    FreeOpen sourceSelf-hosted

  4. C

    Crawl4AI

    Launch

    Open-source web crawler and scraper that produces clean, structured output optimized for LLMs, RAG pipelines, and AI agents. Supports async crawling, CSS/XPath/LLM extraction, and stealth browser control.

    Open source

  5. S

    Siteone Crawler Gui

    Launch

    SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm

    FreeOpen source

  6. D

    Deepcrawl

    Launch

    Deepcrawl is a free, open source toolkit that turns website content into data that AI agents can use. It reads pages, cleans the HTML and extracts content and links in a form that uses fewer tokens when passed to a language model. It runs o

    FreeOpen source

More like 42crawl.fyi

7
C

Canonry

Agent-first AI SEO (AEO/GEO) operating platform

Launch
8
R

RankMeFast

The self-hosted SEO platform

Launch
9
S

Seo-Promt-Master

Google SEO's full docs as an AI prompt machine

Launch
10
C

Cited

Open-source AI citation monitoring platform

Launch
11
R

Reader

Open source web infrastructure for AI

Launch
12
S

Sosse

Selenium Open Source Search Engine & crawler

Launch
13
S

Skyvern

Transform manual browser tasks into automated workflows using AI

Launch
14
G

Geo Optimizer Skill

Open-source Answer Engine Optimization (AEO) & Generative Engine Optimization

Launch
15
G

Getcito Worlds First Open Source Aio Aeo Or Geo Tool

GetCito the World's First Open Source AI Search Optimization

Launch
16
C

Crabwalk

Crabwalk Real-time companion monitor for OpenClaw agents

Launch
17
K

Krawl

Krawl is a customizable, lightweight

Launch
18
S

Spidercreator

Automated web scraping spider generation using Browser Use and LLMs

Launch
19
W

wigolo

The go-to web for your AI coding agent

Launch
20
S

seo-monster

SEOMonster: a local, user-credential-driven MCP server

Get
21
A

Apiclientcodegen

A collection of Visual Studio code generators

Launch
22
B

bisibility

Open-source SEO platform

Launch
23
B

BugTraceAI

Open-source, self-hosted web pentesting suite

Launch
24
P

Parchi

Your AI friend right in your browser

Launch
25
A

Anubis

Web AI firewall utility which protects upstream resources from scraper bots

Launch
26
I

Invisible Playwright MCP

Playwright MCP server undetected by anti-bots

Launch
27
W

WebwhizAI

Train ChatGPT on your website data and build an AI Chatbot

Launch
28
O

Open

Self-hosted AI visibility tracking (AEO / GEO / LLMO)

Launch
29
C

Camofox Browser

Stealth headless browser for AI agents

Launch
30
S

Searchmirror

Open-source, self-hosted rank tracker

Launch
31
O

Ocbot

Your Web4 Agent

Launch
32
S

Scavenger

Crawler (Bot) searching for credential leaks on paste sites

Launch
33
G

Go Away

[Mirror] Self-hosted abuse detection

Launch
34
T

Teracrawl

High-performance web crawler API optimized for LLMs

Launch
35
O

Openbrowserclaw

Browser-native personal AI assistant

Launch
36
L

LinkResearchTools

Identify link opportunities, monitor health

Launch
37
S

StatusScout

Monitor website uptime, detect security vulnerabilities, find broken links

Launch
38
O

Openbrowser

Let AI agents browse the web. An autonomous toolkit for browser-based AI agents

Launch
39
W

WebReaper

AI-native web scraper

Launch
40
B

Browsertrix Crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker

Launch
41
B

Bug Hunter

Adversarial AI bug hunter with auto-fix skill for Claude Code, Cursor, Codex…

Launch
42
H

Hakrawler

Simple, fast web crawler designed for easy

Launch
43
P

Playwright Bot Bypass

Claude Code skill to bypass bot detection (Google CAPTCHA, etc.)

Launch
44
D

dev3000

Captures your web app's complete development timeline

Launch
45
W

Webcap

Web archiving and change detection

Launch
46
A

Ansvisor

Open-source AI Search Intelligence Platform

Launch
47
C

Camofox Browser

Anti-detection browser server for AI agents

Launch
48
G

Geoptimize

Deterministic content-readiness lint

Launch
49
C

Crawley

The unix-way web crawler

Launch
50
K

Kuri

Browser automation, web crawling, and iOS + Android device control for AI agents

Launch

About 42crawl.fyi

4

42crawl.fyi

The answer to web crawling. Period

42crawl is an AI-powered technical SEO and website crawling platform designed for modern search environments. It continuously monitors your site to detect crawl errors, indexing issues, broken links, redirect chains and internal linking gaps before they impact rankings. Unlike traditional desktop crawlers, 42crawl runs in the background and provides ongoing monitoring instead of one-time audits.

Price
Freemium
Category
AI
Website
42crawl.fyi