Skip to content
S

Scrapy

Open source web scraping framework for Python

Open source

#8447in Developer Tools
0In stores

Description

Scrapy is an open source Python framework for crawling websites and extracting structured data. You write spiders that follow links, parse p…

Preview

Highlights

  • Crawl sites and extract structured data in Python
  • Asynchronous and built for large crawls
  • Browser rendering with scrapy-playwright
  • Data validation and alerts with spidermon
  • Proxy rotation through scrapy-zyte-api
  • MCP server so AI agents can inspect live crawls

Ratings & Reviews

No ratings yet

Be the first to rate Scrapy.

People also visit

E

Estela

estela, an elastic web scraping cluster

Developer Tools
Launch
Z

Zendriver

A blazing fast, async-first, undetectable webscraping/web automation…

Productivity
Launch
S

Scrapemate

Golang Crawling and scraping framework

Utilities
Launch
S

Scrapling MCP Server

An adaptive Web Scraping framework

AI Skills
Get
W

Web Scraping

Más de 50 ejemplos de web scraping utilizando: Requests

Utilities
Launch
G

Gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd

Developer Tools
Launch

Scrapy FAQ

Is Scrapy free?

Yes. Scrapy is open source and released under the BSD license.

What language is Scrapy written for?

Python. You can install it with uv add scrapy, and the documentation includes a tutorial spider to start from.

Can Scrapy handle JavaScript-heavy sites?

Yes, with the scrapy-playwright plugin, which renders pages in a real browser while keeping the normal request and response workflow.

Does Scrapy work with AI agents?

Yes. The official Scrapy MCP Server lets agents such as Claude inspect running crawls. The site shows a claude mcp add command to connect it.

Who maintains Scrapy?

The project is maintained by Zyte with more than 500 other contributors. Its GitHub repository shows 65.3k stars and 12.1k forks.

What are the best Scrapy alternatives?

Popular alternatives to Scrapy include Estela, Gerapy and SeleniumBase. See all Scrapy alternatives

Developer Stuff

Badges