Description
Scrapy is an open source Python framework for crawling websites and extracting structured data. You write spiders that follow links, parse pages and send the results onward through item pipelines and feed exports. The core is deliberately lean, and extra features come from plugins. scrapy-playwright renders JavaScript-heavy pages in a real browser while keeping the request and response workflow you already use. spidermon validates scraped data and alerts you by Slack, Discord or email when a spider breaks. scrapy-zyte-api adds automatic proxy rotation and browser fingerprinting. A Scrapy MCP Server lets AI agents such as Claude list crawl jobs, check their status and run Python inside a live crawl. Scrapy is asynchronous and built for large jobs, and it is released under the BSD license. The project is maintained by Zyte with more than 500 other contributors and has been released for fifteen years. The docs include a tutorial spider that crawls quotes.toscrape.com, and the site shows uv add scrapy as the install command.…
Preview
Highlights
- Crawl sites and extract structured data in Python
- Asynchronous and built for large crawls
- Browser rendering with scrapy-playwright
- Data validation and alerts with spidermon
- Proxy rotation through scrapy-zyte-api
- MCP server so AI agents can inspect live crawls
Ratings & Reviews
No ratings yet
Be the first to rate Scrapy.
People also visit
Scrapy FAQ
Is Scrapy free?
Yes. Scrapy is open source and released under the BSD license.
What language is Scrapy written for?
Python. You can install it with uv add scrapy, and the documentation includes a tutorial spider to start from.
Can Scrapy handle JavaScript-heavy sites?
Yes, with the scrapy-playwright plugin, which renders pages in a real browser while keeping the normal request and response workflow.
Does Scrapy work with AI agents?
Yes. The official Scrapy MCP Server lets agents such as Claude inspect running crawls. The site shows a claude mcp add command to connect it.
Who maintains Scrapy?
The project is maintained by Zyte with more than 500 other contributors. Its GitHub repository shows 65.3k stars and 12.1k forks.
What are the best Scrapy alternatives?
Popular alternatives to Scrapy include Estela, Gerapy and SeleniumBase. See all Scrapy alternatives







