Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 👉

admin ui for scrapy/open source scrapinghub

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

豆瓣电影/豆瓣读书 Scarpy 爬虫

狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；爬虫监控可视化; 配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑 an enterprise-grade public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

estela, an elastic web scraping cluster 🕸

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

🕷️ Scrapyd is an application for deploying and running Scrapy spiders.

Implement scrapy with asyncio

admin ui for scrapy/open source scrapinghub

Simple Web UI for Scrapy spider management via Scrapyd

A django admin site for scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.