crawlnet — portfolio

shipped

crawlnet

Distributed asyncio crawler — shared Redis queue, YAML spiders, export to CSV/Excel.

Role
Product & backend
Year
2026
Stack
Python 3.12 · asyncio · Redis · httpx · parsel · SQLAlchemy · FastAPI · aiogram 3
Status
shipped

Problem

Several workers on different machines, no separate coordinator: a shared priority queue with delayed retries, per-host politeness, at-least-once via inflight + reaper, and YAML spiders you can ship without editing code.

Approach

Redis holds the queue and inflight set. Workers claim URLs, fetch with httpx, parse with parsel, and write results through SQLAlchemy. FastAPI exposes control; aiogram covers ops alerts.

Попробовать

Демо на тестовых данных

Загружаем демо…

Engineering notes

Claim/release around Redis lists and an inflight hash. A reaper returns stuck URLs. Spiders stay declarative in YAML so content rules change without a redeploy of worker code.

Stack

Python 3.12, asyncio, Redis, httpx, parsel, SQLAlchemy, FastAPI, aiogram 3.

github.com/blessed-bless/crawlnet