About the project
The task was to combine VK content collection, image and text processing and operator review in one observable flow. Every stage had to leave a verifiable result and support retries without losing the original evidence.
WTG is a VK parser and content pipeline. It collects posts and related material through VK API and Playwright, stores raw evidence, queues OCR and classification and presents the result for operator review, labeling and publication decisions. The case records the interface, architecture, code map and measurements.
What sets the implementation apart
Raw evidence is stored separately from derived data. Queues separate collection, OCR, classification and editorial decisions; the operator sees the source and intermediate results before publishing. The code map connects screens and APIs to modules, tables and checks so a change can be traced end to end.
Application-code composition
Shares use Git application-file bytes at pinned revisions. Each file belongs to one subsystem under scripts/generate-stack-share.mjs rules; tests, assets, dependencies and generated files are excluded. Rounded to 0.1%. This describes code composition, not proficiency.
- The archived case has no live demo link; anonymized screens, diagrams and aggregate measurements document its behavior.
- Job statuses and confidence values are not measurements of OCR or model accuracy.
- Legacy noVNC and unavailable sections are not presented as working features.
- Observed web-process memory growth is reported as a measurement; its code-level root cause was not established.
Screenshots, diagrams and source evidence retain their original language; captions and descriptions are translated.
Architecture and diagrams
- Collection and long-running processing are separate jobs with leases and heartbeats.
- OCR is separate from the web service, with its own queue, statuses and attempts.
- Source publications, intermediate decisions and final materials have different roles.
- The public archive contains anonymized images, documentation and aggregate measurements without user identifiers.
Code map: files and dependencies
This map is tied to version 21841487f524. The table lists the screen or API, related module and data.
| Feature | Screen / API | Module | Data / verification |
|---|---|---|---|
| Authorization | accounts | services/auth.py, vk_user_token.py | auth_accounts, account_communities; browser/API states are considered separately |
| Job creation and execution | schedules, processing | services/admin_control.py, cli/admin_worker.py | admin_jobs, admin_job_step_logs; claim, heartbeat, retry, stale cleanup |
| API scheduling | schedules, group-pool | services/api_posts_schedule.py | api_schedule_steps, api_group_pool_items; schedule generation, queue, cooldown |
| Collection | groups, posts | services/crawler.py, vk_parsers.py | communities, posts, post_media; source, rereading, normalization |
| Profile and marketplace | groups tabs, market | community_profile.py, community_market.py, community_services.py | profile, products, services; filters and group association |
| Quick selection | group-pool | fast_screen_pool.py, fast_wall_probe.py | fast_screen_pool_states, fast_screen_pool_checks; lease, due rows, retention |
| Pipeline | pipeline | services/content_pipeline.py | content_pipeline_runs/posts/items; stages and persisted results |
| OCR | processing, post details | ocr.py, ocr_jobs.py, cli/ocr_worker.py, cli/ocr_sidecar.py | ocr_jobs, post_media; completed/failed, attempts, timeout |
| Canonical materials | results | publication_materials.py, publication-intelligence | publication_decisions/materials/media; decision-to-source relationship and revision |
| Labeling and teacher | labels, processing | publication_teacher.py | teacher runs/results/applications; labeling, discrepancies, application |
| Logs | logs, file-logs | web/app.py, helpers file logs | admin_job_logs, step logs, files; event and file traces |
| Delivery | Telegram / MAX | separate bot repositories | canonical output + bot tables; downstream consumes prepared data |
Using the code map
- Before making a change, find the feature and trace its links to the API, state and storage.
- Check inputs, errors, repeated events and dependent workflows.
- After making a change, update the map and affected diagrams alongside the code.
- Run the related checks and link the new evidence to the source version.
Workflows and API
- Browse anonymized schedules, groups, pipeline and materials.
- Match a screen to its code-map entry and data diagram.
- Open queue/history aggregates and read the measurement date and limitations.
Verification of the published version
CI: prod-deploy ↗ · 61c78116 · historical run of a different version · 2026-06-16T11:57:17Z
CI: prod-deploy ↗ · 1299a357 · historical run of a different version · 2026-06-16T11:45:01Z
- Sanitization of 30 captures was checked against the evidence index and full-size images.
- The code map was matched to UI routes, APIs, modules, tables and checks for revision 2184148.
- Queue, OCR, publication-material and table-size aggregates were retained without user records.
These checks apply to the listed version. Business metrics and model quality on an independent dataset were not measured.
Measurements
- 30 meaningful screenshots of the actual interface retained.
- Each screenshot includes its date, scenario and code revision.
- 527,718 publications and 29,092 OCR jobs recorded; these are different entities, not quality metrics.
Running and operating
The server checkout and published audited repository have different SHAs. Each evidence source is identified separately; no equivalence is claimed between the old image and the checkout contents.
Stack
| Python / FastAPI / Jinja | Operations admin interface, JSON API and orchestration. |
|---|---|
| PostgreSQL / SQLAlchemy / Alembic | Publications, jobs, decision versions, materials and migrations. |
| PaddleOCR | Separate OCR sidecar and worker. |
| Docker / Nginx / systemd | Former deployment of seven services and background jobs. |
Materials and source code
Development and evidence period: June–July 2026. The archived case was prepared on 8 September 2026.
The evidence archive retains its original language.