# WTG

**Archived case · VK parser and content pipeline**

WTG is a VK parser and content pipeline. It collects posts and related material through VK API and Playwright, stores raw evidence, queues OCR and classification and presents the result for operator review, labeling and publication decisions. The case records the interface, architecture, code map and measurements.

Source version: `21841487f52489d8bafaac379b9cc08199937661`

Development and evidence period: June–July 2026. The archived case was prepared on 8 September 2026.

[Project notes](https://komaroff-dev.ru/projects/wtg/) · [Online documentation](https://komaroff-dev.ru/projects/wtg/docs/)

[Source repository](https://github.com/komaroffsergei/wtg-s)

## Task

The task was to combine VK content collection, image and text processing and operator review in one observable flow. Every stage had to leave a verifiable result and support retries without losing the original evidence.

Collecting publications and manually reviewing processed results.

## Work done

- Collection, queues, processing and the operator review screen.
- Admin interface, data models and code map.

## What sets the implementation apart

Raw evidence is stored separately from derived data. Queues separate collection, OCR, classification and editorial decisions; the operator sees the source and intermediate results before publishing. The code map connects screens and APIs to modules, tables and checks so a change can be traced end to end.

## Application-code composition

- Python content pipeline: 60.7%
- Python / VK / Playwright: 19.4%
- FastAPI / operator UI: 12.6%
- Alembic / PostgreSQL: 4.8%
- Python / OCR: 2.5%

Shares use Git application-file bytes at pinned revisions. Each file belongs to one subsystem under scripts/generate-stack-share.mjs rules; tests, assets, dependencies and generated files are excluded. Rounded to 0.1%. This describes code composition, not proficiency.

## Stack

| Technology | Role |
| --- | --- |
| Python / FastAPI / Jinja | Operations admin interface, JSON API and orchestration. |
| PostgreSQL / SQLAlchemy / Alembic | Publications, jobs, decision versions, materials and migrations. |
| PaddleOCR | Separate OCR sidecar and worker. |
| Docker / Nginx / systemd | Former deployment of seven services and background jobs. |

## Architecture and decisions

- Collection and long-running processing are separate jobs with leases and heartbeats.
- OCR is separate from the web service, with its own queue, statuses and attempts.
- Source publications, intermediate decisions and final materials have different roles.
- The public archive contains anonymized images, documentation and aggregate measurements without user identifiers.

### Observed WTG system architecture

![Observed WTG system architecture](https://komaroff-dev.ru/evidence/wtg/diagrams/architecture.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/wtg/diagrams/architecture.mmd)

### VK parser and pipeline data model

![VK parser and pipeline data model](https://komaroff-dev.ru/evidence/wtg/diagrams/data-model.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/wtg/diagrams/data-model.mmd)

### Publication processing sequence

![Publication processing sequence](https://komaroff-dev.ru/evidence/wtg/diagrams/processing-sequence.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/wtg/diagrams/processing-sequence.mmd)

### Target worker separation from the project documentation

![Target worker separation from the project documentation](https://komaroff-dev.ru/evidence/wtg/diagrams/target-processing-layers.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/wtg/diagrams/target-processing-layers.mmd)

## Code map

| Feature | Screen / API | Module | Data / verification |
| --- | --- | --- | --- |
| Authorization | accounts | services/auth.py, vk_user_token.py | auth_accounts, account_communities; browser/API states are considered separately |
| Job creation and execution | schedules, processing | services/admin_control.py, cli/admin_worker.py | admin_jobs, admin_job_step_logs; claim, heartbeat, retry, stale cleanup |
| API scheduling | schedules, group-pool | services/api_posts_schedule.py | api_schedule_steps, api_group_pool_items; schedule generation, queue, cooldown |
| Collection | groups, posts | services/crawler.py, vk_parsers.py | communities, posts, post_media; source, rereading, normalization |
| Profile and marketplace | groups tabs, market | community_profile.py, community_market.py, community_services.py | profile, products, services; filters and group association |
| Quick selection | group-pool | fast_screen_pool.py, fast_wall_probe.py | fast_screen_pool_states, fast_screen_pool_checks; lease, due rows, retention |
| Pipeline | pipeline | services/content_pipeline.py | content_pipeline_runs/posts/items; stages and persisted results |
| OCR | processing, post details | ocr.py, ocr_jobs.py, cli/ocr_worker.py, cli/ocr_sidecar.py | ocr_jobs, post_media; completed/failed, attempts, timeout |
| Canonical materials | results | publication_materials.py, publication-intelligence | publication_decisions/materials/media; decision-to-source relationship and revision |
| Labeling and teacher | labels, processing | publication_teacher.py | teacher runs/results/applications; labeling, discrepancies, application |
| Logs | logs, file-logs | web/app.py, helpers file logs | admin_job_logs, step logs, files; event and file traces |
| Delivery | Telegram / MAX | separate bot repositories | canonical output + bot tables; downstream consumes prepared data |

Before changing a feature, trace its interface, API, state and storage dependencies. Check inputs, errors, retries and related workflows. Update the map and affected diagrams alongside the code, rerun the related checks and attach evidence for the source version.

## Workflows and API

1. Browse anonymized schedules, groups, pipeline and materials.
2. Match a screen to its code-map entry and data diagram.
3. Open queue/history aggregates and read the measurement date and limitations.

## Verification

- Sanitization of 30 captures was checked against the evidence index and full-size images.
- The code map was matched to UI routes, APIs, modules, tables and checks for revision 2184148.
- Queue, OCR, publication-material and table-size aggregates were retained without user records.

[CI: prod-deploy](https://github.com/komaroffsergei/wtg-s/actions/runs/27615782756) — `61c78116acaeb954dce480ac591249e4f5e5bf3e`, 2026-06-16T11:57:17Z; historical run of a different version.

[CI: prod-deploy](https://github.com/komaroffsergei/wtg-s/actions/runs/27615134214) — `1299a357d0f54961a196d2eaecb714a1d5319b60`, 2026-06-16T11:45:01Z; historical run of a different version.

## Results and measurements

- 30 meaningful screenshots of the actual interface retained.
- Each screenshot includes its date, scenario and code revision.
- 527,718 publications and 29,092 OCR jobs recorded; these are different entities, not quality metrics.

These checks apply to the listed version. Business metrics and model quality on an independent dataset were not measured.

![collection history](https://komaroff-dev.ru/evidence/wtg/charts/collection-history.svg)

[collection history: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

![container logs](https://komaroff-dev.ru/evidence/wtg/charts/container-logs.svg)

[container logs: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

![cpu samples](https://komaroff-dev.ru/evidence/wtg/charts/cpu-samples.svg)

[cpu samples: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

![material types](https://komaroff-dev.ru/evidence/wtg/charts/material-types.svg)

[material types: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

![ocr status](https://komaroff-dev.ru/evidence/wtg/charts/ocr-status.svg)

[ocr status: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

![table storage](https://komaroff-dev.ru/evidence/wtg/charts/table-storage.svg)

[table storage: source data](https://komaroff-dev.ru/evidence/wtg/metrics/operational-snapshot.json)

## Limitations

- The archived case has no live demo link; anonymized screens, diagrams and aggregate measurements document its behavior.
- Job statuses and confidence values are not measurements of OCR or model accuracy.
- Legacy noVNC and unavailable sections are not presented as working features.
- Observed web-process memory growth is reported as a measurement; its code-level root cause was not established.

## Running and operating

The server checkout and published audited repository have different SHAs. Each evidence source is identified separately; no equivalence is claimed between the old image and the checkout contents.

## Screenshot index

1. [Collection and processing schedules](https://komaroff-dev.ru/evidence/wtg/full/01-schedules.png) — 2026-09-06T21:16:48.230Z, source `21841487f52489d8bafaac379b9cc08199937661`.

2. [Account states; profiles and identifiers are hidden](https://komaroff-dev.ru/evidence/wtg/full/02-accounts.png) — 2026-09-06T21:37:46.883Z, source `21841487f52489d8bafaac379b9cc08199937661`.

3. [Global group pool; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/03-group-pool.png) — 2026-09-06T21:16:57.865Z, source `21841487f52489d8bafaac379b9cc08199937661`.

4. [Processing, OCR queue and a retained teacher run](https://komaroff-dev.ru/evidence/wtg/full/04-processing.png) — 2026-09-06T21:22:48.059Z, source `21841487f52489d8bafaac379b9cc08199937661`.

5. [Canonical publication materials](https://komaroff-dev.ru/evidence/wtg/full/05-results.png) — 2026-09-06T21:37:50.167Z, source `21841487f52489d8bafaac379b9cc08199937661`.

6. [Community list; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/06-groups.png) — 2026-09-06T21:22:52.400Z, source `21841487f52489d8bafaac379b9cc08199937661`.

7. [Product catalog; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/07-market.png) — 2026-09-06T21:22:55.701Z, source `21841487f52489d8bafaac379b9cc08199937661`.

8. [Pipeline: operator overview](https://komaroff-dev.ru/evidence/wtg/full/08-pipeline.png) — 2026-09-06T21:52:00.656620+00:00, source `21841487f52489d8bafaac379b9cc08199937661`.

9. [Labeling and tags; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/09-labels.png) — 2026-09-06T21:52:00.656620+00:00, source `21841487f52489d8bafaac379b9cc08199937661`.

10. [Job log; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/10-logs.png) — 2026-09-06T21:22:57.278Z, source `21841487f52489d8bafaac379b9cc08199937661`.

11. [File logs with sensitive sections redacted](https://komaroff-dev.ru/evidence/wtg/full/11-file-logs.png) — 2026-09-06T21:23:03.321Z, source `21841487f52489d8bafaac379b9cc08199937661`.

12. [Community record: overview; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/12-group-overview.png) — 2026-09-06T21:23:06.094Z, source `21841487f52489d8bafaac379b9cc08199937661`.

13. [Community record: source publications](https://komaroff-dev.ru/evidence/wtg/full/13-group-posts.png) — 2026-09-06T21:23:08.507Z, source `21841487f52489d8bafaac379b9cc08199937661`.

14. [Community record: products; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/14-group-market.png) — 2026-09-06T21:23:10.026Z, source `21841487f52489d8bafaac379b9cc08199937661`.

15. [Community record: profile](https://komaroff-dev.ru/evidence/wtg/full/15-group-profile.png) — 2026-09-06T21:23:11.568Z, source `21841487f52489d8bafaac379b9cc08199937661`.

16. [Source publication and processing results](https://komaroff-dev.ru/evidence/wtg/full/16-source-post.png) — 2026-09-06T21:23:13.647Z, source `21841487f52489d8bafaac379b9cc08199937661`.

17. [Pipeline item details; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/17-pipeline-item.png) — 2026-09-06T21:23:15.568Z, source `21841487f52489d8bafaac379b9cc08199937661`.

18. [Collection history and statistics; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/22-collection-stats.png) — 2026-09-06T21:23:19.083Z, source `21841487f52489d8bafaac379b9cc08199937661`.

19. [Disabled legacy VNC: historical states](https://komaroff-dev.ru/evidence/wtg/full/23-vnc-status.png) — 2026-09-06T21:23:21.713Z, source `21841487f52489d8bafaac379b9cc08199937661`.

20. [Queues and operation aggregates after data loading](https://komaroff-dev.ru/evidence/wtg/full/24-operations.png) — 2026-09-06T21:50:41.187Z, source `21841487f52489d8bafaac379b9cc08199937661`.

21. [Retained results of a local publication check](https://komaroff-dev.ru/evidence/wtg/full/25-local-check.png) — 2026-09-06T21:23:25.630Z, source `21841487f52489d8bafaac379b9cc08199937661`.

22. [Groups in results; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/26-results-groups.png) — 2026-09-06T21:37:56.997Z, source `21841487f52489d8bafaac379b9cc08199937661`.

23. [Schedule history for 24 June 2026](https://komaroff-dev.ru/evidence/wtg/full/27-schedule-history.png) — 2026-09-06T21:23:38.557Z, source `21841487f52489d8bafaac379b9cc08199937661`.

24. [Extracted products; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/28-results-products.png) — 2026-09-06T21:37:58.150Z, source `21841487f52489d8bafaac379b9cc08199937661`.

25. [Extracted services; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/29-results-services.png) — 2026-09-06T21:37:59.986Z, source `21841487f52489d8bafaac379b9cc08199937661`.

26. [Daily schedule charts; the screen records the stage data, available operator actions and processing result.](https://komaroff-dev.ru/evidence/wtg/full/30-schedule-charts.png) — 2026-09-06T21:26:20.496Z, source `21841487f52489d8bafaac379b9cc08199937661`.

27. [Mobile results; documented table limitations](https://komaroff-dev.ru/evidence/wtg/full/31-results-mobile.png) — 2026-09-06T21:38:03.366Z, source `21841487f52489d8bafaac379b9cc08199937661`.

28. [Telegram WebApp: city selection without a user session](https://komaroff-dev.ru/evidence/wtg/full/32-telegram-profile.png) — 2026-09-06T21:50:51.803Z, source `21841487f52489d8bafaac379b9cc08199937661`.

29. [MAX WebApp: city selection without a user session](https://komaroff-dev.ru/evidence/wtg/full/33-max-profile.png) — 2026-09-06T21:50:58.987Z, source `21841487f52489d8bafaac379b9cc08199937661`.

30. [Public “Gde Dvizh” landing page](https://komaroff-dev.ru/evidence/wtg/full/34-gde-dviz-landing.png) — 2026-09-06T21:38:05.429Z, source `21841487f52489d8bafaac379b9cc08199937661`.

Screenshots, diagrams and the original evidence package retain their source language. Captions and this document are translated into English.

[Original evidence package](https://komaroff-dev.ru/downloads/wtg-evidence.zip)