Badminton Session Tracker
Plan doc v0.4 · 16 Nov 2026 · source: Telegram @sgbadmintontelecom (chat -1001191346636) · target: Cloudflare Pages + D1
v0.3: @handle replaces phone-number collection · dedupe key includes the poster · diagram added. v0.4: Q1–Q7 locked, poster cache is an in-memory LRU(200) — no D1 table, model pinned to deepseek-v4.1-flash.
1. System architecture
Rule The raw message table is the source of truth. Every other table can be dropped and rebuilt from raw/*.jsonl in one command. A bad parse run is re-derived, never patched in place.
Scope
- In: session adverts — friendly games, court bookings with open slots, training sessions, tournament notices.
- Out (v1): coaching-business ads, racket/shuttle sales, unrelated chatter, "looking for coach" posts. They stay in
raw_messageswithparse_status='skip'+ reason, so nothing is silently dropped. - Window: backfill to 1 Nov 2026, then incremental twice daily.
1b. Tech stack & how a run is triggered
Stack
| layer | choice | why |
|---|---|---|
| browser | chromium container igbrowser on glastrier, CDP :9223 | already logged in; only thing that must live on a host with a GUI session |
| scraper + parser + upsert | Go 1.25, one binary badminton-sync (cmd/sync, internal/cdp, internal/parse, internal/d1), CDP over WebSocket | no Python anywhere in the pipeline; static binary, real concurrency for the CDP event stream, easy to run from cron. Toolchain installed under /opt/data/tools/go (tarball, no root) |
| LLM pass | one HTTP POST from the same Go binary to https://opencode.ai/zen/go/v1/chat/completions (opencode-go plan, key OPENCODE_GO_API_KEY), model pinned deepseek-v4.1-flash (canonical deepseek/deepseek-v4.1-flash-1017), JSON mode, temp 0 | verified: that base URL is the Go-plan endpoint (returns the model list); this model advertises structured output. Chosen by you over the cheaper flash alternates |
| DB | Cloudflare D1 (SQLite) over the REST API | no server to babysit; token already in /opt/data/.cloudflare_api.env |
| site + API | Cloudflare Pages: TypeScript everywhere — functions/api/sessions.ts (Pages Function, native TS) + web/src/*.ts bundled by esbuild to public/app.js | types shared between the Function and the client; wrangler handles the Function, one esbuild step for the client |
| scheduler | scripts/badminton_sync.sh under Hermes cron, 07:00 + 21:00 SGT, flock single-instance, wrapping the Go binary | one entrypoint, one lock, one log |
| config/secrets | /opt/data/.cloudflare_api.env (600), provider key from OPENCODE_GO_API_KEY | nothing new to store |
| tests | go test ./... over parse/testdata/fixtures.jsonl + a site smoke check (curl + headless DOM) | parser accuracy is a number, not a vibe |
Not used, on purpose: no Python, no queue/broker, no k8s workload, no Postgres, no client framework/Astro. Volume is a few thousand messages per day at worst — anything more is machinery for machinery's sake.
Trigger chain
Nothing "triggers" a remote agent: the scraper runs on the pod and drives the browser over CDP. The pod is both scheduler and worker, so the trigger is just cron → one script, run in order with hard gates.
cron 07:00 / 21:00 SGT
└─ flock -n /opt/data/tmp/badminton.lock
badminton_sync.sh → badminton-sync (Go)
1. preflight CDP /json/version
chat header, no JOIN btn
2. scrape → raw/YYYY-MM-DD_HHMM.jsonl
3. resolve handles for session senders (LRU 200)
4. parse → staging/{sessions,review}.jsonl
5. backup D1 export → backups/<date>.sql
6. upsert D1 REST, idempotent
7. deploy only if sessions changed
8. report runs row + Telegram digest
- Each step checks the previous exit code; any failure stops the chain and posts the failing step + stderr tail to Telegram. A half-run never reaches D1.
- Steps 2–4 are re-runnable by hand:
badminton_sync.sh --from-scrape/--parse-only. - Re-entry safety:
flock -nmeans a slow run skips the next tick instead of double-writing;state.jsonholds the lowest scrapedmessage_idfor resume. - Login-wall detection (the chromium session expired) is a step-1 failure with its own alert text, because fixing it needs your phone, not a retry.
- If this ever needs to be decoupled, the seam is a
jobsrow in D1 (queued → running → done) that the same script polls. Deliberately not built now.
2. Data schema (D1 / SQLite)
Five tables. raw_messages mirrors Telegram; sessions is one row per bookable slot (with the poster denormalised on it); venues is curated; runs/parse_log are the audit trail.
raw_messages — what Telegram gave us
| column | type | notes |
|---|---|---|
| message_id | INTEGER PK | Telegram's own id (data-message-id). Monotonic, dedupe key. |
| posted_day | TEXT (YYYY-MM-DD) | From timeline date header. Day-accurate at best — see E1. |
| posted_time | TEXT (HH:MM) | SGT, from .message-time. |
| sender_name | TEXT | Display name as rendered ("Roy LHH"). Not unique, not an identity. |
| text | TEXT NOT NULL | Verbatim with phone numbers redacted at capture (9xxx xxxx → [redacted]). Original never stored. |
| has_media | INTEGER | 1 if the post had a photo/poster (details often live in the image → needs_review). |
| permalink | TEXT | https://t.me/sgbadmintontelecom/<message_id> — always available, no PII. |
| scraped_at | TEXT | ISO timestamp of the scrape run. |
contact resolution — no posters table
Posters are not a table. Handles are resolved only for senders whose post classified as a session — a few per run,
not every member who speaks. The cache is an LRU of 200 display names backed by a small JSON file
(state/posters.json, written atomically at the end of each run), so the next cron run starts warm instead of
re-peeking the same organisers. Resolved values are also denormalised onto the session row, so history survives even if the
cache file is deleted.
@name) ·
② handle from the cached profile peek → https://t.me/<handle> · ③ message permalink
t.me/sgbadmintontelecom/<message_id>, which always works from inside the group. Never a phone number.Why a file and not D1: it is derived data, rebuildable by re-peeking, and it must survive process exit. No migration, no table to keep honest, no write amplification on the DB. LRU-200 keeps it bounded (~15 KB).
sessions — what the site queries
| column | type | notes |
|---|---|---|
| id | TEXT PK | sha1(session_date|start_time|venue_key|poster_key) — stable across re-parses. |
| message_id | INTEGER FK | Provenance. One message can yield several sessions (multi-slot posts). |
| poster_name | TEXT | Display name as rendered. poster_key in the dedupe key = handle if resolved, else this name. |
| poster_handle | TEXT | @name if resolved (post text or profile peek), else NULL. Denormalised so the row survives the cache. |
| contact_url | TEXT | t.me/<handle> or the message permalink; never a raw phone number. |
| session_date | TEXT (YYYY-MM-DD) | Normalised. NULL only when the post genuinely omits a resolvable date. |
| start_time / end_time | TEXT (HH:MM) | 24h SGT. end_time NULL if unstated. |
| is_recurring / rrule | INTEGER / TEXT | Weekly fixtures ("every Tue 8-10pm") — calendar renders the next occurrence. |
| venue_key | TEXT FK | Slug into venues; NULL when unmatched (parked in review). |
| courts / courts_note | INTEGER / TEXT | "3.5 courts" → 3 + note. |
| level_raw / level_norm | TEXT | Raw string kept forever; normalised to the LI/MI/HI ladder (§3 S6). |
| price_raw / price_cents / price_basis | TEXT / INTEGER / TEXT | basis ∈ per_pax, per_court, unknown. |
| shuttle_brand | TEXT | "RSL Supreme" — people care. |
| slots_raw | TEXT | "last 2 slots" etc., shown but not machine-usable. |
| parse_status | TEXT | ok · partial · needs_review · skip |
| confidence | REAL | 0–1 from the validators in §5, not the model's self-report. |
venues
| column | type | notes |
|---|---|---|
| venue_key | TEXT PK | "yishun-sports-hall" |
| name / area | TEXT | "Yishun Sports Hall" / "North" |
| aliases | TEXT (JSON) | ["Yishun SH", "YSportshall"] — the fuzzy matcher's training data. |
| lat / lon | REAL | Geocoded once via OSM/Nominatim, cached. Enables the map later. |
| curated | INTEGER | 1 = human-confirmed. Uncurated venues never auto-merge. |
runs & parse_log
| table | columns |
|---|---|
| runs | id, started_at, finished_at, kind (scrape|parse|upsert), messages_seen, sessions_new, sessions_updated, status, note |
| parse_log | message_id, parser_version, model, fields_json, issues_json, created_at — diff parser v3 vs v4 without re-calling the model |
3. Cleaning pipeline
Posts are free-form, bilingual (EN / 中文), full of shorthand. Ten stages, each with a hard rule; anything that fails a stage lands in review.jsonl instead of guessing.
S0 · Capture
Wheel-scroll the chat, emit one JSON object per .Message node. Normalise whitespace (\n{3,} → \n\n), strip zero-width chars, keep emoji. Redact phone numbers on the way in (SG mobile 8/9xxxxxxx, +65 variants, and 4-4 spaced forms) → the DB never holds them.
S1 · Classify: session ad or not
Rules first, model second. Ad if ≥2 of: a date token, a time token, a venue token, a level token, a slot/price token. Chatter and pure adverts ("selling Yonex 88D") are skip with a reason code (no_date, no_venue, sales, coaching_biz).
needs_review, never a guess: they lack date+venue by design.S1b · Poster resolution
The message list exposes only a display name — no username, no peer id. So: the message list exposes only a display name — no username, no peer id. After S1 says "this is a session ad", open that sender's profile once in the web client (read-only click), read @username if present, and keep it in an in-memory LRU of 200 display names (least-recently-used evicted) for the rest of the run. Only session posters are peeked — chatter is never resolved. Handles appearing inside post bodies are captured by regex and win over the peek. Unresolved → permalink fallback. Nothing about posters is persisted except the denormalised field on the session row.
S2 · Extract
Regex pass for the labelled shapes this channel actually uses — Date:, Time:, Venue:, Level:, @venue, free-text lines. Model pass only for what regex leaves empty, batched 20 posts/call, JSON-schema output, temperature 0. Missing → null. The model may only copy spans from the text — every non-null field is checked as a substring/derivative of the raw text (§5).
S3 · Dates
| input seen in channel | rule | output |
|---|---|---|
21 Nov 2026, Sat | absolute | 2026-11-21 |
17/ 11/2026 | D/M/Y (SG default); reject M/D/Y if D>12 | 2026-11-17 |
21/11/26 | DD/MM/YY, century = 2000s | 2026-11-21 |
TUESDAY, tonight, this Sat | resolve against posted_day; roll forward if the word'd weekday is behind | computed + date_assumed=1 |
| weekday mismatch ("21 Nov 2026, Sun") | trust the numeric date, log date_conflict, confidence −0.3 | 2026-11-21 |
| no date at all | no inference from posted_day | NULL + needs_review |
Sessions dated before the run are archived, not deleted (is_past derived) so the calendar can show history.
S4 · Times & multi-slot posts
07:30 pm - 09:30 pm,7.30-9.30pm,1930-2130,8 to 10→ 24h pairs.2-4pm, 2-5pm, 3-6pm→ three session rows (one message, three slots) — why dedupe is content-keyed, not message-keyed.2hrswith a start only →end = start + 2h,end_assumed=1.- Ambiguous am/pm (
7-9) → 19:00–21:00 if the venue is indoor; else needs_review.
S5 · Venues
Three passes: exact alias hit → normalised fuzzy (lowercase, strip @, fold "SH"/"Sports Hall"/"Sportshall", drop parentheticals) with token-set ratio ≥ 0.85 → model tiebreak over the top-3 candidates, candidates recorded in parse_log. New venues are created curated=0 and surface in a weekly "new venues" digest; uncurated venues are never auto-merged.
S6 · Levels
SG ladder: HB < LI < MI < HI < ADV. Parse ranges ("LI to HI", "HB-LI"); anything else ("intermediate", "social") → level_norm='ANY' with raw kept. "intermediate" is never silently mapped to MI.
S7 · Price, courts, shuttles
$7/pax → 700, per_pax; $42/court → 4200, per_court; free → 0; "share cost" → basis=unknown. Courts: "3.5 courts" → courts=3 + note.
S8 · Dedupe & merge
- Key:
session_date + start_time + venue_key + poster_key(handle if resolved, else display name). Two different organisers legitimately run the same court and time — that is not a duplicate, so the poster is part of the key. Reposts by the same organiser collapse. - On collision: keep the earliest
message_idas provenance, append the rest toextra_message_ids, prefer the newest non-null field values (later posts carry fresher slot counts). - Never merge across different
venue_key, different start times, or different posters.
S9 · Validate & score
Confidence starts at 1.0, docked per issue: assumed date (−0.15), assumed end (−0.1), uncurated venue (−0.2), weekday/date conflict (−0.3), am/pm inferred (−0.2), model-only field with no regex anchor (−0.25), poster unresolved (−0.1). parse_status: 0 issues → ok; issues but date+venue+start present → partial; else needs_review.
S10 · Review queue
review.jsonl + a D1 view. Weekly Telegram digest: "14 sessions need review" with raw text inline so you can accept/correct in one reply. Corrections are overrides keyed by message_id, applied after parsing, so a parser upgrade never eats your fixes.
4. Edge cases & the calls I'd make
| id | case | decision |
|---|---|---|
| E1 | Timeline date header only updates on real scroll events | Scroll with real wheel events; record (header, first visible id) per step. Session dates come from post text anyway, so a fuzzy posted_day can't corrupt the calendar. |
| E2 | Poster image carries the details, body says "see poster" | needs_review + has_media=1. OCR later if it turns out to be common. |
| E3 | Chinese-only post | Same pipeline; model handles 中文 labels (日期/时间/地点/水平); regex pass is additive, not required. |
| E4 | "DM me for details", no time | partial: date + venue + poster recorded, start_time NULL. |
| E5 | Message edited after scrape | v1: text_hash compared when recent messages are re-scraped. Full edit hooks: v2. |
| E6 | Session fills up / cancelled later | v1: ignore. v2: session_updates fed by "full", "cancelled", "left 1 slot". |
| E7 | Recurring weekly fixture | One row, is_recurring=1, rrule set; calendar shows next occurrence. |
| E8 | Two organisers, same court + time | Both kept — poster is in the dedupe key (§3 S8). Site groups them visually. |
| E9 | Poster has no public username | contact_url = message permalink (t.me/sgbadmintontelecom/<id>). Everyone can still be reached from inside the group; nothing private is published. |
| E10 | Display name changed between posts | Same poster_key if the handle resolves; otherwise two rows that you can merge in the weekly digest. |
5. Quality gates
- Fixture set: 30 hand-labelled posts (covering E2–E10, EN + 中文) in
parse/fixtures.jsonl. Any parser change must hit ≥90% field accuracy on date/start/venue before shipping. - No-hallucination check: every extracted venue/date/time token must appear in (or be a documented transform of) the redacted raw text; failures log to
issues_jsonand drop confidence. - No-phone check: regex sweep over every table and every API response for SG phone shapes must return zero hits — enforced as a test, not a promise.
- Run audit:
runsrow per execution; parse must balanceseen = ok + partial + review + skipexactly. - Idempotency: re-running parse+upsert changes zero rows.
- Site: API count == D1 count for the same filter; 390px layout checked; a session with no handle still shows a working contact link.
6. PII policy
Stance: collect the minimum, publish only a way to reach a public account.
- Phone numbers are redacted at capture and never enter the DB — not extracted, not parsed, not stored.
- Contact happens through
@handle(a public identity the poster chose to expose) or the message permalink. - Display names: kept in the DB for dedupe/provenance; the public site shows the handle, not the display name.
- If a poster has both, handle wins. If neither, the permalink is the contact.
7. Decisions (locked 16 Nov 2026)
| id | decision |
|---|---|
| Q1 | Public site, handles + message permalinks only. No sender full names, no phone numbers. |
| Q2 | Scrape + publish at 07:00 and 21:00 SGT. |
| Q3 | Sessions-only cards. No raw full-text search on the site (raw text stays in D1 for the review queue). |
| Q4 | Parse model pinned to deepseek-v4.1-flash (canonical deepseek/deepseek-v4.1-flash-1017) on the opencode-go endpoint. |
| Q5 | Hard stop at 1 Nov 2026. No deep backfill for v1. |
| Q6 | Level ladder HB < LI < MI < HI < ADV confirmed; unknown levels become ANY. |
| Q7 | Resolve handles only for senders who post sessions. Cache is an LRU of 200 names persisted to state/posters.json (survives cron restarts), evict least-recently-used. No posters table. |
Open Nothing. This doc is the approved spec — next action is build step 1 (Go toolchain + D1 schema + repo skeleton), which produces no user-visible change.
8. Build order (after your calls)
- Go toolchain under
/opt/data/tools/go, D1 database + schema + migrations + repo skeleton,go.mod(no scraping yet). - Scraper + day bucketing; dump 2 Nov → now into
raw/; report counts before parsing. - Poster resolution pass (profile peek) for the senders seen in that dump.
- Fixtures + regex pass + validators; measure on the fixture set.
- Model pass for the remainder; review queue + weekly digest.
- Venue curation of the top 30; geocode.
- Pages site: API, calendar, list, contact resolution, filters; deploy + verify.
- Cron twice daily + failure alerts + vault note.
Plan file: /opt/data/.hermes/plans/2026-11-16_214800-badminton-sessions-tracker.md. Decisions Q1–Q7 gate step 1.