Badminton Session Tracker

Plan doc v0.4 · 16 Nov 2026 · source: Telegram @sgbadmintontelecom (chat -1001191346636) · target: Cloudflare Pages + D1

v0.3: @handle replaces phone-number collection · dedupe key includes the poster · diagram added. v0.4: Q1–Q7 locked, poster cache is an in-memory LRU(200) — no D1 table, model pinned to deepseek-v4.1-flash.

1. System architecture

1 · COLLECT (glastrier) 2 · CLEAN (agent pod) 3 · SERVE (cloudflare) Telegram channel @sgbadmintontelecom chat -1001191346636 chromium (igbrowser) glastrier 10.21.4.4 CDP :9223 · logged-in badminton-sync scrape Go · wheel-scroll · resume data-message-id keys raw/*.jsonl append-only, verbose source of truth poster resolve profile peek → @handle poster cache · LRU 200 state/posters.json cron 07:00 / 21:00 SGT fail-loud → Telegram S1 classify session ad vs chatter skip + reason S2–S7 normalise regex first, LLM tail date · time · venue · level S8 dedupe + S9 score date+start+venue+poster confidence, issue list staging/*.jsonl sessions · review + parse_log review queue → you overrides (human fixes) upsert (D1 REST) idempotent by session id + export backup first D1 · badminton raw · sessions · venues cache (LRU) · runs · parse_log Pages fn /api/sessions filter day · venue · level no phone fields ever badminton.rowentey.xyz calendar grid + list tap → @handle site reads D1 Solid arrow = data flow · dashed = control/back-reference. Everything left of the D1 box is rebuildable from raw/*.jsonl.

Rule The raw message table is the source of truth. Every other table can be dropped and rebuilt from raw/*.jsonl in one command. A bad parse run is re-derived, never patched in place.

Scope

1b. Tech stack & how a run is triggered

Stack

layerchoicewhy
browserchromium container igbrowser on glastrier, CDP :9223already logged in; only thing that must live on a host with a GUI session
scraper + parser + upsertGo 1.25, one binary badminton-sync (cmd/sync, internal/cdp, internal/parse, internal/d1), CDP over WebSocketno Python anywhere in the pipeline; static binary, real concurrency for the CDP event stream, easy to run from cron. Toolchain installed under /opt/data/tools/go (tarball, no root)
LLM passone HTTP POST from the same Go binary to https://opencode.ai/zen/go/v1/chat/completions (opencode-go plan, key OPENCODE_GO_API_KEY), model pinned deepseek-v4.1-flash (canonical deepseek/deepseek-v4.1-flash-1017), JSON mode, temp 0verified: that base URL is the Go-plan endpoint (returns the model list); this model advertises structured output. Chosen by you over the cheaper flash alternates
DBCloudflare D1 (SQLite) over the REST APIno server to babysit; token already in /opt/data/.cloudflare_api.env
site + APICloudflare Pages: TypeScript everywhere — functions/api/sessions.ts (Pages Function, native TS) + web/src/*.ts bundled by esbuild to public/app.jstypes shared between the Function and the client; wrangler handles the Function, one esbuild step for the client
schedulerscripts/badminton_sync.sh under Hermes cron, 07:00 + 21:00 SGT, flock single-instance, wrapping the Go binaryone entrypoint, one lock, one log
config/secrets/opt/data/.cloudflare_api.env (600), provider key from OPENCODE_GO_API_KEYnothing new to store
testsgo test ./... over parse/testdata/fixtures.jsonl + a site smoke check (curl + headless DOM)parser accuracy is a number, not a vibe

Not used, on purpose: no Python, no queue/broker, no k8s workload, no Postgres, no client framework/Astro. Volume is a few thousand messages per day at worst — anything more is machinery for machinery's sake.

Trigger chain

Nothing "triggers" a remote agent: the scraper runs on the pod and drives the browser over CDP. The pod is both scheduler and worker, so the trigger is just cron → one script, run in order with hard gates.

cron 07:00 / 21:00 SGT
  └─ flock -n /opt/data/tmp/badminton.lock
       badminton_sync.sh → badminton-sync (Go)
         1. preflight  CDP /json/version
                       chat header, no JOIN btn
         2. scrape     → raw/YYYY-MM-DD_HHMM.jsonl
         3. resolve    handles for session senders (LRU 200)
         4. parse      → staging/{sessions,review}.jsonl
         5. backup     D1 export → backups/<date>.sql
         6. upsert     D1 REST, idempotent
         7. deploy     only if sessions changed
         8. report     runs row + Telegram digest
  • Each step checks the previous exit code; any failure stops the chain and posts the failing step + stderr tail to Telegram. A half-run never reaches D1.
  • Steps 2–4 are re-runnable by hand: badminton_sync.sh --from-scrape / --parse-only.
  • Re-entry safety: flock -n means a slow run skips the next tick instead of double-writing; state.json holds the lowest scraped message_id for resume.
  • Login-wall detection (the chromium session expired) is a step-1 failure with its own alert text, because fixing it needs your phone, not a retry.
  • If this ever needs to be decoupled, the seam is a jobs row in D1 (queued → running → done) that the same script polls. Deliberately not built now.

2. Data schema (D1 / SQLite)

Five tables. raw_messages mirrors Telegram; sessions is one row per bookable slot (with the poster denormalised on it); venues is curated; runs/parse_log are the audit trail.

raw_messages — what Telegram gave us

columntypenotes
message_idINTEGER PKTelegram's own id (data-message-id). Monotonic, dedupe key.
posted_dayTEXT (YYYY-MM-DD)From timeline date header. Day-accurate at best — see E1.
posted_timeTEXT (HH:MM)SGT, from .message-time.
sender_nameTEXTDisplay name as rendered ("Roy LHH"). Not unique, not an identity.
textTEXT NOT NULLVerbatim with phone numbers redacted at capture (9xxx xxxx → [redacted]). Original never stored.
has_mediaINTEGER1 if the post had a photo/poster (details often live in the image → needs_review).
permalinkTEXThttps://t.me/sgbadmintontelecom/<message_id> — always available, no PII.
scraped_atTEXTISO timestamp of the scrape run.

contact resolution — no posters table

Posters are not a table. Handles are resolved only for senders whose post classified as a session — a few per run, not every member who speaks. The cache is an LRU of 200 display names backed by a small JSON file (state/posters.json, written atomically at the end of each run), so the next cron run starts warm instead of re-peeking the same organisers. Resolved values are also denormalised onto the session row, so history survives even if the cache file is deleted.

Contact resolution order (what the site shows): ① handle found in the post body (@name) · ② handle from the cached profile peek → https://t.me/<handle> · ③ message permalink t.me/sgbadmintontelecom/<message_id>, which always works from inside the group. Never a phone number.

Why a file and not D1: it is derived data, rebuildable by re-peeking, and it must survive process exit. No migration, no table to keep honest, no write amplification on the DB. LRU-200 keeps it bounded (~15 KB).

sessions — what the site queries

columntypenotes
idTEXT PKsha1(session_date|start_time|venue_key|poster_key) — stable across re-parses.
message_idINTEGER FKProvenance. One message can yield several sessions (multi-slot posts).
poster_nameTEXTDisplay name as rendered. poster_key in the dedupe key = handle if resolved, else this name.
poster_handleTEXT@name if resolved (post text or profile peek), else NULL. Denormalised so the row survives the cache.
contact_urlTEXTt.me/<handle> or the message permalink; never a raw phone number.
session_dateTEXT (YYYY-MM-DD)Normalised. NULL only when the post genuinely omits a resolvable date.
start_time / end_timeTEXT (HH:MM)24h SGT. end_time NULL if unstated.
is_recurring / rruleINTEGER / TEXTWeekly fixtures ("every Tue 8-10pm") — calendar renders the next occurrence.
venue_keyTEXT FKSlug into venues; NULL when unmatched (parked in review).
courts / courts_noteINTEGER / TEXT"3.5 courts" → 3 + note.
level_raw / level_normTEXTRaw string kept forever; normalised to the LI/MI/HI ladder (§3 S6).
price_raw / price_cents / price_basisTEXT / INTEGER / TEXTbasis ∈ per_pax, per_court, unknown.
shuttle_brandTEXT"RSL Supreme" — people care.
slots_rawTEXT"last 2 slots" etc., shown but not machine-usable.
parse_statusTEXTok · partial · needs_review · skip
confidenceREAL0–1 from the validators in §5, not the model's self-report.

venues

columntypenotes
venue_keyTEXT PK"yishun-sports-hall"
name / areaTEXT"Yishun Sports Hall" / "North"
aliasesTEXT (JSON)["Yishun SH", "YSportshall"] — the fuzzy matcher's training data.
lat / lonREALGeocoded once via OSM/Nominatim, cached. Enables the map later.
curatedINTEGER1 = human-confirmed. Uncurated venues never auto-merge.

runs & parse_log

tablecolumns
runsid, started_at, finished_at, kind (scrape|parse|upsert), messages_seen, sessions_new, sessions_updated, status, note
parse_logmessage_id, parser_version, model, fields_json, issues_json, created_at — diff parser v3 vs v4 without re-calling the model

3. Cleaning pipeline

Posts are free-form, bilingual (EN / 中文), full of shorthand. Ten stages, each with a hard rule; anything that fails a stage lands in review.jsonl instead of guessing.

S0 · Capture

Wheel-scroll the chat, emit one JSON object per .Message node. Normalise whitespace (\n{3,} → \n\n), strip zero-width chars, keep emoji. Redact phone numbers on the way in (SG mobile 8/9xxxxxxx, +65 variants, and 4-4 spaced forms) → the DB never holds them.

S1 · Classify: session ad or not

Rules first, model second. Ad if ≥2 of: a date token, a time token, a venue token, a level token, a slot/price token. Chatter and pure adverts ("selling Yonex 88D") are skip with a reason code (no_date, no_venue, sales, coaching_biz).

Ambiguous ones ("anyone wants to play this weekend?") → needs_review, never a guess: they lack date+venue by design.

S1b · Poster resolution

The message list exposes only a display name — no username, no peer id. So: the message list exposes only a display name — no username, no peer id. After S1 says "this is a session ad", open that sender's profile once in the web client (read-only click), read @username if present, and keep it in an in-memory LRU of 200 display names (least-recently-used evicted) for the rest of the run. Only session posters are peeked — chatter is never resolved. Handles appearing inside post bodies are captured by regex and win over the peek. Unresolved → permalink fallback. Nothing about posters is persisted except the denormalised field on the session row.

S2 · Extract

Regex pass for the labelled shapes this channel actually uses — Date:, Time:, Venue:, Level:, @venue, free-text lines. Model pass only for what regex leaves empty, batched 20 posts/call, JSON-schema output, temperature 0. Missing → null. The model may only copy spans from the text — every non-null field is checked as a substring/derivative of the raw text (§5).

S3 · Dates

input seen in channelruleoutput
21 Nov 2026, Satabsolute2026-11-21
17/ 11/2026D/M/Y (SG default); reject M/D/Y if D>122026-11-17
21/11/26DD/MM/YY, century = 2000s2026-11-21
TUESDAY, tonight, this Satresolve against posted_day; roll forward if the word'd weekday is behindcomputed + date_assumed=1
weekday mismatch ("21 Nov 2026, Sun")trust the numeric date, log date_conflict, confidence −0.32026-11-21
no date at allno inference from posted_dayNULL + needs_review

Sessions dated before the run are archived, not deleted (is_past derived) so the calendar can show history.

S4 · Times & multi-slot posts

S5 · Venues

Three passes: exact alias hit → normalised fuzzy (lowercase, strip @, fold "SH"/"Sports Hall"/"Sportshall", drop parentheticals) with token-set ratio ≥ 0.85 → model tiebreak over the top-3 candidates, candidates recorded in parse_log. New venues are created curated=0 and surface in a weekly "new venues" digest; uncurated venues are never auto-merged.

S6 · Levels

SG ladder: HB < LI < MI < HI < ADV. Parse ranges ("LI to HI", "HB-LI"); anything else ("intermediate", "social") → level_norm='ANY' with raw kept. "intermediate" is never silently mapped to MI.

S7 · Price, courts, shuttles

$7/pax → 700, per_pax; $42/court → 4200, per_court; free → 0; "share cost" → basis=unknown. Courts: "3.5 courts" → courts=3 + note.

S8 · Dedupe & merge

S9 · Validate & score

Confidence starts at 1.0, docked per issue: assumed date (−0.15), assumed end (−0.1), uncurated venue (−0.2), weekday/date conflict (−0.3), am/pm inferred (−0.2), model-only field with no regex anchor (−0.25), poster unresolved (−0.1). parse_status: 0 issues → ok; issues but date+venue+start present → partial; else needs_review.

S10 · Review queue

review.jsonl + a D1 view. Weekly Telegram digest: "14 sessions need review" with raw text inline so you can accept/correct in one reply. Corrections are overrides keyed by message_id, applied after parsing, so a parser upgrade never eats your fixes.

4. Edge cases & the calls I'd make

idcasedecision
E1Timeline date header only updates on real scroll eventsScroll with real wheel events; record (header, first visible id) per step. Session dates come from post text anyway, so a fuzzy posted_day can't corrupt the calendar.
E2Poster image carries the details, body says "see poster"needs_review + has_media=1. OCR later if it turns out to be common.
E3Chinese-only postSame pipeline; model handles 中文 labels (日期/时间/地点/水平); regex pass is additive, not required.
E4"DM me for details", no timepartial: date + venue + poster recorded, start_time NULL.
E5Message edited after scrapev1: text_hash compared when recent messages are re-scraped. Full edit hooks: v2.
E6Session fills up / cancelled laterv1: ignore. v2: session_updates fed by "full", "cancelled", "left 1 slot".
E7Recurring weekly fixtureOne row, is_recurring=1, rrule set; calendar shows next occurrence.
E8Two organisers, same court + timeBoth kept — poster is in the dedupe key (§3 S8). Site groups them visually.
E9Poster has no public usernamecontact_url = message permalink (t.me/sgbadmintontelecom/<id>). Everyone can still be reached from inside the group; nothing private is published.
E10Display name changed between postsSame poster_key if the handle resolves; otherwise two rows that you can merge in the weekly digest.

5. Quality gates

6. PII policy

Stance: collect the minimum, publish only a way to reach a public account.

7. Decisions (locked 16 Nov 2026)

iddecision
Q1Public site, handles + message permalinks only. No sender full names, no phone numbers.
Q2Scrape + publish at 07:00 and 21:00 SGT.
Q3Sessions-only cards. No raw full-text search on the site (raw text stays in D1 for the review queue).
Q4Parse model pinned to deepseek-v4.1-flash (canonical deepseek/deepseek-v4.1-flash-1017) on the opencode-go endpoint.
Q5Hard stop at 1 Nov 2026. No deep backfill for v1.
Q6Level ladder HB < LI < MI < HI < ADV confirmed; unknown levels become ANY.
Q7Resolve handles only for senders who post sessions. Cache is an LRU of 200 names persisted to state/posters.json (survives cron restarts), evict least-recently-used. No posters table.

Open Nothing. This doc is the approved spec — next action is build step 1 (Go toolchain + D1 schema + repo skeleton), which produces no user-visible change.

8. Build order (after your calls)

  1. Go toolchain under /opt/data/tools/go, D1 database + schema + migrations + repo skeleton, go.mod (no scraping yet).
  2. Scraper + day bucketing; dump 2 Nov → now into raw/; report counts before parsing.
  3. Poster resolution pass (profile peek) for the senders seen in that dump.
  4. Fixtures + regex pass + validators; measure on the fixture set.
  5. Model pass for the remainder; review queue + weekly digest.
  6. Venue curation of the top 30; geocode.
  7. Pages site: API, calendar, list, contact resolution, filters; deploy + verify.
  8. Cron twice daily + failure alerts + vault note.

Plan file: /opt/data/.hermes/plans/2026-11-16_214800-badminton-sessions-tracker.md. Decisions Q1–Q7 gate step 1.