diff --git a/README.md b/README.md index 126430b..c3a9cd4 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,63 @@ # bike-app -A self-hosted cycling app: syncs rides from a **Bryton Rider 650**, tracks mileage like Strava, and -adds a **spare-parts inventory** and a **maintenance record** with mileage-milestone reminders. +A self-hosted cycling app ("Velodrome"): syncs rides from a **Bryton Rider 650**, tracks mileage +like Strava, and adds a **spare-parts inventory** and a **maintenance record** with +mileage-milestone reminders. -**Status: planning complete, no code written yet.** +**Status: Phase 0 complete and deployed.** Auth, the single-container image, CI/CD into a +self-hosted Gitea registry, and a real HTTPS deployment are live and verified. No ride ingestion +yet — that's Phase 1. See the roadmap in [`docs/PLAN.md`](docs/PLAN.md). ## Why -Today the Rider 650 syncs over Bluetooth to the Bryton Active phone app, which forwards to Strava — -but Active has no background sync, so you have to remember to open the app. Research found a better -path that removes the phone entirely: +The Rider 650 syncs over Bluetooth to the Bryton Active phone app, which forwards to Bryton's cloud +and on to Strava. Two problems: Active has no background sync, so you have to remember to open the +app; and Strava's API can only ever hand back decoded, smoothed streams — never the original file. + +This app polls Bryton's cloud directly and takes the **original, unmodified FIT bytes**: ``` -ride ends -> Rider 650 joins home Wi-Fi (Main Menu -> Data Sync) - -> uploads to Bryton cloud +ride ends -> BLE -> Bryton Active app -> Bryton cloud -> this app's poller fetches the ORIGINAL FIT file -> rides, wear tracking, and push reminders ``` -That's also *higher fidelity* than the current route — Strava's API can only ever return smoothed -streams, never the original file. +**What that does and doesn't fix.** It does not remove the phone: you still open Active once after a +ride, and nothing in this app can reach across that gap (the Rider 650 has no Wi-Fi, and its BLE +sync protocol is undocumented — see `docs/PLAN.md`, "How rides actually reach the app"). What it +fixes is everything after that tap — full-resolution original bytes in your own database, every +field the head unit recorded, wear recalculated, reminders armed, and no third party able to change +the terms later. ## Docs | File | What's in it | |---|---| | [`docs/PLAN.md`](docs/PLAN.md) | The full implementation plan: stack, schema, ingestion pipeline, auth, notifications, phased roadmap, CI/CD, risks, verification | -| [`docs/RESEARCH.md`](docs/RESEARCH.md) | Raw findings: the Bryton cloud protocol (endpoints, headers, auth), FIT library comparisons, maintenance interval tables, self-hostable geo services, Gitea Actions gotchas | -| [`docs/DECISIONS.md`](docs/DECISIONS.md) | Every decision taken, what was rejected, and why | +| [`docs/DECISIONS.md`](docs/DECISIONS.md) | Every decision taken, what was rejected, and why — including the ones later reversed, with the reasoning intact | +| [`docs/RESEARCH.md`](docs/RESEARCH.md) | Raw findings: the Bryton cloud protocol, FIT library comparisons, maintenance interval tables, self-hostable geo services, Gitea Actions gotchas | +| [`CLAUDE.md`](CLAUDE.md) | Conventions, non-negotiable invariants, branching and PR workflow | +| [`deploy/README.md`](deploy/README.md) | How to build, run, and bootstrap the deployed container | -## Planned stack +## Stack as built -Python 3.12 / FastAPI / SQLAlchemy async / PostgreSQL 16 + PostGIS, `procrastinate` for jobs, -SvelteKit static SPA as an installable PWA, MapLibre GL, all behind Caddy in Docker Compose. -Source control and CI in self-hosted Gitea with an act_runner on the same box. +Python 3.12 / FastAPI / SQLAlchemy 2.0 async / **SQLite** (D15 — reversed the original +Postgres+PostGIS choice mid-Phase-0), SvelteKit static SPA as an installable PWA, MapLibre GL to +come in Phase 1, all served by Caddy from a **single container** (D16). Source control and CI in +self-hosted Gitea with act_runner on the same host. -Four containers in v1, under 2GB RAM. +The two consequences of the SQLite decision worth knowing before reading any code: user isolation is +enforced entirely in the repository layer (`apps/api/velodrome/db.py`'s `Scope`), with no database +RLS behind it; and the original job-queue choice (`procrastinate`, Postgres-only) needs a +replacement before Phase 1's ingestion pipeline can be built. ## Next steps -1. **Verify on the Rider 650:** does `Main Menu -> Data Sync` upload *automatically* on joining - Wi-Fi, or only on manual trigger? This determines how completely the phone leaves the loop. -2. **Plug the 650 in over USB** and `ls -R` the mounted volume to confirm the real `.fit` path - (documented as `Bryton/Activities/`, but worth confirming). -3. **Grab a real `.fit` file** from it and run it through `fitdecode` — Bryton's encoder is not - Garmin's, and the schema should be checked against reality before it's written. -4. Then Phase 0: scaffolding and CI (see the roadmap in `docs/PLAN.md`). +1. **Pick the two Phase 1 blockers** deferred by D15: the background job queue, and how to store + ride tracks without PostGIS. +2. **Verify against the physical device before building on it** (the lesson of D20): the USB `.fit` + path layout, and that the `intervalssync` protocol still retrieves activities from a current + Bryton account. +3. **Grab a real `.fit` file** and run it through `fitdecode` — Bryton's encoder is not Garmin's, + and the schema should be checked against reality before it's written. +4. Then Phase 1: ingestion (see the roadmap in `docs/PLAN.md`). diff --git a/deploy/README.md b/deploy/README.md index 80fdc5c..892b5be 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -154,8 +154,20 @@ It does **not** SSH into the host and recreate the running container — rolling Unraid (pulling it and clicking "Apply" on the container, or via Unraid's own update-checking) is left as a manual/Unraid-side step, not something CI does unattended. +Unraid's own "check for updates" is **not a reliable signal for this container specifically** — see +`docs/DECISIONS.md` D19. Because Gitea Actions builds on this same host's `dockerd`, every CI run +keeps the local `:latest` tag fresh regardless of whether the *running container* was ever +recreated from it, so the checker can say "up to date" while the running container is genuinely +stale. Don't wait for that badge; recreate deliberately after a merge you know should ship. + ## What's not here yet -Backups (`docs/PLAN.md` calls for a systemd timer running `restic` against `/data`, independent of -CI) and the `import_inbox` USB-watch bind mount are both Phase 1+ concerns — nothing in the schema -uses them yet. +- Backups (`docs/PLAN.md` calls for a systemd timer running `restic` against `/data`, independent + of CI) and the `import_inbox` USB-watch bind mount — both Phase 1+ concerns, nothing in the + schema uses them yet. +- An auto-updater for the running container (attempted with Watchtower, deferred — D19). +- Persisting the host-local trust material from D17/D19 (`/etc/hosts`, `certs.d`, the CA bundle + entry) across a reboot — currently lost on restart, deliberately left that way pending a + decision about editing `/boot/config/go` (D19). +- Migrating Gitea + its Actions runners off this Unraid host onto a dedicated VM — the root cause + behind several of the fixes above, raised as a real future decision, not started (D19). diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index fdac4a5..cf690cd 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -26,18 +26,31 @@ Bryton BLE is a dead end regardless, and the *server* does all syncing. **Kept as insurance:** the backend stays strictly API-first with a CI-enforced OpenAPI contract, so if Apple ever makes the PWA route untenable, a native client is a code-generation exercise. -### D3 — Bryton cloud poller is the primary ingestion path +### D3 — Bryton cloud poller is the primary ingestion path — **premise corrected, decision survives** **Chosen:** server-side poller against the reverse-engineered Bryton Active API, every 15-20 min. **Rejected:** Strava as a source (no `export_original` — decoded smoothed streams only; plus the June 2026 tier restructure caps new apps at 10 users and requires a paid dev subscription); BLE/ANT-FS direct (nobody has reverse-engineered Bryton's BLE — weeks of work, breaks on firmware -updates); depending on the Bryton Active phone app (the original complaint). -**Why:** the Rider 650 has on-device Wi-Fi (`Main Menu -> Data Sync`) and uploads to Bryton's cloud -with no phone involved, and the cloud API returns the **original unmodified FIT bytes**. That's both -zero-touch *and* higher fidelity than the current Strava route. +updates, and separately impossible from an iOS PWA, which has no Web Bluetooth at all). +**Why:** the cloud API returns the **original unmodified FIT bytes** — higher fidelity than the +Strava route, and everything downstream of the cloud is ours. **Fallbacks, both built:** USB watch folder (also the historical-backfill mechanism, so it stays exercised rather than bit-rotting) and manual upload. +> **Corrected 2026-09-22.** This entry originally justified itself with "the Rider 650 has on-device +> Wi-Fi (`Main Menu -> Data Sync`) and uploads to Bryton's cloud with no phone involved… That's both +> zero-touch *and* higher fidelity," and listed "depending on the Bryton Active phone app (the +> original complaint)" as *rejected*. **The Wi-Fi premise was false** — the Rider 650 has ANT+ and +> Bluetooth only, and its only sync route is BLE to the Active app (confirmed on the physical +> device; see `docs/PLAN.md`, "How rides actually reach the app"). So the rejected option is in fact +> the only one available, and the chain is +> `head unit → BLE → Active app → Bryton cloud → poller`. +> +> **The decision itself still stands** — polling Bryton's cloud for original FIT bytes remains the +> best available primary path, and nothing downstream of the cloud depended on how rides got into +> it. What changes is the *claim*: this is one-tap, not zero-touch, and it does not fix the original +> complaint. See D20 for the process lesson. + ### D4 — Python / FastAPI / Postgres+PostGIS — **database choice superseded by D15** **Chosen:** Python 3.12, FastAPI, Pydantic v2, SQLAlchemy 2.0 async, Alembic, PostgreSQL 16 + PostGIS 3.4. **Rejected:** TypeScript full-stack, Go. @@ -358,8 +371,100 @@ the deployed image has to carry. --- +### D19 — Auto-update: attempted, deferred; Unraid's own update checker needed a separate fix + +**The immediate bug:** Unraid's Docker "check for updates" reported `not available` for `velodrome` +after D17's registry move. Root cause, found by reading the actual PHP source +(`dynamix.docker.manager`'s `DockerClient.php`): it queries the registry's manifest API directly +over `curl` from PHP, which is a completely different trust path from `dockerd`'s own — it doesn't +read Docker's `/etc/docker/certs.d` at all, only the OS-wide CA bundle. **Fixed** by also adding the +D17 self-signed cert to `/usr/local/share/ca-certificates/` and running `update-ca-certificates` on +the Unraid host — a third, independent place this cert now needs to be trusted (alongside +`certs.d` and the `/etc/hosts` entry from D17), and like those two, not yet persisted across a +reboot (`/boot/config/go` again — same deliberate non-decision as D17). + +**A second, structural problem this exposed, not fixed:** even with the checker itself working, +"up to date" on this host doesn't reliably mean the *running container* matches the registry. +Gitea Actions builds directly on this same host's `dockerd` (DooD), which means every CI build also +leaves its own result sitting in the **local image cache** tagged `:latest` — so the local-vs-remote +digest comparison Unraid's checker does is comparing the registry against a tag that CI keeps fresh +on its own, independent of whether the `velodrome` *container* was ever recreated from it. Confirmed +directly: the checker reported "up to date" while the running container's actual manifest digest +(read via `docker inspect`) provably differed from the registry's current `Docker-Content-Digest`. +This is a consequence of building CI on the same host as the app runs, not a bug to patch around — +see the Gitea-to-VM item below. + +**Attempted: Watchtower**, label-scoped (`WATCHTOWER_LABEL_ENABLE=true` + a +`com.centurylinklabs.watchtower.enable=true` label on `velodrome` only, specifically so it can never +touch any of the ~40 other containers on this host) with the CA bundle mounted in for the same +registry-trust reason as above. **Failed on the first attempt** — `containrrr/watchtower`'s +published image talks a Docker API version (1.25) too old for this host's `dockerd`, a stale-image +problem, not a design problem. Not yet retried with a maintained fork. The `velodrome` container +does carry the watch-enable label already (added when it was recreated to pick up D18's CLI), so +turning this on later is "run the right watchtower image," not "redesign anything." + +**Why not have CI redeploy the container directly** (it already has host `dockerd` access via DooD): +considered and explicitly rejected, again — see D16/D17's reasoning, which this doesn't change. +Turning every merge to `main` into an unattended production change on a personal server is a bigger +step than "install an auto-updater," and wasn't asked for. + +**Until this is finished:** redeploying after a merge is `docker pull` + recreate, same as any +manual deploy — `deploy/README.md`'s "Publishing the image" section. + +--- + +### D20 — The Rider 650 has no Wi-Fi; verify device capabilities on the device + +**What happened:** the plan's headline section, "The sync breakthrough," asserted that the Rider 650 +has on-device Wi-Fi and a `Main Menu → Data Sync` entry that uploads rides to Bryton's cloud with no +phone involved. It does not. The Rider 650 has ANT+ and Bluetooth only; its sole sync route is BLE to +the Bryton Active app. Confirmed on the physical device, and corroborated by BikeRadar's hands-on +("ANT+ and Bluetooth connectivity", syncing via "Bryton's Active App"). The most likely origin of +the error is conflation with the Rider 750 / S800, which do have Wi-Fi. + +**Why it survived so long:** the plan *did* contain the right check — "First action before writing +any code: on the Rider 650, go to `Main Menu → Data Sync`… and confirm a test ride uploads without +the phone." It was never run, and nothing downstream required it to have been. An entire phase was +planned, and Phase 0 fully built and deployed, on top of an unverified device capability that was +written down in the declarative voice of a finding rather than the provisional voice of an +assumption. + +**What it cost, and didn't:** less than it first appeared. Everything downstream of Bryton's cloud — +ingestion, dedupe, schema, wear engine, garage, notifications, all of Phase 0 — never depended on +how a ride reached that cloud, so no built code was invalidated. What was invalidated was the +*product promise*: Phase 1 was called "Zero-touch ride history" and claimed to fix the original +complaint (having to remember to open the Active app). It does not. It is one-tap, and the +complaint stands. That renaming, not a refactor, was the actual repair. + +**The rule going forward:** a physical-device capability that a phase depends on is confirmed **on +the device** before it is written down as fact. Model-adjacent sources (a spec page for a different +unit in the same family, a review of a sibling model) do not count. Until confirmed, such a claim is +written as an open question in `docs/RESEARCH.md`, not as a premise in `docs/PLAN.md` — and any +phase resting on it carries the verification as its first task, not as a footnote. The same applies +to the remaining unverified device claims: the USB `.fit` path layout, and whether the +`intervalssync` protocol still retrieves activities from a current Bryton account. + +--- + ## Deliberately deferred +- **Finish the Watchtower auto-updater** (D19) — retry with a maintained image; `velodrome` is + already labeled for it. +- **Migrate Gitea + its Actions runners to a dedicated VM**, off the Unraid host the app itself + runs on. Raised explicitly (not yet started) after D17/D19 both turned out to be fighting the + same root cause from different angles: CI sharing a `dockerd` with ~40 unrelated production + containers means every registry-trust fix and every update-check quirk this session hit was more + contained, and more repeatable to reason about, than it should have needed to be. A dedicated VM + removes that coupling entirely — CI's own Docker config becomes free to change without any + blast-radius conversation about Plex or Vaultwarden ever again. Real migration work (new VM, + moving Gitea's and both runners' appdata, re-pointing `192.168.0.3`, updating every reference to + it across this repo and this session's own tooling), not a quick fix — a deliberate choice to do + later, not an oversight now. +- **Persist the D17/D19 host-local trust files across a reboot** (`/etc/hosts`, `certs.d`, the CA + bundle addition) via `/boot/config/go`. Left un-persisted through both decisions specifically + because editing anything under `/boot` was raised as a real concern mid-session — worth revisiting + together once, for all three at once, rather than as three separate asks. + - **Routing** (Valhalla/Photon/Overpass) — Phase 5, optional. Several GB of RAM for something Komoot already does well. - **Local LLM ride summaries** (Ollama) — Phase 4, behind a compose profile. diff --git a/docs/PLAN.md b/docs/PLAN.md index f85e75e..82ba6b5 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -9,8 +9,8 @@ self-hosted Gitea and an act_runner on the same box for automated builds. The specific pain driving this: today your Rider 650 syncs over Bluetooth to the Bryton Active phone app, which forwards to Strava — but Active has no background sync, so **you have to remember -to open the app**. Research turned up a better path that removes the phone from the loop entirely -(see "The sync breakthrough" below). +to open the app**. This app is not, on its own, able to fix that last part — see "How rides actually +reach the app" below for why, and what it does fix. You also want **mileage-milestone notifications** — "every 200 miles, clean and lube the drivetrain" — which makes the maintenance side push-based rather than something you have to remember to go look at. @@ -24,35 +24,57 @@ Directory is empty; this is greenfield. Decisions already made: --- -## The sync breakthrough +## How rides actually reach the app -Two facts verified during research change the design: +> **Corrected 2026-09-22, after checking the actual device.** An earlier version of this plan opened +> with a "sync breakthrough": the claim that the Rider 650 has on-device Wi-Fi and a `Data Sync` +> menu that uploads to Bryton's cloud with no phone involved. **That is false.** The Rider 650 has +> ANT+ and Bluetooth only — no Wi-Fi — and the only sync route it offers is Bluetooth to the Bryton +> Active app. The unit's menu has no `Data Sync` entry, and BikeRadar's hands-on confirms +> connectivity is "ANT+ and Bluetooth" with syncing via "Bryton's Active App." The likely source of +> the error is conflation with the Rider 750 / S800, which *do* have Wi-Fi. This mattered: it was +> the headline premise of the whole plan and it survived into a written roadmap unverified. The +> lesson is recorded in `docs/DECISIONS.md` — verify device capabilities against the physical device +> before building a plan on them, not against model-adjacent sources. -1. **Your Rider 650 has on-device Wi-Fi.** Its main menu has a `Data Sync` entry where the head unit - itself joins a Wi-Fi hotspot and uploads tracks to Bryton's cloud — **no phone, no Active app**. -2. **Bryton's cloud API is fully reverse-engineered** and returns the **original, unmodified FIT - bytes**. Working MIT reference implementation: `github.com/jorge-huxley/intervalssync` - (Python, updated 2026-09-17). - -So the pipeline becomes fully hands-off: +The real chain, which is what everything downstream is built on: ``` -ride ends → Rider 650 joins home Wi-Fi → uploads to Bryton cloud +ride ends → BLE → Bryton Active app on your phone → Bryton cloud → your server's poller fetches the original FIT → app ``` -**First action before writing any code:** on the Rider 650, go to `Main Menu → Data Sync`, join your -home Wi-Fi, and confirm a test ride uploads without the phone. If it only syncs on manual menu -trigger rather than automatically, the fallback is still good (USB, below) — but verify this, because -it determines whether Phase 2 fully solves your complaint. +**The one fact that still holds, and is the load-bearing one:** Bryton's cloud API is fully +reverse-engineered and returns the **original, unmodified FIT bytes**. Working MIT reference +implementation: `github.com/jorge-huxley/intervalssync` (Python, updated 2026-09-17). Everything +this app does downstream of Bryton's cloud — ingestion, dedupe, the schema, the wear engine, the +garage, notifications — never depended on *how* a ride got into that cloud, which is why losing the +Wi-Fi premise costs far less than it first appears. + +**What this does and doesn't fix.** It does not fix the original complaint. You still have to open +the Active app for a ride to leave the head unit; nothing in this app can reach across that gap +(see "Why not Bluetooth direct" below). What it does fix is everything *after* that: one tap and +the ride is permanently yours — full-resolution, original bytes, in your own database, with wear +recalculated and maintenance reminders armed, and no third party able to change the terms later. + +**Worth trying, costs nothing, no code:** an **iOS Shortcuts personal automation** to open Bryton +Active for you — triggered on the Rider 650's Bluetooth disconnecting, or on arriving home. If iOS +honours it reliably, most of the hands-free behaviour comes back without any architectural change. +Try this before concluding the one-tap step is permanent. **Why not Strava as the source:** Strava's API has no `export_original` endpoint — you get decoded, smoothed streams, never the original file. Its June 2026 tier restructure also caps new apps at 10 users and requires the *developer* to hold a paid Strava subscription. Dead end; skip it. -**Why not Bluetooth direct:** Bryton's BLE sync protocol is not reverse-engineered by anyone — -no Gadgetbridge support, no ANT-FS, no published UUIDs. No third-party app, native or otherwise, -can pull rides off the head unit. Explicitly out of scope. +**Why not Bluetooth direct:** Bryton's BLE sync protocol is not reverse-engineered by anyone — no +Gadgetbridge support, no ANT-FS, no published UUIDs. Independently of the protocol, **iOS gives web +apps no Bluetooth at all** (Web Bluetooth is unimplemented in WebKit, with no public Apple +position), so the installed PWA structurally cannot talk to the head unit even if the protocol were +known. Pulling rides off the unit directly would mean reverse-engineering an undocumented protocol +*and* running it on non-iOS hardware in the house (a Pi, an old Android phone). That's a research +project of unknown size, not a schedulable phase. Out of scope — but it is the only route that +would truly remove the phone, so it is the thing to revisit if the one-tap step ever becomes +intolerable. --- @@ -180,6 +202,35 @@ hashes), `api_tokens` (scoped, for Home Assistant/Grafana). `counts_for_wear` (user override). - `fit_time_created` + `device_serial` → partial unique index. This is the natural dedupe key. +**Capture everything the head unit emits, not a whitelist.** A hard requirement, not a nice-to-have: +whatever fields a Rider 650 puts in a FIT file should end up queryable and displayable, including +fields that aren't in the standard FIT profile. `fitdecode` surfaces all of it — every message type +(including ones it doesn't recognise, by message number), every field (unrecognised ones as +`unknown_`), and developer fields with their definition metadata. The parser must therefore be +**field-agnostic by construction**: iterate the messages that are actually present and persist what +is found, rather than reading a fixed list of known field names and silently dropping the rest. + +Concretely, three things this implies beyond the tables above: +- **`activity_streams` takes any channel.** `channel` is already a free string, so a new or unknown + per-record field (`unknown_61`, a developer field, a Bryton-specific extension) becomes a stream + row with no schema change. No whitelist anywhere in the parse path. +- **`activity_fit_messages`** — the non-time-series long tail, which the current tables have nowhere + to put: `device_info` (firmware, battery, every paired sensor), `event` (start/stop/lap triggers, + battery and sensor warnings), `hrv`, `zones_target`, `workout`/`workout_step`, `sport`, plus any + message type we don't recognise. Stored as `(activity_id, message_type, message_index, fields + json)` — JSON is correct here precisely because the shape is unknown and variable, which is the + opposite of the streams case where it's uniform and huge. +- **`activity_field_inventory`** — per activity, which channels and message types actually turned up, + with units and value ranges. This is what lets the UI render *"everything we got from this ride"* + dynamically instead of hardcoding a field list that goes stale the moment Bryton's firmware adds + something. It also makes "what does this head unit actually record?" answerable without scanning + every stream. + +None of this risks anything, because of invariant #1: the raw bytes are retained forever, so a +parser that learns to understand more fields later is a `parser_version` bump and a reparse, not a +migration or a data-loss event. Verbosity is a projection-widening exercise and can be iterated on +safely. + **Streams — columnar arrays, one row per channel** (`activity_streams`: `channel`, `n`, `scale`, `values_i32[]`). Decision and justification: - **Size.** A 3h ride at 1Hz × ~9 channels: normalized per-sample rows ≈ 1.2MB + 0.3MB index; @@ -226,6 +277,18 @@ contains every chain you retired since 2019. **Weather:** `weather_observations` keyed on a **0.05° grid cell + UTC hour**, so nearby rides reuse cached data, plus a per-activity `activity_weather` rollup. +**Live tracking (Phase 1B):** `live_sessions` (`user_id`, `started_at`, `ended_at`, +`share_token_hash` — only the hash, same discipline as invites and sessions — `expires_at`, +`obfuscate_endpoints_m`, and a nullable `activity_id` reconciled after the real FIT file arrives) +plus `live_positions` (`session_id`, `ts`, lat/lon as int32 semicircles, `altitude_cm`, +`speed_mms`, `accuracy_m`, and a nullable JSON column for whatever optional sensor metrics a +screen-on client manages to send). `live_positions` is append-only and high-write relative to +everything else here; it is also the **only** table in the schema that is deliberately *not* +permanent — once a session is reconciled to its activity, the positions are redundant against the +FIT file's own record, and can be pruned on a retention window without losing anything. This is the +single exception to "every table is a rebuildable projection of raw bytes," and it is an exception +precisely because live telemetry has no raw file behind it. + ### The wear engine `service_rules` carries `metric` (`distance | ride_time | calendar`), `threshold`, `basis` @@ -434,13 +497,22 @@ Other Bryton hardening: map nonstandard manufacturer/product IDs via serial pref values; store laps verbatim but **never derive session totals by summing laps** (use the `session` message); compute `moving_time` from records with speed > 0.5 m/s if absent. +**The parser reads what's there, not what it expects.** Per the "capture everything" requirement in +the Schema section: walk every message and every field `fitdecode` yields, persist unrecognised ones +under their raw identifiers (`unknown_`, developer fields with their definition metadata) rather +than skipping them, and record what was found in `activity_field_inventory`. A field the parser +doesn't have a name for is still worth storing and still worth showing — Bryton's encoder is not +Garmin's, and the whole point is to see everything the head unit actually recorded. A parser change +that *narrows* what gets captured is a regression, and the golden-fixture corpus should catch it: +assert on the field inventory of a known file, not just on the handful of summary numbers. + **Sources** implement a common `ActivitySource` protocol: - **Upload** — `POST /api/v1/uploads`, multipart, 50MB cap, accepts `.fit`, `.fit.gz`, `.gpx`, `.tcx`. - **USB watcher** — 60s periodic scan of `/import/inbox/**/*.fit`. A host udev rule on volume label `Bryton` mounts the device **read-only** and rsyncs into the inbox. **Discover the subfolder at runtime by recursive glob** — sources disagree on whether it's `Activities/`, `Actives/`, or root, so don't hardcode it (your 650 is documented as `Bryton/Activities/`, but verify). -- **Bryton cloud (Phase 2)** — Meteor DDP over SockJS to `m3.brytonactive.com`: `login` with the +- **Bryton cloud (Phase 1 — the primary path)** — Meteor DDP over SockJS to `m3.brytonactive.com`: `login` with the SHA-256 digest → `subscribe("activityList")` → read `userActivities` (**filter `_deleted` tombstones**) → diff against `raw_files.source_ref` → `GET /api/activity?id=` with `X-User-Id`, `X-Auth-Token`, `x-api-key`, `User-Agent: okhttp/4.12.0` → raw original FIT bytes. Poll every 20 min, @@ -499,13 +571,25 @@ lacks** — the cookie is transport convenience only. These are the payoff for self-hosting — things Strava structurally cannot do. **Free, because they're schema properties:** -- **No privacy zones, ever.** No third party holds your data, so show real door-to-door routes. +- **No privacy zones on your own archive.** No third party holds your data, so show real + door-to-door routes. (This reasoning covers the *private* archive only — a publicly shareable + live-tracking link is a different risk and gets its own treatment; see the live tracking phase.) +- **Every field the head unit recorded, not the handful a platform chose to keep.** Strava's API + gives you decoded, smoothed streams for a fixed set of channels; upload a FIT file there and the + unrecognised and vendor-specific fields are simply gone. Here the original bytes are retained + forever *and* the parser stores unknown and developer fields under their raw identifiers, so the + UI can show everything the Rider 650 actually wrote — including fields nobody has named yet. - **Unlimited full-resolution retention**, forever, of the original files. - **Cost-per-km on every component**, and per-kind averages ("my chains cost £0.019/km"). - **Receipts and photos attached** to parts, service events, and bikes. - **Wet-weighted wear** — rim pads genuinely wear ~4× faster in the rain, and you have the weather data. **Cheap and high value:** +- **Live tracking on your own terms** (Phase 1B) — a share link your family opens with no Bryton + account, no third party holding the trace, that keeps working if Bryton's service dies, and whose + history lands in your own database next to the ride it belongs to. Note honestly that Bryton's own + Live Track already does the live-map part and the phone has to be present either way; what + self-hosting buys is ownership, not capability. - **Mileage-milestone and maintenance pushes** straight to your phone — the reason the garage data is worth keeping. Strava's gear tracking can't do interval reminders at all. - **Grafana pointed straight at Postgres** — roughly an afternoon, the cheapest analytics in the plan. @@ -529,23 +613,114 @@ These are the payoff for self-hosting — things Strava structurally cannot do. Estimates assume one developer working evenings and weekends. -**Phase 0 — Scaffolding (2 weeks).** Monorepo, `uv`/`ruff`/`mypy --strict`, FastAPI skeleton with -`/healthz` and OpenAPI, Alembic baseline (users/invites/sessions **with RLS policies from the first -migration**), SvelteKit static SPA shell with login, manifest + service worker + precache passing -Lighthouse installability, VAPID keypair, Caddy, compose, Gitea Actions green, image in the registry, -deployed. -*Done when:* you log in at the real URL, add it to your iPhone home screen, it launches standalone — -and a push to `main` rebuilds and redeploys it. +**Phase 0 — Scaffolding.** ✅ **Done, with two deliberate deviations from this original description — +both recorded in `docs/DECISIONS.md`, not silent drift.** Monorepo, `uv`/`ruff`/`mypy --strict`, +FastAPI skeleton with `/healthz` and OpenAPI, Alembic baseline (users/invites/sessions), SvelteKit +static SPA shell with login, manifest + service worker, Caddy, Gitea Actions green, image in the +registry, deployed to a real Unraid host behind real HTTPS. -**Phase 1 — Zero-touch ride history (6–8 weeks).** Ingestion core (all three dedupe layers, course -discrimination, quarantine); **the Bryton cloud poller as the primary path**, polling every 15 min with -the full `integration_health` alerting stack; USB watcher for historical backfill and as the break-glass -path; manual upload; activity list/detail with MapLibre and stream charts; totals and trends by -week/month/year and per bike, in miles; bikes CRUD; **odometer milestone notifications** (they only need -activities, so they ship here); invites; nightly `pg_dump -Fc` + restic. -*Done when:* you finish a ride, tap Data Sync on the 650, put the bike away, and it's on your phone -within 15 minutes with **zero further interaction** — and you stop opening Strava to look at your own -data. **This phase fixes your original complaint; protect it from scope creep.** +- **SQLite, not Postgres+RLS.** D15 reversed D4 mid-Phase-0, after the RLS version was already + shipped and merged. Isolation is now enforced entirely at the repository layer (`db.py`'s + `Scope`), not database RLS. See D15 for the full cost/benefit record. +- **CI builds and pushes on every push to `main`, but does not auto-redeploy the running + container.** D16/D17 made this deliberate: the runner shares this Unraid host's own `dockerd` + (DooD), and an unattended redeploy of a container on a personal server with no human gate was + judged the wrong default. A push to `main` gets you a new image in the registry within minutes; + getting it onto the running container is still a manual step (`deploy/README.md`). An + auto-updater (Watchtower or similar) was attempted and deferred — see "Deliberately deferred" + in `docs/DECISIONS.md`. +- **VAPID/push notifications were never started.** Correctly so — per this doc's own PWA-decision + table, that's gated behind standalone-mode detection and belongs to a later phase, not Phase 0. + +*Done when — status:* Logging in at the real URL (`https://bike.bbergle.com`) and seeing an +authenticated view of your own account is **verified**, including the session actually persisting +(`scripts/smoke-test.sh`, added after a real Secure-cookie-over-HTTP bug on the first deploy — see +that script's header comment). **Not yet tried:** adding it to an iPhone home screen and confirming +a standalone launch — nobody has actually done this yet, so it isn't checked off, even though the +manifest and service worker are in place. + +**Phase 1 — One-tap ride history (6–8 weeks).** Ingestion core (all three dedupe layers, course +discrimination, quarantine, and the capture-everything field handling from the Schema section); **the +Bryton cloud poller as the primary path**, polling every 15 min with the full `integration_health` +alerting stack; USB watcher for historical backfill and as the break-glass path; manual upload; +activity list/detail with MapLibre and stream charts; totals and trends by week/month/year and per +bike, in miles; bikes CRUD; **odometer milestone notifications** (they only need activities, so they +ship here); invites; nightly SQLite snapshot + restic (not `pg_dump` — see D15). +*Done when:* you finish a ride, open the Active app once, put the bike away, and within 15 minutes +it's in your app — full-resolution original bytes, every field the head unit recorded, wear +recalculated — with **no further interaction**, and you stop opening Strava to look at your own data. + +> **Renamed from "Zero-touch" deliberately.** The original criterion said "tap Data Sync on the 650… +> zero further interaction," which the device cannot do (see "How rides actually reach the app"). +> The honest bar is **one tap in the Active app**, not zero. This phase therefore does *not* fully +> fix the original complaint — it fixes everything downstream of it. Removing that last tap needs +> either the iOS Shortcuts automation trick (free, unproven, try it) or reverse-engineering +> Bryton's BLE on non-iOS hardware (unbounded, out of scope). Don't let this phase quietly grow to +> chase it. **Protect it from scope creep.** + +**Phase 1A — Make it yours: the UI pass (open-ended, done together).** Phase 1 deliberately ships a +plain, functional UI — correctness of the *data* first, because a beautiful page over wrong numbers +is worse than an ugly page over right ones. This phase is the opposite: no new data, no new +pipeline, just making the thing feel like yours, working through it together rather than against a +spec written in advance. + +What it covers: the ride list and ride detail layout; which numbers are hero numbers and which are +buried; chart design for the stream data; the **verbose field surface** — everything the head unit +recorded, driven off `activity_field_inventory` so unknown and vendor-specific fields appear rather +than being silently hidden; dark mode; the mobile layout, since the real reading device is a phone +on a home screen; and the empty/loading/error states that a plain Phase 1 will have done crudely. + +*Why it's a phase and not a task:* UI taste isn't specifiable up front by either of us — it needs +real rides on a real screen and a few rounds of "no, bigger / not that / what if the map was the +whole page." Budgeting it as its own phase makes that iteration legitimate rather than scope creep +against Phase 1. +*Done when:* you'd rather open this than Strava to look at a ride you just did — and you can find +every field the Rider 650 recorded without asking where it went. + +**Phase 1B — Live tracking (3–5 weeks).** A self-hosted equivalent of Bryton's Live Track: someone +at home opens a link and watches your position and live metrics move on a map. + +**Read the constraints before designing anything here — they are hard, and they shape the feature:** +- **The phone must be in your pocket.** Bryton's own Live Track requires the Active app running and + relaying over BLE (Rider 650 manual, "LIVE TRACK"); the head unit has no independent uplink. No + self-hosted design changes that. +- **An installed iOS PWA cannot do this.** iOS suspends JS when backgrounded or screen-locked, so a + PWA can only track foreground with the screen on (Wake Lock, iOS 18.4+, helps but doesn't lift + the restriction). And WebKit implements **no Web Bluetooth at all**, so the PWA cannot read + HR/power/cadence sensors under any circumstances. +- Therefore: **the tracking client is not our PWA.** It is an existing, backgrounded app POSTing to + our API. + +**The shape that actually works:** **OwnTracks** (free, open-source, App Store) in HTTP mode, +POSTing to an authenticated endpoint on our server — genuinely backgrounded, screen off, phone in +pocket, ~30s–few-minute fixes in "move" mode. That gives **position + GPS speed**, and the server +derives the rest from the position stream: distance, elapsed and moving time, current/average pace, +elevation gain (via the Phase 3 DEM), and progress against the route if one is loaded. That is a +genuinely useful live metric set with no BLE at all. + +**What is *not* achievable backgrounded: heart rate, power, cadence.** Those need BLE, which means +either a screen-on phone mounted on the bars running a BLE-capable browser (a second-class, +opt-in mode — not the installed PWA), a companion Android device or LTE tracker in a jersey pocket, +or a native app and a $99/yr Apple Developer account. Ship the location-first version; treat sensor +metrics as a separate, explicitly optional follow-on, and don't let them block the useful 80%. + +**Two design rules this phase must not break:** +1. **Live positions are telemetry, not a ride.** They must never become an `activity` — the real + activity still arrives as original FIT bytes via Bryton (invariant #6, one ingestion path). A + live session is linked to the activity it corresponds to after the fact; it is not a second, + lower-fidelity source of truth. Getting this wrong produces duplicate, worse rides. +2. **"No privacy zones, ever" does not extend to a public live link.** That stance is sound for + *your own archive on your own server*; it is not sound for a URL that shows strangers your + current location, or your home, in real time. This phase needs: expiring share tokens, explicit + start/stop (plus an auto-end on inactivity so a forgotten session doesn't broadcast indefinitely), + and the option to blur the first and last N metres. + +*Done when:* your family can open a link while you're out, see where you are and how far you've +gone, and the link stops working when the ride does. + +> **On the phase numbering:** 1A and 1B are inserted rather than renumbering Phases 2–5, because +> "Phase 2"/"Phase 3" are referenced from `docs/DECISIONS.md`, `deploy/README.md` and code comments, +> and silently shifting their meaning is exactly the kind of stale cross-reference D20 is about. **Phase 2 — The garage (5–7 weeks).** Components and time-ranged installs; inventory with the stock ledger and install-from-stock; service events with photo/receipt attachments; **the seeded service-rule @@ -615,7 +790,9 @@ heatmap/segment recompute. for a private single-maintainer instance — but never make this repo public or add untrusted collaborators without disabling Actions on fork PRs. -**Backups are a systemd timer on the host, not a Gitea Action** — `pg_dump -Fc` + restic to B2/S3 with +**Backups are a systemd timer on the host, not a Gitea Action** — a consistent SQLite snapshot +(`sqlite3 .backup` or `VACUUM INTO`, **not** a raw file copy of a live database; `pg_dump` no longer +applies, see D15) + restic to B2/S3 with **append-only repo credentials** (so a compromised app host can't delete history), plus a restic snapshot of `blobstore/`. Backups must not depend on CI, because CI is the thing most likely to be broken when you need a restore. Quarterly `restore-drill.sh` restores into a throwaway stack and @@ -625,12 +802,19 @@ asserts activity counts match. ## Top risks -**1. The Bryton private API breaks silently.** Hardcoded API key, undocumented protocol, zero -stability guarantee — and the failure mode is *silence*. Mitigations: the poller is one -`ActivitySource` among several, and the USB path ships first in Phase 1 so the system is never -*dependent* on it; nightly canary; immediate alerting on `auth`/`protocol` errors; vendored protocol -pinned to an upstream SHA so fixes are a diff, not a re-derivation; dual-source dedupe on the FIT -natural key means you can fall back to USB mid-week and lose nothing and duplicate nothing. +**1. The Bryton chain breaks silently — and it is now a longer chain than originally planned.** With +the Wi-Fi premise gone, every automatically-ingested ride passes through +`head unit → BLE → Active app → Bryton cloud → poller`, and only the last link is ours. Two of those +links can fail quietly: the app not being opened (rides simply never leave the unit — invisible to +the server, which cannot distinguish "no rides uploaded" from "you didn't ride"), and Bryton's +private API itself (hardcoded key, undocumented protocol, zero stability guarantee). Mitigations: +the poller is one `ActivitySource` among several, and the USB path ships in Phase 1 so the system is +never *dependent* on the cloud; nightly canary; immediate alerting on `auth`/`protocol` errors; +vendored protocol pinned to an upstream SHA so fixes are a diff, not a re-derivation; dual-source +dedupe on the FIT natural key means you can fall back to USB mid-week and lose nothing and duplicate +nothing. **New mitigation the longer chain earns:** a "nothing ingested in N days" nudge, so the +silent failure mode of simply forgetting to open Active surfaces as a notification rather than as a +gap you notice months later. **2. Losing or corrupting years of ride history.** The realistic threats are mundane — a parser bug writes wrong elevation to 4,000 rides, a migration drops a column, a disk dies. The raw-bytes-are-truth @@ -650,12 +834,28 @@ because if the project stalls right after it, it has still succeeded. ## Verification -**Before coding:** on the Rider 650, `Main Menu → Data Sync` → join home Wi-Fi → ride → confirm the -activity reaches Bryton's cloud with the phone switched off. Then plug it in over USB and `ls -R` the -mounted volume to confirm the actual `.fit` path. +**Before coding — and this section is the reason the Wi-Fi premise survived as long as it did, so +treat it as load-bearing, not boilerplate.** The original version of this checklist said to verify +`Main Menu → Data Sync` on the Rider 650. That check was never run, and the feature does not exist; +a false premise sat at the top of this plan through an entire phase of work. Any capability of a +physical device that a phase depends on gets confirmed **on the device** before it is written down +as a fact. -**Phase 0:** `curl https://host/healthz` returns 200; push to `main` produces a new registry image and -a redeployed container; `docker compose logs` shows migrations applied. +Still worth doing before Phase 1: +- Plug the Rider 650 in over USB and `ls -R` the mounted volume to confirm the actual `.fit` path + (sources disagree: `Activities/`, `Actives/`, or root). +- Ride, open the Active app, and confirm the activity reaches Bryton's cloud — then confirm the + `intervalssync` protocol actually retrieves it, before building a poller on the assumption. +- Test the iOS Shortcuts automation (open Active on Rider 650 BLE disconnect, or on arriving home) + and see whether it fires reliably. This determines whether the one-tap step is permanent. + +**Phase 0 — done, verified for real, not just assumed from CI going green:** `curl https://host/healthz` +returns 200; a push to `main` produces a new registry image (`docs/DECISIONS.md` D17's release +workflow) — redeploying the running container from it is a manual step (D16/D17), not automatic; +`docker exec velodrome velodrome create-admin` bootstraps the first user (D18); logging in at the +real HTTPS URL and staying logged in on the next request is checked by `scripts/smoke-test.sh`, not +eyeballed in a browser, after that exact failure mode (a `Secure` cookie silently dropped when +tested over plain HTTP) actually happened on the first deploy. **Phase 1 — ingestion:** - Upload a real Rider 650 `.fit` → activity appears with correct distance, elevation, and map track. @@ -671,6 +871,20 @@ a redeployed container; `docker compose logs` shows migrations applied. user B, `SELECT * FROM activities`, expect zero of user A's rows. - Cross 1,000 miles on a bike → exactly one milestone notification, naming the ride that crossed it. +**Phase 1A — UI:** open a ride you actually did on your actual phone and check you can find every +field the head unit recorded without hunting; confirm a file containing an unknown or developer +field still surfaces it (feed a golden fixture with a deliberately nonstandard field and check it +renders rather than vanishing). + +**Phase 1B — live tracking:** +- Start a session, lock the phone, put it in a jersey pocket, ride — confirm positions keep arriving + with the screen off. This is the whole feature; if it only works screen-on, it has failed. +- Open the share link on a device that has never logged in → the map moves. +- Let the token expire (or end the ride) → the same link stops working. +- Confirm a live session **never** produces an `activity` row, and that once the real FIT file lands + via Bryton, the session reconciles to it rather than sitting alongside as a duplicate. +- Enable endpoint obfuscation, then check the public link genuinely does not reveal your house. + **Phase 2 — garage and notifications:** - Create a bike, install a chain, import 3 rides → chain shows summed distance. Edit the install date backwards → the number self-corrects with no manual recomputation. diff --git a/docs/RESEARCH.md b/docs/RESEARCH.md index c9475c8..56c04eb 100644 --- a/docs/RESEARCH.md +++ b/docs/RESEARCH.md @@ -8,20 +8,31 @@ architecture design; Sonnet for the two breadth surveys). Confidence is flagged ## 1. Getting data off the Bryton Rider 650 -### Device facts (verified 2026-09-20) +### Device facts (dated 2026-09-20 — item 2 was WRONG, see correction) -The Rider 650 is **modern generation**. Two independent extraction paths, both good: +> ⚠️ **Correction, 2026-09-22.** Item 2 below is false and was never verified on the device. The +> Rider 650 has **no Wi-Fi** — ANT+ and Bluetooth only — and no `Data Sync` menu entry. Its only +> sync route is BLE to the Bryton Active app. Confirmed on the physical unit, corroborated by +> BikeRadar's hands-on ("ANT+ and Bluetooth connectivity", syncing via "Bryton's Active App"). The +> likely origin is conflation with the Rider 750 / S800, which do have Wi-Fi. The header on this +> section originally read "verified 2026-09-20" — it was not verified; that word is the reason the +> claim propagated into `docs/PLAN.md` as a premise and survived an entire phase of work. See +> `docs/DECISIONS.md` D20. **Item 1 (USB) is still believed correct but is also unverified against +> the device — treat it as an open question, not a finding, until someone plugs the unit in.** + +The Rider 650 is **modern generation**. Extraction paths: 1. **USB mass storage.** Mounts as a plain FAT volume labelled `Bryton`; activities are native Garmin-format `.fit` files in `Bryton/Activities/`. No driver, no udev rule needed beyond convenience — it is plain `usb-storage`. Filter on `ID_FS_LABEL=Bryton`. *Caveat:* sources disagree across models about whether the folder is `Activities/`, `Actives/`, or the volume root. Discover it at runtime with a recursive glob; don't hardcode. -2. **On-device Wi-Fi.** `Main Menu -> Data Sync` lets the head unit join a Wi-Fi hotspot directly - and upload tracks to Bryton's cloud **with no phone and no Bryton Active app**. This is the - key finding — it removes the phone from the pipeline entirely. - *Still to verify:* whether it uploads automatically on joining Wi-Fi, or only when you - manually trigger Data Sync from the menu. Two-minute test; do it first. + *Status:* **unverified on the device.** +2. ~~**On-device Wi-Fi.** `Main Menu -> Data Sync` lets the head unit join a Wi-Fi hotspot directly + and upload tracks to Bryton's cloud **with no phone and no Bryton Active app**.~~ + **FALSE — the device has no Wi-Fi and no such menu.** See the correction above. The real path is + `head unit -> BLE -> Bryton Active app -> Bryton cloud`, so the phone cannot be removed from the + pipeline by any means available to this project. ### Bryton Active cloud API (reverse-engineered, working) @@ -330,8 +341,12 @@ can't delete history. 3-2-1. **Test restores** — an untested backup is a hypot ## 5. Open questions to resolve before/while building -1. **Does Rider 650 Data Sync upload automatically on joining Wi-Fi, or only on manual trigger?** - Determines how completely the phone is removed from the loop. Two-minute test. +1. ~~**Does Rider 650 Data Sync upload automatically on joining Wi-Fi, or only on manual trigger?**~~ + **RESOLVED 2026-09-22 — the question was malformed.** There is no Wi-Fi and no Data Sync on this + device; the phone cannot be removed from the loop at all. See the correction at the top of this + file and `docs/DECISIONS.md` D20. *This was the single most consequential open question in this + list, it was marked "two-minute test," and it went unanswered while an entire phase was planned + and built on the assumed answer.* 2. **Exact on-device `.fit` path** — documented as `Bryton/Activities/` for the 650, but sources disagree across models. One `ls -R` settles it. 3. **Bryton's FIT encoder quirks** — untested against `fitdecode`. Get one real 650 file and run it