refactor(api): move from Postgres+RLS to single-engine SQLite
Reverses a shipped, tested, merged decision (D4/PR #2) rather than building on it — see docs/DECISIONS.md D15 for the full record: what was rejected (Postgres as a second container; Postgres+PostGIS bundled inside the single container via a supervisor), what this costs (no database-level RLS, no PostGIS, procrastinate needs replacing — all stated as a concern before this was decided, and reaffirmed anyway, which is the user's call to make about their own instance). The one invariant-critical consequence: isolation between users now rests entirely on the repository-layer scope (db.py's `Scope.select()`), not two layers. CLAUDE.md's invariant #4 is revised accordingly. This is not a downgrade-and-hope — `Scope` is built so an unfiltered query against a user-owned table is structurally harder to write than a scoped one (there is no method on `Scope` that returns one), and tests/test_auth.py::test_scoped_session_blocks_cross_user_reads replaces the old RLS proof with the same empirical standard: it doesn't trust the query builder filters correctly because the code reads correctly, it registers two real users and checks. test_unscoped_session_can_see_every_user_when_misused is the deliberately alarming companion — it demonstrates exactly what a reviewer must now catch, since nothing else will. Six real, non-obvious SQLite behaviours found and fixed by actually running this against a real file, not assumed from docs: - Foreign keys, ON DELETE CASCADE included, are OFF by default per connection — deleting a user silently left orphaned sessions/api_tokens, no error either way. Fixed with PRAGMA foreign_keys=ON on every connect. - Transactions default to DEFERRED, which only takes a write lock on the first actual write — a real check-then-act race for invite redemption (two concurrent redemptions could both read used_count < max_uses as true before either commits). Fixed by disabling the driver's implicit BEGIN and issuing BEGIN IMMEDIATE ourselves — SQLAlchemy's own documented recipe for this, not improvised. - DateTime(timezone=True) does NOT round-trip tzinfo on SQLite — a tz-aware datetime goes in, a naive one comes back out, and every `expires_at < datetime.now(UTC)` comparison in auth/service.py then raises TypeError. Fixed once at the Base level with a UTCDateTime TypeDecorator rather than per-column. - Uuid(as_uuid=True) stores as 32-char hex with NO hyphens on SQLite, not str(uuid)'s hyphenated form. A test fixture that raw-inserted the hyphenated form left rows the ORM's own later UPDATE (via invite.used_count += 1's autoflush) could never match by primary key, updating zero rows and raising StaleDataError. Fixed by using .hex to match exactly what the ORM itself writes. - BEGIN IMMEDIATE applies to every transaction, reads included — a long-lived test fixture that autobegins a transaction via a bare read and never explicitly closes it holds SQLite's exclusive write lock for the rest of the test, and a later scoped_session() call fails with "database is locked". Not an app-code bug (every real session block closes cleanly on exit), but real enough to document since the next person writing a test against the db_auth fixture will hit it too. - Python's sqlite3 module deprecates its own implicit datetime adapter as of 3.12 — silent today, warns on every raw-SQL datetime bind. Only ever hit test fixture code (the ORM path never uses it, confirmed by running the ORM-only health test with warnings promoted to errors and it stayed clean); fixed there with an explicit .isoformat() rather than left for a future Python version to turn into a real failure. Also, since with_for_update() silently no-ops on SQLite (confirmed — SQLAlchemy emits no SQL for it, no error either) rather than actually locking anything: removed it from register()'s invite-redemption query and corrected the comment to attribute the concurrency guarantee to BEGIN IMMEDIATE, where it now actually lives. One PR, not several, for the same reason PR #2 was: the migration, the models, db.py, and the docs recording why are five views of one decision — splitting them wouldn't make review easier, just disconnected. 552 insertions / 548 deletions across 17 files, most of it necessarily touching what PR #2 shipped rather than net-new code. Deliberately deferred, not solved here: PostGIS's replacement for spatial storage, procrastinate's replacement for background jobs, and the EXCLUDE USING gist constraint's replacement for component_installs — none of those tables exist yet (Phase 1-2), so none of it is broken, and docs/DECISIONS.md D15 records exactly what each future phase needs to decide before it can be built. .gitea/workflows/deploy pipeline (PR #4, built for the old 3-container Postgres compose stack) was closed as superseded rather than merged; the single-container image build is follow-up work, not part of this change. Verified: ruff check, ruff format --check, and mypy --strict all clean. 13/13 pytest passing against a real SQLite file, including with DeprecationWarning promoted to an error (confirms the sqlite3 adapter deprecation fix actually holds, not just that it's quiet by default). Full alembic upgrade -> downgrade -1 -> upgrade cycle run clean. alembic check clean with no include_object filter needed at all now (SQLite starts with nothing but what our own migrations create — no PostGIS/TIGER noise to filter out in the first place). CI's exact migration command sequence reproduced locally end to end before touching the workflow file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+33
-15
@@ -61,8 +61,8 @@ can pull rides off the head unit. Explicitly out of scope.
|
||||
| Layer | Choice | Why |
|
||||
|---|---|---|
|
||||
| API | Python 3.12, FastAPI, Pydantic v2, SQLAlchemy 2.0 async, Alembic | `fitdecode` is Python-only, so Python is forced; FastAPI emits OpenAPI 3.1 free |
|
||||
| DB | PostgreSQL 16 + PostGIS 3.4 (`postgis/postgis:16-3.4`) | Needed for heatmaps, bbox queries, self-segment matching |
|
||||
| Jobs | **procrastinate** (Postgres-backed queue) | Transactional enqueue; no Redis; built-in cron and retries |
|
||||
| DB | **SQLite** (single file, inside the app container) | Chosen over Postgres+PostGIS specifically to keep the whole deploy to one container — see `docs/DECISIONS.md` D15 for the full reasoning and what it costs (no database-level RLS, no PostGIS, `procrastinate` needs replacing) |
|
||||
| Jobs | **TBD before Phase 1** — `procrastinate` no longer fits (Postgres-only) | Nothing enqueues a job yet; pick this when the ingestion pipeline actually needs it, not before |
|
||||
| Frontend | SvelteKit `adapter-static` **SPA + PWA** — no Node process in prod | Static files served by Caddy; enforces API-first by construction |
|
||||
| Maps | MapLibre GL JS from day one | Renders raster tiles now, self-hosted PMTiles vector later — a config change, not a rewrite |
|
||||
| FIT parsing | `fitdecode` | Thread-safe, preserves header+CRC, correct developer-field handling; `python-fitparse`'s own maintainers point here |
|
||||
@@ -106,27 +106,45 @@ code-generation exercise against a stable contract rather than a rewrite.
|
||||
|
||||
## Service topology
|
||||
|
||||
**v1 — 4 containers, under 2GB RAM total:**
|
||||
- `caddy` — serves the static SPA, proxies `/api/*`. Same-origin, so no CORS. Your existing reverse proxy terminates TLS in front.
|
||||
- `api` — uvicorn/FastAPI. Runs Alembic on entrypoint.
|
||||
- `worker` — same image, `procrastinate worker`. Owns periodic tasks too, so no separate scheduler.
|
||||
- `db` — postgis/postgis:16-3.4.
|
||||
**Revised by D15 — single container**, not the multi-container compose topology this section
|
||||
originally described. Caddy and the FastAPI app run together in one image via a lightweight
|
||||
process supervisor; SQLite is a file inside the same container's persistent volume, not a
|
||||
separate service. See `docs/DECISIONS.md` D15 for why, and `deploy/`'s own README for the actual
|
||||
supervisor config once it exists.
|
||||
|
||||
Volumes: `pgdata`, `blobstore` (content-addressed raw FIT + attachments), `import_inbox` (USB watch folder bind-mount).
|
||||
Volumes: one persistent volume holding the SQLite file, `blobstore` (content-addressed raw FIT +
|
||||
attachments), and `import_inbox` (USB watch folder bind-mount).
|
||||
|
||||
**Phase 2 adds zero containers** (poller is a periodic task; Open-Meteo is a public API).
|
||||
**Phase 3 adds two:** `tileserver` (tileserver-gl-light + regional PMTiles, ~200MB–1GB) and `topodata` (Open Topo Data + region-clipped SRTM).
|
||||
**Phase 4+ behind opt-in compose profiles:** `ollama`, `grafana`, and optionally `valhalla`/`photon`/`overpass`.
|
||||
**Later phases that would have been "add a container" under the old topology** — `tileserver`,
|
||||
`topodata`, `ollama`, `grafana`, `valhalla`/`photon`/`overpass` — still make sense as genuinely
|
||||
separate containers even under a single-container-for-the-app model (they're independent services
|
||||
with their own resource profiles, not part of "the app"). Whether the app container talks to them
|
||||
over a shared Docker network or they stay fully optional add-ons is a decision for whichever phase
|
||||
actually needs the first one — not resolved here speculatively.
|
||||
|
||||
**Explicitly NOT on day one:** Valhalla, Photon, Overpass, Nominatim (>1TB/128GB RAM — never), Ollama, Redis, MinIO, Grafana, a separate scheduler.
|
||||
**Explicitly NOT on day one:** Valhalla, Photon, Overpass, Nominatim (>1TB/128GB RAM — never), Ollama, Redis, MinIO, Grafana.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
|
||||
Conventions: UUIDv7 PKs (time-ordered, URL-safe, client-generatable). All `timestamptz` UTC. **All
|
||||
physical quantities as SI integers** — distance in metres, time in seconds, speed in mm/s, altitude
|
||||
in cm, money in minor units. Units are a presentation concern.
|
||||
**Note on what's below vs. what's actually built:** the Identity tables (§ Tables > Identity) are
|
||||
real, built, and running on SQLite — `apps/api/velodrome/models/identity.py` and
|
||||
`apps/api/alembic/versions/0001_baseline.py` are the source of truth for those, not this
|
||||
prose. Everything past Identity (activities, streams, bikes/components, service rules,
|
||||
notifications, weather) was designed against Postgres/PostGIS conventions — `geometry(...)`
|
||||
columns, `ARRAY`, `INET`, GIST indexes, RLS policies — before D15 moved the database to SQLite.
|
||||
None of it is built yet, so none of it is broken; it just needs a real pass for SQLite
|
||||
compatibility (TEXT/BLOB for geometry, JSON-encoded TEXT for arrays, plain TEXT for IP addresses,
|
||||
application-layer exclusion checks instead of `EXCLUDE USING gist`) when each phase actually
|
||||
builds it, informed by whatever's learned finishing the SQLite migration on Identity first —
|
||||
not a mechanical find-and-replace on speculative schema now.
|
||||
|
||||
Conventions: UUIDv7 PKs (time-ordered, URL-safe, client-generatable). All timestamps UTC (SQLite
|
||||
has no native timezone-aware timestamp type — see `apps/api/velodrome/models/base.py` for how
|
||||
Identity stores them; the same convention applies going forward). **All physical quantities as SI
|
||||
integers** — distance in metres, time in seconds, speed in mm/s, altitude in cm, money in minor
|
||||
units. Units are a presentation concern.
|
||||
|
||||
### Two architectural rules that everything else depends on
|
||||
|
||||
|
||||
Reference in New Issue
Block a user