OMNI1 Superapp · since Sep 1, 2026
Two-Door Deploys
How we ship the Superapp now, why it changed, and the few rules that keep the portal alive. Written for everyone who deploys — human or agent.
What happened on September 1
The customer portal answered 502 Bad Gateway for several hours, and three deploys in a row failed. The change being shipped was frontend-only — renaming a tab, removing a popup. It was never at fault. The pipeline was:
- Every deploy ran the same ritual regardless of the change: full database dump → checkout →
pip install→ resync → backend restart → frontend build. - The dump's runtime hung on one table:
scout_competitors_v2, 1447 MB of 1909 MB — raw scraping data, 76 % of the database. It blew past the GitHub Action's 40-minute limit. - Cancelling the Action does not kill the server-side process. The dump kept running and held a read lock on all tables.
- Backend startup runs ~50 implicit
ALTER TABLEs, which need exclusive locks — and waited with no timeout behind the dump. - Port 8000 was never bound → nginx answered 502. The next deploy started its own dump and queued into the same blockade. Found on the server: two stuck runs, one 1h21 old.
Four decisions that each sounded reasonable — backup before every deploy, schema check at startup, one script for everything, no cleanup on abort — only fail together. That's why nobody saw it coming.
The fix: two doors behind a router
A change pays only the price of its own risk.
Door A — Frontend no DB, ~5 min
- Runs when only
frontend/changed - Builds next to the live version (
frontend/builds/<sha>/), never in place - Atomic symlink switch + restart of the Next.js service
- Health check on page content, not just status codes
- Fails? Symlink flips back automatically. Last 3 builds kept.
- Never touches the database, runs a dump, or restarts the backend
Door B — Backend dump + restart, ≤40 min
- Runs when
tool/,deploy/or requirements changed - Pre-deploy dump — schema + all business tables; the scraping table is schema-only (24.7 MB instead of 556 MB). Hard 15-min timeout.
- pip + resync each capped at 10 min; a resync failure is red, not a warning
- Health check verifies the running git SHA and DB backend
- Fails? Previous code is restored automatically. The schema does not roll back — restoring the dump is a deliberate human decision.
The router (deploy.sh on the VPS) holds a lock so two deploys can never overlap, kills its own dump if a run is aborted, picks the door(s) by diffing what actually changed, and writes one audit line per run to /var/log/omni1-deploy.log — who deployed what, when, through which door, with what result.
How you deploy
From your own clone, with your work merged and pushed to origin/master:
bash ops/deploy.sh --frontend # Door A: frontend/ only
bash ops/deploy.sh --backend # Door B: tool/, deploy/, requirements
bash ops/deploy.sh # not sure? the router decides by diffThe script tags the commit, pushes the tag, and follows the GitHub run for that exact SHA until it finishes — then checks the live site. If your local HEAD isn't origin/master, it refuses: you'd be shipping someone else's state.
Never do this
Each rule exists because it went wrong once:
- Never patch code directly on the server. The next deploy overwrites it, and until then nobody knows what's actually running. (Aug 22, Aug 23)
- Never tag a commit that isn't pushed and merged. The launcher and router both check — don't work around them. (Aug 16)
- Never start a second deploy while one runs. The lock stops you; wait instead. (Sep 1)
- Never trust "the latest run". Follow the run for your commit SHA. (Aug 15)
- SSH
i/o timeoutin the Action?gh run rerunonce before debugging — it's usually a busy server, not your change. (Aug 15)
What's still on the list
Schema migrations still run implicitly at backend startup, which is why a schema change is one-way. The planned next step moves them into an explicit tool/migrate.py with a lock timeout — and moves scout_competitors_v2 out of the operational database entirely.