The RelOps Fleet Dashboard — one place to see everything, fix anything, and lose no more sleep over mystery workers.
Hangar pulls data from Taskcluster, SimpleMDM, Puppet, and Google Sheets and stitches it into a single live view of your entire test infrastructure. Pool health, hardware generations, task failures, quarantined machines, missing workers — all in one dark-themed dashboard. No more tab soup.
Currently tracking Mozilla's CI fleet. Built to grow with the rest of the infrastructure.
| Layer | Technology |
|---|---|
| Frontend | React 19 + TypeScript, Vite, Tailwind CSS, Recharts, TanStack Table |
| Backend | FastAPI (Python 3.11), SQLAlchemy 2, APScheduler |
| Database | PostgreSQL 16 |
| Infrastructure | GCP (Cloud Run, Cloud SQL, Cloud Load Balancing, IAP, Cloud Build) |
| Resource | Value |
|---|---|
| URL | https://hangar.relops.mozilla.com |
| GCP Project | relops-dashboard |
| Cloud Run | hangar (us-central1) |
| Auth | Google IAP — @mozilla.com accounts only, no GCP access required |
| CI/CD | Cloud Build triggers on push to main |
# 1. Configure environment
cp .env.example .env
# edit .env — see Environment Variables below
# 2. Start the database
docker compose up -d db
# 3. Build frontend
cd frontend && npm install && npm run build && cd ..
# 4. Start backend (serves built frontend at /)
docker compose up backendOpen http://localhost:8000.
cd frontend
npm install
npm run dev # http://localhost:5173 → proxies API to :8000cd backend
pip install -r requirements.txt
export DATABASE_URL=postgresql://relops:relops@localhost:5432/relops
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
# requires: docker compose up -d dbAll variables are optional unless marked required.
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
postgresql://relops:relops@localhost:5432/relops |
Required. PostgreSQL DSN |
SIMPLEMDM_API_KEY |
— | SimpleMDM REST API key |
TC_ROOT_URL |
https://firefox-ci-tc.services.mozilla.com |
Taskcluster root URL |
TC_CLIENT_ID |
— | TC client ID (public pools work unauthenticated) |
TC_ACCESS_TOKEN |
— | TC access token |
GOOGLE_SHEETS_ID |
— | Master inventory spreadsheet ID |
GOOGLE_EXPORT_SHEET_ID |
— | Export/dashboard spreadsheet ID |
GOOGLE_CREDENTIALS_JSON |
— | Path to service account JSON |
PUPPET_REPO_URL |
https://github.com/mozilla-platform-ops/ronin_puppet |
Puppet repo |
PUPPET_REPO_PATH |
/tmp/ronin_puppet |
Local clone location |
SYNC_INTERVAL_TC |
300 |
Taskcluster sync interval (seconds) |
SYNC_INTERVAL_SIMPLEMDM |
900 |
SimpleMDM sync interval |
SYNC_INTERVAL_SHEETS |
1800 |
Google Sheets sync interval |
SYNC_INTERVAL_PUPPET |
3600 |
Puppet sync interval |
TC_MISSING_THRESHOLD_HOURS |
24 |
Hours before raising a missing_from_tc alert |
┌─────────────────────────────────────────────────────┐
│ React SPA (Vite) │
│ Overview · Workers · Alerts · Pools · Tart VMs │
└──────────────────────┬──────────────────────────────┘
│ REST
┌──────────────────────▼──────────────────────────────┐
│ FastAPI backend │
│ /api/workers /api/fleet /api/alerts /api/tart-health│
│ │
│ APScheduler ─── sync/taskcluster.py (5 min) │
│ ├── sync/simplemdm.py (15 min) │
│ ├── sync/google_sheets.py(30 min) │
│ └── sync/puppet.py (60 min) │
└──────────────────────┬──────────────────────────────┘
│ SQLAlchemy
┌──────────────────────▼──────────────────────────────┐
│ PostgreSQL 16 │
│ workers · alerts · sync_logs · failure_events │
│ tart_slot_health (pushed by the on-network agent) │
└─────────────────────────────────────────────────────┘
User (mozilla.com Google account)
│
▼
Global Load Balancer (34.54.129.77)
│ Cloud Armor (OWASP rules + rate limiting)
│ Identity-Aware Proxy (IAP) — google auth gate
▼
Cloud Run — hangar (us-central1, min 1 instance)
│ INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER (direct URL blocked)
▼
Cloud SQL — Postgres 16 (private IP via VPC connector)
| Source | What it provides | Sync |
|---|---|---|
| Taskcluster | Worker state, quarantine status, last active, latest task | GraphQL + REST |
| SimpleMDM | MDM enrollment, OS version, serial number, custom attributes | REST (paginated) |
| Puppet | Worker role, pool assignment | Git clone of ronin_puppet |
| Google Sheets | Canonical state (production / loaner / defective / spare / staging), notes | Sheets API v4 |
Your morning briefing. Fleet-wide summary stats, sync health, top-10 failing machines and tests over the last 7 days, generation breakdown chart, workers-by-pool bar chart.
The full roster. Filterable and sortable across generation, state, pool, and MDM/TC status. Search by hostname or serial number. Up to 2,000 rows.
Everything Hangar knows about a single worker — Puppet role, sheet state, MDM enrollment, TC history. Edit notes and jump out to Taskcluster.
The stuff that needs your attention. Types: missing_from_tc, quarantined, mdm_unenrolled, pool_mismatch. Add notes, acknowledge, resolve.
Per-pool health scores, staleness breakdowns (active <24 h / 1–7 d / 7–30 d / 30 d+ / never seen), and job source distribution. Branch-override visibility (counts and which pools/branches are pinned).
Side-by-side hardware generation comparison — state breakdowns, inactive machines, and retirement candidates.
Per-slot health for the tart VM hosts behind gecko-t-osx-1500-m-vms (macOS → Tart VMs in the sidebar).
The unit is the slot, not the host: each host runs two VMs that fail independently, and Taskcluster's view — which is what Pool Health shows — can't see inside one. In July 2026 five of 26 slots were out of production for weeks because three guests were crash-rebooting every ~84 s while tart run on the host stayed up 11+ days, so every host-level signal read green.
Columns cover the worker id, TC registration, tart run uptime, whether credentials are host-injected or image-baked, and guest disk headroom; rows expand to the problem list plus VM state, clock skew, configured-vs-actual identity, puppet SHA and cert expiry.
Two behaviours worth knowing:
- Stale reads as
unknown, neverok. Rows older than 30 minutes are reported stale — a wedged collector must not leave the fleet looking green. - Cert expiry is judged against the cert's own lifetime, not a fixed number of days. These hosts hold 168 h certs, so "expires within 7 days" is the steady state; a warning means the renew daemon has actually missed its window.
Data arrives from an on-network agent, not from Hangar — see Tart VM slot health below.
The headline capability: wipe, re-enroll, and fully re-provision a Mac worker end-to-end from a button in the dashboard — no SSH, no console, no pasted secrets.
On a worker's detail page, authorized operators (IAP allowlist) get a Reprovision cockpit: live readiness (quarantined? idle? M4/EACS-supported?), an Execute button behind a confirm, and a live Apple-rainbow terminal timeline that streams every phase of the run.
How it reaches the datacenter safely. Hangar runs on Cloud Run and can't SSH into MDC1 — so it never does:
- Execute enqueues a
ReprovisionJob(one open job per host). - An on-network runner — Puppet-managed on an MDC1 mini — polls Hangar over mTLS (outbound only, no inbound to the datacenter), claims the job, and runs the
reprovisionCLI. - Each phase streams back as a job event and renders live: quarantine → drain → EACS wipe → DEP re-enroll → SecureToken mint → Bootstrap Token → self-provision (mTLS vault · puppet · Taskcluster).
The runner holds all SSH/admin creds; Hangar holds none. A server-side reaper closes any job whose runner vanishes, so a lost run never wedges a host.
Proven in prod: a Hangar-enqueued job reprovisioned
macmini-m4-80end-to-end, driven by the Puppet-managed runner onmacmini-m4-81, streaming the whole flow into the cockpit.
Same architectural constraint as Reprovision: Cloud Run can't reach MDC1, so Hangar never SSHes a tart host. An on-network agent does the collecting and pushes results in.
macmini-m4-81 (Puppet-managed) Hangar
com.mozilla.hangar-tart-health-agent
every 600s:
ssh admin@<13 tart hosts> ──collect──▶
POST /api/tart-health/agent/push ─────▶ require_runner (mTLS client cert)
↓
evaluate() → ok/warn/crit/unknown
↓
browser ──▶ GET /api/tart-health ─────────▶ require_access (IAP)
The agent is hangar-tart-health-agent in relops-bootstrap, the third daemon on the reprovision runner host, reusing its venv, env file and mTLS cert.
Severity is derived server-side, in evaluate(), not by the agent. The agent is a dumb collector, so thresholds can be retuned by deploying Hangar alone — nothing on-network changes — and the API and UI can't disagree about what crit means.
Only /api/tart-health/agent/* is routed to the non-IAP mTLS backend (terraform/lb_runner.tf). GET /api/tart-health deliberately stays on the IAP frontend.
require_runner does no per-path scoping, so that URL map is the only thing keeping the read rollup behind human auth. Routing a broad /api/tart-health/* to the runner backend as a convenience would expose fleet data to anything inside runner_source_cidrs holding a valid cert, with no Google identity attached.
Diagnostic, if a push ever stops landing: on the runner hostname a path that reaches the right backend returns 403 (Cloud Armor source deny from anywhere but MDC1). A 301 to the IAP domain means the path isn't in the allowlist at all.
tart_health_guests is currently false, so guest-level checks — disk headroom, clock skew, worker identity — are not collected and those columns are blank. Those are the checks that catch a crash-looping guest inside a healthy host, i.e. the July 2026 failure mode. Until it's enabled, treat a green Tart VMs page as "the hosts look fine", not "the fleet is fine".
workers — one row per hostname, columns from all four sources
alerts — active/resolved per-worker alerts
sync_logs — audit trail for each sync run (source, duration, records updated, errors)
failure_events — TC task failures indexed by hostname and task name
tart_slot_health — latest health row per tart VM slot (PK hostname+slot), pushed by the agent
Worker state precedence: sheet_state (if set) → inferred from TC/Puppet membership → unknown
Health score: fraction of production workers that are MDM-enrolled, not quarantined, and active within 24 hours.
GET /api/workers list + filter + search + sort
GET /api/workers/{hostname} full worker record
PATCH /api/workers/{hostname}/notes update dashboard notes
GET /api/fleet/summary dashboard stats
GET /api/fleet/pools per-pool health
GET /api/fleet/pending-counts TC pending tasks per pool
GET /api/fleet/pool-sources running task project breakdown
GET /api/fleet/failures?days=7 top failing machines + tests
GET /api/fleet/consolidation hardware generation analysis
GET /api/alerts list (filter: type, active_only)
PATCH /api/alerts/{id}/acknowledge
PATCH /api/alerts/{id}/resolve
GET /api/prs/ronin ronin_puppet PR queue + voting
POST /api/prs/ronin/{n}/upvote
POST /api/prs/ronin/{n}/downvote
GET /api/tart-health tart VM slot rollup, worst-first (IAP)
GET /api/tart-health/{hostname} one host's slots (IAP)
POST /api/tart-health/agent/push on-network agent push (mTLS runner cert)
POST /api/sync/run trigger manual sync
GET /api/health liveness check
hangar/
├── backend/
│ ├── app/
│ │ ├── main.py FastAPI app, lifespan, route registration
│ │ ├── config.py Pydantic Settings — all env vars
│ │ ├── database.py SQLAlchemy engine + table init
│ │ ├── models.py ORM models
│ │ ├── api/
│ │ │ ├── workers.py
│ │ │ ├── fleet.py
│ │ │ ├── alerts.py
│ │ │ ├── tart_health.py tart slot rollup + agent push, severity logic
│ │ │ └── prs.py ronin_puppet PR voting
│ │ └── sync/
│ │ ├── scheduler.py APScheduler job registration
│ │ ├── taskcluster.py GraphQL + REST sync, alert generation
│ │ ├── simplemdm.py Paginated REST sync + custom attributes
│ │ ├── puppet.py Git clone + inventory.d parse
│ │ └── google_sheets.py Sheets API v4 read
│ ├── Dockerfile
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── App.tsx React Router setup
│ │ ├── api.ts Typed API client + all TS interfaces
│ │ ├── components/
│ │ │ ├── Layout.tsx App shell — sidebar nav (collapses to hamburger drawer on mobile)
│ │ │ ├── CommandPalette.tsx ⌘K quick search
│ │ │ └── KeyboardShortcuts.tsx
│ │ └── pages/
│ │ ├── Overview.tsx
│ │ ├── Workers.tsx
│ │ ├── WorkerDetail.tsx
│ │ ├── Alerts.tsx
│ │ ├── Pools.tsx
│ │ ├── TartVMs.tsx per-slot tart VM health
│ │ └── Consolidation.tsx
│ ├── vite.config.ts Dev proxy → :8000
│ └── package.json
├── terraform/ All GCP infrastructure
│ ├── main.tf Provider, APIs, VPC, VPC connector
│ ├── run.tf Cloud Run service
│ ├── lb.tf Load balancer, IAP, Cloud Armor, SSL cert
│ ├── lb_runner.tf mTLS frontend — runner/agent path allowlist
│ ├── iam.tf Service accounts, IAM bindings, IAP access
│ ├── sql.tf Cloud SQL Postgres 16
│ ├── secrets.tf Secret Manager secrets
│ ├── variables.tf All configurable vars
│ └── terraform.tfvars Local var overrides (gitignored)
├── cloudbuild.yaml CI/CD — triggers on push to main
├── docker-compose.yml Local dev (postgres + backend)
└── .env.example Template for local dev
| Shortcut | Action |
|---|---|
⌘ K |
Open command palette (jump to any worker) |
⌘ / |
Open keyboard shortcuts help |
Esc |
Close any modal |