Troubleshooting
Diagnose database, Redis, storage, image, auth, delivery, Worker, and URL failures.
Diagnose from stateful dependencies outward. Capture the trusted revision, last change time, Compose status, failed response, and redacted logs before restarting anything.
Access
You need host status and log access plus the minimum product access required for non-sensitive smoke tests.
docker compose -f docker-compose.prod.yaml ps -a
docker compose -f docker-compose.prod.yaml logs --tail=200 migrate minio-setup
docker compose -f docker-compose.prod.yaml logs --tail=200Do not publish resolved Compose configuration. Redact credentials, cookies, signed URLs, personal data, and content from incident records.
Diagnose in order
- Postgres: confirm its declared readiness check succeeds.
- Migration: confirm the one-shot exits 0. Core API, Delivery API, and Worker remain gated when it fails.
- Redis: confirm its authenticated health check succeeds. Delivery API does not wait for Redis and can report database health while cache or invalidation is broken.
- Object storage: confirm the service is reachable and the durable volume is present.
- Object setup: confirm the one-shot exits 0 and creates the expected public-download policy. Applications do not wait for it.
- Core API and Worker: inspect database, Redis, and public object-endpoint errors.
- Delivery API: inspect database startup, Redis response-cache, and invalidation errors.
- Admin app: check the HTTP route, sign-in flow, and Core/Delivery URLs baked into its image.
- Image service and proxy: test its internal object path, shared signing secret, public route, and one signed transform.
Probe the narrow database and HTTP routes, then reproduce the failed feature:
curl -fsS "$CORE_API_PUBLIC_URL/health"
curl -fsS "$DELIVERY_API_PUBLIC_URL/health"
curl -fsS "$ADMIN_APP_PUBLIC_URL/" > /dev/nullMatch the symptom
| Symptom | First checks |
|---|---|
| Backend consumers do not start | Postgres health, migration exit, percent-encoded DATABASE_URL |
| Upload fails | Object-setup exit, bucket and credentials, public S3 reachability from Core API and Worker |
| Worker processing stalls | Worker database, Redis, and public object-endpoint logs |
| Image transform fails | Image service’s internal object path, shared image secret, public image route |
| Sign-in loops | Admin/Core origins, exact Google callback, Workspace domain, cookie scope, stale Admin build |
| Delivery is unavailable | Delivery database startup, public route, authorization, then Redis logs |
| Delivery content is stale | Redis response cache and invalidation logs; a green /health proves only Postgres |
| Admin calls the wrong URL | Core and Delivery build inputs; rebuild Admin after correcting them |
| Public route fails | DNS, certificate, reverse-proxy target, and the documented container port |
For object failures, test both paths: Core API and Worker use S3_PUBLIC_URL, while the Image service uses the internal object endpoint. Browser access to the public object origin does not prove containers can hairpin to it.
After fixing the root cause, recreate only affected consumers and repeat the smallest original reproduction plus dependent operations. If the cause is still unknown, return the last coherent configuration or release change. Preserve pgdata, redis_data, and minio_data; never use volume deletion as troubleshooting.
Restart loops can hide the first useful failure. The project has not captured a reliable log-signature matrix for bad database, Redis, object, image, auth, proxy, DNS, migration, or volume failures.