diff --git a/docs/compose.md b/docs/compose.md new file mode 100644 index 00000000..9420d114 --- /dev/null +++ b/docs/compose.md @@ -0,0 +1,70 @@ +# Ridgeback Docker‑Compose Overview + +This document explains the `compose.yml` configuration used to run Ridgeback. It covers each service, its purpose, key environment variables, and how the components interact. + +## Service Overview + +| Service | Purpose | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| `ridgeback_create_volumes` | Create host directories (postgres, logs, celery, rabbitmq, server, logrotate) with the correct UID/GID. | +| `ridgeback_postgres` | PostgreSQL database instance for Ridgeback. | +| `ridgeback_memcached` | Memcached cache server. | +| `ridgeback_rabbitmq` | RabbitMQ message broker with management UI. | +| `ridgeback` | Django application that hosts the Ridgeback web interface and API. | +| `ridgeback_celery_beat` | Celery beat scheduler that triggers periodic tasks. | +| `ridgeback_celery_*_queue` | Various Celery workers that process different task queues (command, action, status, submit‑job, set‑permission, cleanup). | +| `ridgeback_logrotate` | Periodically rotates application logs. | +| `ridgeback_db_backup` | Schedules regular database backups. | + +--- + +### 1. Database & Cache Services + +| Service | Image | Key Environment Variables | Notes | +| --------------------- | ---------------------------------- | --------------------------------------------------- | -------------------------------------------------------- | +| `ridgeback_postgres` | `postgres:17` | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | Exposes port `${RIDGEBACK_DB_PORT}` | +| `ridgeback_memcached` | `mskcc/memcached-nc:1.6.39` | `RIDGEBACK_MEMCACHED_PORT` | Exposes port `${RIDGEBACK_MEMCACHED_PORT}` | +| `ridgeback_rabbitmq` | `rabbitmq:4.0.6-management-alpine` | `RABBITMQ_DEFAULT_USER`, `RABBITMQ_DEFAULT_PASS` | Management UI on `${RIDGEBACK_RABBITMQ_MANAGEMENT_PORT}` | + +All three depend on the volume‑creation service. + +### 2. Celery Workers & Beat + +All workers share the anchor `x-ridgeback_celery` for common settings (image, user, network, env_file, volumes). Each worker overrides the `command` to start its specific queue. + +| Worker | Queue | Concurrency | Command | +| --------------------------------------- | ------------------------------------ | ----------- | -------------------------------------- | +| `ridgeback_celery_beat` | N/A (beat) | 1 | Starts Celery beat with schedule file. | +| `ridgeback_celery_command_queue` | `${RIDGEBACK_COMMAND_QUEUE}` | 30 | Worker for command queue. | +| `ridgeback_celery_action_queue` | `${RIDGEBACK_ACTION_QUEUE}` | 10 | Worker for action queue. | +| `ridgeback_celery_check_status_queue` | `${RIDGEBACK_CHECK_STATUS_QUEUE}` | 10 | Worker for status checks. | +| `ridgeback_celery_submit_job_queue` | `${RIDGEBACK_SUBMIT_JOB_QUEUE}` | 5 | Worker for job submission. | +| `ridgeback_celery_set_permission_queue` | `${RIDGEBACK_SET_PERMISSIONS_QUEUE}` | 10 | Worker for permission setting. | +| `ridgeback_celery_cleanup_queue` | `${RIDGEBACK_CLEANUP_QUEUE}` | 2 | Worker for cleanup tasks. | + +All depend on PostgreSQL, Memcached, RabbitMQ, and `ridgeback_celery_beat` for health. + +### 5. Auxiliary Services + +| Service | Image | Purpose | +| --------------------- | ----------------------------------- | ------------------------------------------- | +| `ridgeback_logrotate` | `mskcc/voyager-compose-utils:1.0.0` | Rotates logs weekly | +| `ridgeback_db_backup` | Same image | Schedules database backups using `pg_dump`. | + +Both depend on the beat and command‑queue workers. + +### 6. Network + +- **`voyager_net`** – Bridge network shared by all services and also [Beagle](https://github.com/mskcc/beagle) + +## Key Environment Variables + +| Variable | Description | +| -------------------------------------------------------------------------- | ----------------------------------------------- | +| `RIDGEBACK_VERSION` | Docker image tag for Ridgeback. | +| `DOCKER_UID`, `DOCKER_GID` | UID/GID for container processes. | +| `RIDGEBACK_DB_USERNAME`, `_PASSWORD`, `_NAME` | PostgreSQL credentials. | +| `RIDGEBACK_RABBITMQ_USERNAME`, `_PASSWORD` | RabbitMQ credentials. | +| `CLUSTER_FILESYSTEM_MOUNT`, `CLUSTER_SCRATCH_MOUNT`, `CLUSTER_ADMIN_MOUNT` | Bind mounts for cluster file system access. | +| `CLUSTER_CODE_PATH` | Path for the ridgeback code base on the cluster | +| `LOGROTATE_*`, `DB_BACKUP_*` | Log rotation and backup scheduling options. |