Scaling & High Availability

Eviworx ITSM is designed for horizontal scaling. Worker services can be distributed across multiple instances via docker compose --scale. Redis serves as the central coordination layer for locks, queues and PubSub.

📈

Features

✓ Job worker runs multiple instances (Redis locks)

✓ Workflow engine runs multiple instances (step locks)

✓ Report generator with automatic job distribution

✓ Email worker with parallel sending

✓ Notification worker with parallel delivery

✓ Traefik for load balancing and routing

✓ Backend as one instance, scaled vertically

Horizontal Scaling with Docker Compose

Worker services can be scaled to multiple instances using the --scale flag:

# Scale job-worker to 3 instances
docker compose up -d --scale job-worker=3

# Scale multiple services at once
docker compose up -d --scale job-worker=3 --scale email-worker=2 --scale report-generator=2

# Scale workflow-engine to 2 instances
docker compose up -d --scale workflow-engine=2

# Check status
docker compose ps

Note: When scaling, the container_name must be removed from docker-compose.yaml since container names must be unique. Docker will then assign unique names automatically.

Job-Worker Multi-Instance

How It Works:

• Each instance registers with a unique INSTANCE_ID
• Distributed locks via Redis prevent the same CronJob from being executed by multiple instances simultaneously
• Heartbeat every 15 seconds (stale instances are detected)
• executedBy field tracks which instance executed the job
Lock Mechanism:

1. Instance A tries to acquire lock: SET lock:cronjob:123 NX EX 300
2. If successful → job is executed
3. Instance B tries same lock → FAIL (already locked)
4. After completion → lock is released
Configuration:

# docker-compose.yaml (remove container_name!)
job-worker:
  # container_name: eviworx-job-worker  ← REMOVE
  environment:
    - INSTANCE_ID=  # Auto-generated per instance

Workflow-Engine Multi-Instance

How It Works:

• Step-level locks: each workflow step is atomically locked
• Redis PubSub for event distribution (workflow:step:complete)
• Circuit breaker prevents cascade failures
• Stuck recovery: workflows stuck > 10min are automatically recovered
Parallel Processing:

• Instance A processes Workflow 1, Step 3
• Instance B processes Workflow 2, Step 1
• Instance A processes Workflow 3, Step 5
→ No conflicts through step-level locking

Report-Generator Multi-Instance

How It Works:

• BullMQ distributes report jobs automatically to available instances
• Read-only DB access (helpdesk_readonly) - no write conflict possible
• Each instance can independently generate reports
• Ideal for high report load (many parallel CSV/PDF exports)
Scaling:
docker compose up -d --scale report-generator=3

Worker Heartbeat Monitoring

Heartbeat System:

• Each worker sends a heartbeat every 15 seconds
• Heartbeat includes: instance ID, timestamp, status, version
• Stale detection: the heartbeat entry expires after 30 seconds; without a new heartbeat the instance counts as offline
Health Endpoints:

Job-Worker:           http://localhost:3001/health/live
Workflow-Engine:      http://localhost:3003/health/live
Report-Generator:     http://localhost:3004/health/live
Email-Worker:         Port 3005 (HEALTH_PORT)
Notification-Worker:  Port 3006 (HEALTH_PORT)
AV-Worker:            Port 3007 (HEALTH_PORT)

Load Balancing via Traefik

Traefik as Reverse Proxy:

• Automatic service discovery via Docker labels
• TLS termination (SSL certificates)
• HTTP → HTTPS redirect
• Health-check-based routing (only healthy instances receive traffic)
• security_opt: no-new-privileges:true

In operation:

• No manual reload on configuration changes
• Automatic detection of new container instances
• Built-in health check integration
• Dashboard for monitoring (optional)

Backend: single instance

The load-heavy work – jobs, reports, email, notifications and workflows – runs in workers that can be started multiple times. The backend runs on one instance and is sized vertically. The reason is the rate limiters: they count requests in the backend process's memory. Two instances behind the load balancer would each keep their own counter, so the effective limit would be twice what was configured, and a login protection of "5 failures" would in fact allow 10. Therefore do not multiply the backend with --scale.

Where the backend keeps its state:

Area	Storage location
Sessions, permissions, caches	Redis, shared across instances
Rate limiters (HTTP)	Backend process memory; a restart resets them
API key limits	Redis

For the vast majority of installations this is not a bottleneck: the compute-heavy work sits with the workers, while the backend mainly serves API requests.

Redis as Coordination Layer

Redis Functions for Scaling:

1. Distributed Locks
   → CronJob execution (only one instance executes)
   → Workflow step locking (no duplicate processing)
2. BullMQ Queues
   → Email queue (fair distribution across instances)
   → Notification queue
   → Report queue
3. PubSub
   → workflow:step:complete events
   → Real-time event distribution to all instances
4. Cache
   → Session cache (shared across instances)
   → Query cache, SLA calendar cache
5. Audit-Fallback
   → On DB failure, audit events are stored in Redis (7 day TTL)
Important:
Redis is password-protected. All REDIS_URL connections use the format: redis://:PASSWORD@redis:6379

Resource Planning

Development Environment (Minimum)

Resource	Recommendation
CPU	4 Cores
RAM	8 GB
Storage	20 GB SSD
Instances	All services 1x each

Production Environment (Recommended)

Resource	Recommendation
CPU	8+ Cores
RAM	16-32 GB
Storage	200+ GB SSD (depends on upload volume)
Job-Worker	2-3 instances
Email-Worker	1-2 instances
Workflow-Engine	1-2 instances
Report-Generator	1-2 instances
Notification-Worker	1 instance

High-Load Environment (Enterprise)

Resource	Recommendation
CPU	16+ Cores
RAM	32-64 GB
Storage	500+ GB NVMe SSD
Job-Worker	3-5 instances
Email-Worker	2-3 instances
Workflow-Engine	2-3 instances
Report-Generator	2-4 instances
PostgreSQL	Dedicated server recommended
Redis	Dedicated server recommended

Scaling Checklist

Remove container_name from docker-compose.yaml (for scaled services)
REDIS_PASSWORD is set and identical everywhere
Redis has enough memory for queues and locks
PostgreSQL max_connections is adjusted (default: 100)
Health checks are enabled for all workers
Monitoring for worker heartbeats configured
Resource limits (memory/CPU) set per service
Traefik configured for load balancing

Next Steps

← Docker Compose

Container configuration and service details

Environment Variables →

All configuration variables in detail