Distributed Task Queues#
Source: particle1331/distributed-task-queue
Introduction#
Prediction systems (e.g. offline or batch) can be modeled as workers picking up on a task queue. In this notebook, we will look at an implementation of such a system. The workers will be distributed in the sense that each worker runs in its own container. We will use the open-source framework Celery:
Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.
An overview of the system is depicted in Fig. 76:
Fig. 76 Four components of the distributed task system. A task server produces standard tasks which are consumed by worker instances. A fault-tolerant message broker distributes the tasks reliably to the workers. The results backend is used for storing results which can be queried asynchronously.#
In particular, we will consider the scenario where we have (1) instances running long-running tasks on two processes such that (2) the instance can terminate any time (e.g. with autoscaling). Any time-consuming or resource-intensive computation that can be offloaded from the main application as background task such as ML model training and batch inference can be orchestrated using this approach.
Celery application#
Any machine with a copy of the Celery application code and a connection to the broker is automatically a part of the distributed system. This is demonstrated below by connecting our local machine to the broker and pushing tasks to the queue. So first we have to install the app library particle1331/distributed-task-queue:
!rm -rf ./task-queue/distributed-task-queue
!git clone --single-branch --branch main https://github.com/particle1331/distributed-task-queue.git ./task-queue/distributed-task-queue > /dev/null
!pip install -U git+https://github.com/particle1331/distributed-task-queue.git@main --force-reinstall > /dev/null
!echo "\nSuccessfully installed: $(pip list | grep distributed-task-queue)/$(wget --header='Accept: application/vnd.github.VERSION.sha' -qO- commit_id http://api.github.com/repos/particle1331/distributed-task-queue/commits/main | head -c 7)"
Cloning into './task-queue/distributed-task-queue'...
remote: Enumerating objects: 222, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 222 (delta 85), reused 60 (delta 60), pack-reused 114
Receiving objects: 100% (222/222), 1.05 MiB | 1.99 MiB/s, done.
Resolving deltas: 100% (125/125), done.
Running command git clone --filter=blob:none --quiet https://github.com/particle1331/distributed-task-queue.git /private/var/folders/jq/9vsvd9252_349lsng_5gc_jw0000gn/T/pip-req-build-5c0aoav6
Successfully installed: distributed-task-queue 0.0.0/60efa9b
The code is also copied for building the app locally later:
import os; os.chdir("./task-queue/")
!tree ./distributed-task-queue
./distributed-task-queue
├── Dockerfile.api
├── Dockerfile.worker
├── LICENSE
├── Makefile
├── README.md
├── api
│ ├── __init__.py
│ ├── main.py
│ ├── models.py
│ ├── requirements.txt
│ ├── results.py
│ ├── tasks.py
│ └── utils.py
├── architecture.png
├── docker-compose.yml
├── dtq
│ ├── __init__.py
│ ├── app.py
│ ├── requirements.txt
│ └── tasks.py
├── pyproject.toml
└── setup.cfg
3 directories, 20 files
Finally, we configure the connection:
import os
os.environ["REDIS_HOST"] = "localhost"
os.environ["REDIS_PORT"] = "6379"
os.environ["RABBITMQ_HOST"] = "localhost"
os.environ["RABBITMQ_PORT"] = "5672"
Remark. One caveat is that task producers and deployed workers can have different versions of the code. This can be fixed by containerization and orchestrating rebuilds.
Services#
In this section, we will send tasks to the workers directly using our copy of the dtq code. This contains the configured Celery app which at this point is already connected to the broker. See diagram above (Fig. 76) with the Tasks API replaced by a terminal. The ensemble of services will be built and run using Docker compose:
# ./docker-compose.yml
version: "3"
services:
api:
build:
dockerfile: Dockerfile.api
ports:
- "8000:8000"
depends_on:
- redis
- rabbitmq
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672
command: ["--port", "8000", "--host", "0.0.0.0"]
worker:
build:
dockerfile: Dockerfile.worker
depends_on:
- redis
- rabbitmq
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672
deploy:
replicas: 2
command: ["-A", "dtq.app", "worker", "--concurrency", "2", "-l", "INFO"]
redis:
image: redis:latest
ports:
- "6379:6379"
rabbitmq:
image: rabbitmq:3.12.0-management
ports:
- "5672:5672"
- "15672:15672"
flower:
image: mher/flower
ports:
- "5555:5555"
command:
- "celery"
- "--broker=amqp://guest@rabbitmq:5672//"
- "flower"
- "--broker_api=http://guest:guest@rabbitmq:15672/api//"
depends_on:
- rabbitmq
Running the stack in the background:
!docker compose -f ./distributed-task-queue/docker-compose.yml up -d --build
Show code cell output
?25l[+] Building 0.0s (0/0) docker:desktop-linux
?25h?25l[+] Building 0.0s (0/0) docker:desktop-linux
?25h?25l[+] Building 0.0s (0/2) docker:desktop-linux
?25h?25l[+] Building 0.2s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.1s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.3s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.3s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.5s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.4s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.6s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.6s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.8s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.7s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.9s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.9s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.1s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.0s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.2s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.2s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.4s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.3s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.5s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.5s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.7s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.6s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 1.8s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.8s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.0s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 1.9s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.1s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.1s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.3s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.2s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.4s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.4s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.6s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.5s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.7s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.7s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 2.9s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 2.8s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.0s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.0s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.2s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.1s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.3s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.3s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.5s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.4s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.6s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.6s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.8s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 3.7s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 3.9s (14/20) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [api internal] load metadata for docker.io/library/python:3.9.15-slim 3.8s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [worker internal] load build context 0.0s
=> => transferring context: 1.21kB 0.0s
=> [api 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:ffc6cb648d 0.0s
=> [api internal] load build context 0.0s
=> => transferring context: 7.08kB 0.0s
=> CACHED [worker 2/7] WORKDIR /opt 0.0s
=> CACHED [worker 3/7] COPY ./dtq/requirements.txt . 0.0s
=> CACHED [worker 4/7] RUN pip install -r requirements.txt 0.0s
=> CACHED [worker 5/7] RUN pip install setproctitle 0.0s
=> CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps 0.0s
=> CACHED [worker 7/7] COPY /dtq/ ./dtq/ 0.0s
?25h?25l[+] Building 4.0s (22/22) docker:desktop-linux
=> [api internal] load build context 0.0s
=> => transferring context: 7.08kB 0.0s
=> CACHED [api 2/7] WORKDIR /opt 0.0s
=> CACHED [worker 3/7] COPY ./dtq/requirements.txt . 0.0s
=> CACHED [worker 4/7] RUN pip install -r requirements.txt 0.0s
=> CACHED [worker 5/7] RUN pip install setproctitle 0.0s
=> CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps 0.0s
=> CACHED [worker 7/7] COPY /dtq/ ./dtq/ 0.0s
=> [worker] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:15aacea1a3c26b4263b28bf091223528cc5e3a4741206 0.0s
=> => naming to docker.io/library/distributed-task-queue-worker 0.0s
=> CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt 0.0s
=> CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt 0.0s
=> CACHED [api 5/8] RUN pip install -r dtq-requirements.txt 0.0s
=> CACHED [api 6/8] RUN pip install -r api-requirements.txt 0.0s
=> CACHED [api 7/8] COPY ./dtq ./dtq 0.0s
=> CACHED [api 8/8] COPY ./api . 0.0s
=> [api] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:233ecd18c4b7096d01414d847d731453fa972188b0bc2 0.0s
=> => naming to docker.io/library/distributed-task-queue-api 0.0s
?25h?25l[+] Building 4.0s (22/22) FINISHED docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [api internal] load metadata for docker.io/library/python:3.9.15-slim 3.8s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [worker internal] load build context 0.0s
=> => transferring context: 1.21kB 0.0s
=> [api 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:ffc6cb648d 0.0s
=> [api internal] load build context 0.0s
=> => transferring context: 7.08kB 0.0s
=> CACHED [api 2/7] WORKDIR /opt 0.0s
=> CACHED [worker 3/7] COPY ./dtq/requirements.txt . 0.0s
=> CACHED [worker 4/7] RUN pip install -r requirements.txt 0.0s
=> CACHED [worker 5/7] RUN pip install setproctitle 0.0s
=> CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps 0.0s
=> CACHED [worker 7/7] COPY /dtq/ ./dtq/ 0.0s
=> [worker] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:15aacea1a3c26b4263b28bf091223528cc5e3a4741206 0.0s
=> => naming to docker.io/library/distributed-task-queue-worker 0.0s
=> CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt 0.0s
=> CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt 0.0s
=> CACHED [api 5/8] RUN pip install -r dtq-requirements.txt 0.0s
=> CACHED [api 6/8] RUN pip install -r api-requirements.txt 0.0s
=> CACHED [api 7/8] COPY ./dtq ./dtq 0.0s
=> CACHED [api 8/8] COPY ./api . 0.0s
=> [api] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:233ecd18c4b7096d01414d847d731453fa972188b0bc2 0.0s
=> => naming to docker.io/library/distributed-task-queue-api 0.0s
?25h?25l[+] Running 0/0
⠋ Network distributed-task-queue_default Creating 0.0s
?25h?25l[+] Running 1/1
✔ Network distributed-task-queue_default Created 0.1s
⠋ Container distributed-task-queue-rabbitmq-1 Creating 0.1s
⠋ Container distributed-task-queue-redis-1 Creating 0.1s
?25h?25l[+] Running 3/3
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
⠋ Container distributed-task-queue-flower-1 Creating 0.1s
⠋ Container distributed-task-queue-worker-2 Creating 0.1s
⠋ Container distributed-task-queue-worker-1 Creating 0.1s
⠋ Container distributed-task-queue-api-1 Creating 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
?25h?25l[+] Running 7/7
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
?25h
!docker compose -f ./distributed-task-queue/docker-compose.yml ps | sort
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
distributed-task-queue-api-1 distributed-task-queue-api "uvicorn main:app --…" api 3 seconds ago Up Less than a second 0.0.0.0:8000->8000/tcp
distributed-task-queue-flower-1 mher/flower "celery --broker=amq…" flower 3 seconds ago Up Less than a second 0.0.0.0:5555->5555/tcp
distributed-task-queue-rabbitmq-1 rabbitmq:3.12.0-management "docker-entrypoint.s…" rabbitmq 3 seconds ago Up 1 second 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp
distributed-task-queue-redis-1 redis:latest "docker-entrypoint.s…" redis 3 seconds ago Up 1 second 0.0.0.0:6379->6379/tcp
distributed-task-queue-worker-1 distributed-task-queue-worker "celery -A dtq.app w…" worker 3 seconds ago Up Less than a second
distributed-task-queue-worker-2 distributed-task-queue-worker "celery -A dtq.app w…" worker 3 seconds ago Up Less than a second
Before proceeding we check whether all system components are available:
import time
from redis import StrictRedis
from dtq.app import app
def healthcheck(app):
"""Ping and wait up to one second for responses."""
r = StrictRedis(
host=os.environ["REDIS_HOST"],
port=os.environ["REDIS_PORT"],
socket_timeout=3
)
workers = app.control.inspect().ping() # True if at least one response
return bool(workers) and r.ping()
MAX_RETRIES = 5
for i in range(MAX_RETRIES):
try:
if healthcheck(app):
print("Celery app connection is established.")
time.sleep(1.0)
break
except Exception as e:
if i == MAX_RETRIES - 1:
raise Exception("Celery results backend or workers unavailable.")
Celery app connection is established.
Workers#
Building a worker simply involves providing a copy of the Celery application code:
# ./Dockerfile.worker
FROM python:3.9.15-slim
WORKDIR /opt
COPY ./dtq/requirements.txt .
RUN pip install -r requirements.txt
RUN pip install setproctitle
RUN apt-get update && apt-get install -y procps
COPY /dtq/ ./dtq/
ENTRYPOINT ["celery"]
From the compose file, two replicas of the worker service is built. Each worker runs the ff. command at startup:
celery -A dtq.app worker --concurrency 2 -l INFO
This runs two worker processes for each worker with log level INFO. It is recommended to set concurrency equal to the CPU count of the worker machine. Here we set to 2 for simplicity. Distinguishing between the worker and its processes is important when reading the documentation.
Installing setproctitle allows us to see the type of process in ps listings. The worker runs two child processes and a main worker process:
!docker exec distributed-task-queue-worker-1 ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 8.9 0.9 45876 38032 ? Ss 20:56 0:01 [celeryd: celery@364128f8e977:MainProcess] -active- (-A dtq.app worker --concurrency 2 -l INFO)
root 8 0.0 0.8 43484 32996 ? S 20:56 0:00 [celeryd: celery@364128f8e977:ForkPoolWorker-1]
root 9 0.0 0.8 43488 32868 ? S 20:56 0:00 [celeryd: celery@364128f8e977:ForkPoolWorker-2]
root 10 0.0 0.0 8336 2688 ? Rs 20:57 0:00 ps aux
Tasks#
Tasks are the building blocks of Celery applications. Every task class has a unique name, and this name is referenced in broker messages so a worker can find the right function to execute. Ideally task functions should be idempotent, i.e. can be called multiple times with the same arguments with no side-effects.
from dtq.tasks import sleep
??sleep
Signature: sleep(wait=0, return_value=1)
Call signature: sleep(*a, **kw)
Type: PromiseProxy
String form: <@task: dtq.tasks.sleep of distributed-task-queue at 0x10554cbb0>
File: ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:
@app.task
def sleep(wait=0, return_value=1):
time.sleep(wait)
return return_value
Class docstring:
Task base class.
Note:
When called tasks apply the :meth:`run` method. This method must
be defined by all tasks (that is unless the :meth:`__call__` method
is overridden).
Tasks pushed to the queue will have not have results until it has been fully processed by a worker, so its return value is an celery.result.AsyncResult object:
result = sleep.delay(wait=3)
result
<AsyncResult: ada4cc70-9b55-4da5-a8e3-4385ce98b8f6>
Getting the actual result. This will block execution until result is ready:
import time
start = time.time()
print(result.get())
print(f"Time elapsed: {time.time() - start:.3f}s")
1
Time elapsed: 3.046s
A task that has an exception is acknowledged (i.e. removed from the queue) with FAILURE state:
from dtq.tasks import random_fail
while True:
result = random_fail.delay()
try:
result.get()
except ZeroDivisionError:
break
result.id, result.status, result.result
('c8e5beaa-8da6-442e-8c94-2276d9c6051a',
'FAILURE',
ZeroDivisionError('division by zero'))
??random_fail
Signature: random_fail()
Call signature: random_fail(*a, **kw)
Type: PromiseProxy
String form: <@task: dtq.tasks.random_fail of distributed-task-queue at 0x10554cbb0>
File: ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:
@app.task
def random_fail():
time.sleep(0.1)
x = random.choice([0, 1])
return 1 / x
Class docstring:
Task base class.
Note:
When called tasks apply the :meth:`run` method. This method must
be defined by all tasks (that is unless the :meth:`__call__` method
is overridden).
Flower UI#
Flower UI can be used to visualize task results. Flower has an extensive API for sending and querying tasks and changing worker configurations, but as mentioned above we will develop our own tasks and results API for our use-case.
Show code cell content
from selenium import webdriver
TAKE_SCREENSHOT = True
def take_screenshot(url: str, zoom: str, savename: str, wait=4.0):
"""Automate taking screenshot of our services' UI / docs. This requires chromedriver:
https://www.geeksforgeeks.org/how-to-run-selenium-running-test-on-chrome-using-webdriver/"""
if os.path.isfile("/usr/local/bin/chromedriver") and TAKE_SCREENSHOT:
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--window-size=1200,660")
driver = webdriver.Chrome(options=options)
driver.get(url)
time.sleep(wait)
driver.refresh()
viewport_height = int(driver.execute_script("return window.innerHeight"))
viewport_width = int(driver.execute_script("return window.innerWidth"))
new_viewport_height = int(viewport_height * (100 / zoom))
new_viewport_width = int(viewport_width * (100 / zoom))
driver.set_window_size(new_viewport_width, new_viewport_height)
driver.save_screenshot(f"{savename}.png")
driver.quit()
# Getting failed task example
FLOWER_URL = "http://localhost:5555"
take_screenshot(FLOWER_URL + "/tasks", 67, "./img/04-celery-tasks", wait=3)
take_screenshot(FLOWER_URL + f"/task/{result.id}", 75, "./img/04-celery-stacktrace")
take_screenshot(FLOWER_URL + "/workers", 100, "./img/04-celery-workers-tasks")
Fig. 77 Tasks shown in the Flower UI. Observe that tasks are automatically distributed to each worker:#
Fig. 78 Stack trace of failed task is also included in the UI.#
Fig. 79 Counts for successful and failed tasks by worker are shown in Flower.#
Celery config#
Depending on the use-case and program behavior that you want to achieve, you will have to spend some time reading the docs to get the correct config variables and testing if the settings achieve the desired behavior. In this section, we will focus on the contents of CELERY_CONFIG. Each key of this dictionary is a configuration variable. All configuration variables can be accessed as a dict object in app.conf.
# ./dtq/app.py
import os
from celery import Celery
REDIS_HOST = os.environ["REDIS_HOST"]
REDIS_PORT = os.environ["REDIS_PORT"]
RABBITMQ_HOST = os.environ["RABBITMQ_HOST"]
RABBITMQ_PORT = os.environ["RABBITMQ_PORT"]
PROJECT_NAME = "distributed-task-queue"
BACKEND_URL = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
BROKER_URL = f"pyamqp://guest:guest@{RABBITMQ_HOST}:{RABBITMQ_PORT}//"
app = Celery(
PROJECT_NAME,
broker=BROKER_URL,
backend=BACKEND_URL,
include=("dtq.tasks",),
)
CELERY_CONFIG = {
"task_acks_late": True,
"worker_prefetch_multiplier": 1,
"task_default_priority": 0,
"task_queue_max_priority": 10,
"task_create_missing_queues": False,
"result_expires": None,
"result_extended": True,
}
app.conf.update(CELERY_CONFIG)
Remark. Results expire in 1 day by default, so we set 'results_expires' to None. This is bad in practice, as we can easily run out of storage. This highlights the fact that Celery config has a lot of sharp corners. Similarly, the broker needs specific configuration. There is really no way around it but reading the docs and experimenting certain scenarios.
Acknowledgement#
A task message is not removed from the queue until it has been acknowledged by a worker. A worker can reserve many messages in advance, and even if the worker is killed (e.g. when autoscaling) the message may be redelivered to another worker. The default behavior is that a task is acknowledged once a worker starts processing a task:
Since the worker cannot detect if your tasks are idempotent, the default behavior is to acknowledge the message in advance, just before it’s executed, so that a task invocation that already started is never executed again.
For our use-case, we set task_acks_late to True so that received tasks are not acknowledged until it is completed. This is nice since it allows tasks to remain in the queue so that other workers can pick it up whenever one worker is terminated.
Messages in the broker which are not yet received by a worker are called ready.
Remark. Note that worker shutdown is different from simply killing a worker process.
tasks = []
for i in range(6):
task = sleep.delay(6.0)
tasks.append(task)
Fig. 80 Message counts in the broker tab showing the count of unacklowleged tasks in the queue. This shows 2 ready and 4 unacknowledged messages. This makes sense since we are queueing 6 tasks, while we only have 4 worker processes.#
Remark. Messages equals the sum of unacked and ready (Fig. 80). This is not true by default. We configured Celery so that a task is acknowledged only after a worker is done processing it. The default behavior is that a task is acknowledged once a worker starts processing a task:
Since the worker cannot detect if your tasks are idempotent, the default behavior is to acknowledge the message in advance, just before it’s executed, so that a task invocation that already started is never executed again.
Let us demonstrate redelivery of tasks when a worker is terminated:
import pandas as pd
from dtq.app import app
def get_active_tasks() -> pd.DataFrame:
time.sleep(0.5)
inspector = app.control.inspect()
active = inspector.active() or {}
output = []
for worker in active.keys():
for metadata in active[worker]:
delivery_info = metadata.pop("delivery_info")
metadata["task_id"] = metadata.pop("id")
metadata.update(delivery_info)
output.append(metadata)
return pd.DataFrame(output)
result = sleep.delay(wait=1200)
get_active_tasks()
| name | args | kwargs | type | hostname | time_start | acknowledged | worker_pid | task_id | exchange | routing_key | priority | redelivered | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | dtq.tasks.sleep | [] | {'wait': 1200} | dtq.tasks.sleep | celery@364128f8e977 | 1.708635e+09 | False | 9 | 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 | celery | 0 | False |
Shutting down this worker:
active_tasks = get_active_tasks().query(f"task_id=='{result.task_id}'")
worker_id = active_tasks.loc[0, "hostname"].replace("celery@", "")
!docker restart {worker_id}
Task should be picked up by the other worker. Notice redelivered is now True:
get_active_tasks()
| name | args | kwargs | type | hostname | time_start | acknowledged | worker_pid | task_id | exchange | routing_key | priority | redelivered | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | dtq.tasks.sleep | [] | {'wait': 1200} | dtq.tasks.sleep | celery@bdfaad682753 | 1.708635e+09 | False | 9 | 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 | celery | 0 | True |
Revoking tasks#
Tasks can be revoked using app.control.revoke. This sets task status to REVOKED such that any worker will not pick up this task from the queue. But if the task has already started processing, revoking the task will not cancel it:
while len(app.control.ping()) == 0: # See remark below
time.sleep(1.0)
app.control.revoke(result.task_id)
print(f"Task {result.task_id} revoked.")
print(f"Remaining tasks: {len(get_active_tasks())}")
get_active_tasks()
Task 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 revoked.
Remaining tasks: 1
| name | args | kwargs | type | hostname | time_start | acknowledged | worker_pid | task_id | exchange | routing_key | priority | redelivered | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | dtq.tasks.sleep | [] | {'wait': 1200} | dtq.tasks.sleep | celery@bdfaad682753 | 1.708635e+09 | False | 9 | 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 | celery | 0 | True |
Restarting the worker. Note that task is not picked up by the other available worker:
while len(app.control.ping()) < 2: # Need always one worker present (i.e. 1 after restarting the other)
time.sleep(1.0)
active_tasks = get_active_tasks().query(f"task_id=='{result.task_id}'")
worker_id = active_tasks.loc[0, "hostname"].replace("celery@", "")
print(f"Restarting worker celery@{worker_id}...")
!docker restart {worker_id} > /dev/null
print(f"Remaining tasks: {len(get_active_tasks())}")
Restarting worker celery@bdfaad682753...
Remaining tasks: 0
Remark. Note that revoke works by sending a broadcast message to all the workers, the workers then keep a list of revoked tasks in memory. When a worker starts up it will synchronize revoked tasks with other workers in the cluster. If all workers restart (or are terminated), the list of revoked ids will also vanish! ⚠
This is bad for our setup since it only has two workers. Indeed, an earlier version of the code had this bug (Fig. 82). A non-fragile setup is therefore to have at least 3 workers. To check worker availability, we do:
app.control.ping()
[{'celery@364128f8e977': {'ok': 'pong'}},
{'celery@bdfaad682753': {'ok': 'pong'}}]
Stopping a worker process#
To abort an active task without having to restart or terminate a worker,
we can use revoke with terminate=True. Note that this will terminate the worker process not the task itself! This distinction is important — the process may have already started processing another task at the point when the signal is sent. Never call this programmatically! ⚠
while len(app.control.ping()) == 0: # See remark above
time.sleep(1.0)
result = sleep.delay(wait=300)
print(f"Task {result.id} started.")
app.control.revoke(result.id, terminate=True)
print(f"Task {result.id} revoked with SIGTERM.")
print(f"Remaining tasks: {len(get_active_tasks())}")
Task c1e1324c-19a5-44f3-ae19-e55129ee5e1a started.
Task c1e1324c-19a5-44f3-ae19-e55129ee5e1a revoked with SIGTERM.
Remaining tasks: 0
Remark. The revoke function has an optional signal argument which defaults to 'SIGTERM'. This can be the uppercase name of any signal defined in the signal module from the standard library. This allows time for cleanup. If the process does not stop after a considerable amount of time, then a 'SIGKILL' signal may be sent. But this comes with some complications.
Fig. 81 Revoked tasks.#
Fig. 82 Task revoke fails when remote broadcast is done improperly. To summarize, only revoke when there is at least one worker. And be careful when shutting down all workers, as data shared by the workers is lost.#
Task prioritization#
Suppose we have priority levels for our tasks with higher priority tasks to be processed first once a worker is free. The following config keys seek to accomplish this:
CELERY_CONFIG = {
...,
"worker_prefetch_multiplier": 1,
"task_default_priority": 0,
"task_queue_max_priority": 10,
"task_create_missing_queues": False,
...
}
The number of reserved messages by each worker process is equal to the prefetch multiplier which defaults to 4. For long-running tasks the network overhead of picking up from the queue is negligible, so setting the prefetch multiplier to 1 is fine. Moreover, this allows messages to be ordered in the queue which is supported by RabbitMQ. Here we have to match the max priority with the correct queue arguments of RabbitMQ (i.e. x-max-priority of 10.) This allows us to set a priority for each task.
The last variable is for raising QueueNotFound when a task is sent to an undefined queue. Without this, client-side logic could send tasks into a queue that is automatically created, but does not have a consuming worker. This can be hard to spot in production.
Sending tasks with priority:
sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)
sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)
sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)
sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)
sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)
sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)
sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)
sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)
sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)
sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3)
sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3)
sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3);
Here we use apply_async which is just delay but allows us to set task parameters such as priority which as configured defaults to 0. Tasks with higher integer priority value should be picked up first when using a RabbitMQ broker:
Fig. 83 Tasks ordered by started. Notice that tasks with priority 0 and 1 start first, because they are called first. But once these tasks block the workers, tasks with higher priority are processed first.#
Tasks and Results API#
Endpoints for creating tasks are exposed to other parts of the business which input specific metadata that it can use as arguments to the request during downstream processing. This also standardizes task creation and retrieval in one place. Also, using an API ensures that all data communicated between services are validated.
# ./docker-compose.yml
version: "3"
services:
api:
build:
dockerfile: Dockerfile.api
ports:
- "8000:8000"
depends_on:
- redis
- rabbitmq
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672
command: ["--port", "8000", "--host", "0.0.0.0"]
# ... other services
Environmental variables are loaded which identify the broker and backend URLs. The container for this is built as follows. It copies both the celery application code and the API code and installs their requirements. Note that the requirements file of the Celery app is used to install its respective dependencies preventing version drift (i.e. celery appears once in the two requirements files).
# ./Dockerfile.api
FROM python:3.9.15-slim
WORKDIR /opt
COPY ./dtq/requirements.txt dtq-requirements.txt
COPY ./api/requirements.txt api-requirements.txt
RUN pip install -r dtq-requirements.txt
RUN pip install -r api-requirements.txt
COPY ./dtq ./dtq
COPY ./api .
EXPOSE 8000
ENTRYPOINT ["uvicorn", "main:app"]
The main module simply loads tasks are results routers:
# ./api/main.py
import results
import tasks
from fastapi import FastAPI
from fastapi.responses import RedirectResponse
app = FastAPI()
app.include_router(results.router)
app.include_router(tasks.router)
@app.get("/", include_in_schema=False)
async def docs_redirect():
return RedirectResponse(url="/docs")
Below pending (queued) and active tasks (processing) have their specific schemas:
# ./api/models.py
...
class Task(BaseModel):
task_id: str
status: str
class ActiveTask(BaseModel):
task_id: str
task: str
args: list
kwargs: dict
worker: str
time_start: datetime
acknowledged: bool
priority: int
redelivered: bool
worker_pid: int
class PendingTask(BaseModel):
task_id: str
task: str
retries: int
timelimit: list
root_id: str
parent_id: Optional[str]
origin: str
args: list
kwargs: dict
...
Fig. 84 FastAPI app port forwarded to localhost:8000.#
Tasks endpoints#
First we look at the first two task endpoints. These push the corresponding tasks to the queue. Here we can add validation for query parameters such as constraining wait to be nonnegative. Data validation is one of the primary benefits of having an API for producing tasks. Note that the we can now set priorities for each task:
# ./api/tasks.py
from fastapi import APIRouter, status
from fastapi.responses import JSONResponse
from models import ActiveTask, PendingTask, Task
from utils import exist_workers, poll_messages
from dtq import tasks
from dtq.app import app as celery_app
router = APIRouter(prefix="/tasks", tags=["tasks"])
@router.post("/sleep", response_model=Task)
async def sleep(wait: float, return_value: int = 1, priority: int = 0) -> str:
result = tasks.sleep.apply_async(
kwargs={"wait": wait, "return_value": return_value},
priority=priority,
)
task_id = result.task_id
return {"task_id": task_id, "status": result.status}
@router.post("/random_fail", response_model=Task)
async def random_fail():
result = tasks.random_fail.delay()
task_id = result.task_id
return {"task_id": task_id, "status": result.status}
...
Trying these out:
import requests
API_URL = "http://localhost:8000"
sleep_response = requests.post(f"{API_URL}/tasks/sleep?wait=5&return_value=1&priority=0")
sleep_response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551', 'status': 'PENDING'}
fail_response = requests.post(f"{API_URL}/tasks/random_fail")
fail_response.json()
{'task_id': '62fc3ab0-3fca-424e-acb3-5376082a7b8b', 'status': 'PENDING'}
Results endpoints#
The results/{task_id} endpoint returns the task result and status. It returns null result when the task is still processing or completed but failed. See the docs for built-in states. Note that Celery assigns the exception for failed tasks in the .result attribute of the AsyncResult object.
# ./api/results.py
from celery.result import AsyncResult
from fastapi import APIRouter
from models import Result
router = APIRouter(prefix="/results", tags=["results"])
@router.get("/{task_id}", response_model=Result)
async def fetch_result(task_id):
result = AsyncResult(task_id)
return {
"task_id": result.task_id,
"status": result.status,
"successful": result.successful(),
"args": result.args,
"kwargs": result.kwargs,
"date_done": result.date_done,
"result": result.result,
}
The results data model is as follows. Note that result is optional since since a task has delayed result (or None when failed). Recall in the configuration we have results_extended set to True. This is so that task args and kwargs are sent to the results backend. Here result has type str since it can be an exception or some return value (e.g. float or int).
# ./api/models.py (continued)
...
class Result(BaseModel):
task_id: str
status: str
successful: bool
result: Optional[str]
args: Optional[list]
kwargs: Optional[dict]
date_done: Optional[datetime]
@validator("result", pre=True)
def validate_result(cls, value):
if value is None:
return None
return str(value)
Sleep task above still running. The successful field defaults to False while others are null.
task_id = sleep_response.json()["task_id"]
response = requests.get(f"{API_URL}/results/{task_id}")
response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551',
'status': 'PENDING',
'successful': False,
'result': None,
'args': None,
'kwargs': None,
'date_done': None}
Result of the random_fail endpoint:
time.sleep(0.5)
task_id = fail_response.json()["task_id"]
response = requests.get(f"{API_URL}/results/{task_id}")
response.json()
{'task_id': '62fc3ab0-3fca-424e-acb3-5376082a7b8b',
'status': 'FAILURE',
'successful': False,
'result': 'division by zero',
'args': [],
'kwargs': {},
'date_done': '2024-02-22T20:58:38.894050'}
Note result is the exception as string. Getting the result once the long-running task is done:
from celery.result import AsyncResult
task_id = sleep_response.json()["task_id"]
result = AsyncResult(task_id)
while not result.ready():
continue
response = requests.get(f"http://127.0.0.1:8000/results/{task_id}")
response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551',
'status': 'SUCCESS',
'successful': True,
'result': '1',
'args': [],
'kwargs': {'wait': 5.0, 'return_value': 1},
'date_done': '2024-02-22T20:58:43.771632'}
Pending and active tasks#
The next endpoints list active and pending tasks. Recall our workers are configured to prefetch one task from the queue. A good mental model is that Celery can only see what the workers can see. For example, revoke does not work when there are no workers! Pending tasks therefore have to be queried directly from RabbitMQ using a library such as pika:
# ./api/utils.py
...
def poll_messages(queue="celery"):
parameters = pika.ConnectionParameters(
host=os.environ["RABBITMQ_HOST"],
port=os.environ["RABBITMQ_PORT"],
)
conn = pika.BlockingConnection(parameters)
queue_args = {"x-max-priority": 10}
channel = conn.channel()
channel.queue_declare(queue, durable=True, arguments=queue_args)
messages = []
while True:
response = channel.basic_get(queue=queue, auto_ack=False)
method_frame, properties, _ = response
if method_frame is None:
break
metadata = properties.headers
fields = [
"id",
"task",
"retries",
"timelimit",
"root_id",
"parent_id",
"origin",
]
msg = {k: metadata[k] for k in fields}
msg["task_id"] = msg.pop("id")
msg["args"] = ast.literal_eval(metadata["argsrepr"])
msg["kwargs"] = ast.literal_eval(metadata["kwargsrepr"])
messages.append(msg)
conn.close()
return messages
Remark. Queue declare above should match the queue configuration in the main app. Best practice is to have a unified configuration which we import. But here we just hard code.
Getting active tasks (i.e. started by a worker and removed from the queue):
# tasks.py (continued)
...
@router.get("/active")
async def active_tasks() -> list[ActiveTask]:
inspector = celery_app.control.inspect()
active = inspector.active() or {}
result = []
for worker in active.keys():
for metadata in active[worker]:
delivery_info = metadata.pop("delivery_info")
metadata["task_id"] = metadata.pop("id")
metadata["task"] = metadata.pop("name")
metadata["worker"] = metadata.pop("hostname")
metadata["priority"] = delivery_info["priority"]
metadata["redelivered"] = delivery_info["redelivered"]
result.append(metadata)
return result
@router.get("/pending")
async def pending_tasks(queue: str = "celery") -> list[PendingTask]:
messages = poll_messages(queue)
return messages
...
Remark. For querying individual active tasks, we can define an endpoint that uses:
inspector.query_task(task_id)
Queueing up some tasks to try out the endpoints:
for i in range(6):
requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")
Expecting a list of length four corresponding to four worker processes:
active = requests.get(f"{API_URL}/tasks/active").json()
active
Show code cell output
[{'task_id': '7b0dee69-aab7-4033-ada7-c9f7f84f2437',
'task': 'dtq.tasks.sleep',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'worker': 'celery@bdfaad682753',
'time_start': '2024-02-22T20:58:43.799137Z',
'acknowledged': False,
'priority': 0,
'redelivered': False,
'worker_pid': 9},
{'task_id': '044238f6-277b-4c1d-824c-65529e27ae04',
'task': 'dtq.tasks.sleep',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'worker': 'celery@bdfaad682753',
'time_start': '2024-02-22T20:58:43.839229Z',
'acknowledged': False,
'priority': 0,
'redelivered': False,
'worker_pid': 8},
{'task_id': '6cb3d744-6b32-4bf6-94b7-9dd7d6975a33',
'task': 'dtq.tasks.sleep',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'worker': 'celery@364128f8e977',
'time_start': '2024-02-22T20:58:43.818486Z',
'acknowledged': False,
'priority': 0,
'redelivered': False,
'worker_pid': 10},
{'task_id': '7fe15dbe-ac70-4382-a89d-b61d2887e30a',
'task': 'dtq.tasks.sleep',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'worker': 'celery@364128f8e977',
'time_start': '2024-02-22T20:58:43.850899Z',
'acknowledged': False,
'priority': 0,
'redelivered': False,
'worker_pid': 8}]
Expecting a list of length two:
pending = requests.get(f"{API_URL}/tasks/pending").json()
pending
Show code cell output
[{'task_id': 'a5c75cc8-0113-44cc-979a-146fb9b66c45',
'task': 'dtq.tasks.sleep',
'retries': 0,
'timelimit': [None, None],
'root_id': 'a5c75cc8-0113-44cc-979a-146fb9b66c45',
'parent_id': None,
'origin': 'gen1@0a8d0d78cf08',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1}},
{'task_id': 'a0cf96cd-61cf-4915-96e2-48d60d5f91a7',
'task': 'dtq.tasks.sleep',
'retries': 0,
'timelimit': [None, None],
'root_id': 'a0cf96cd-61cf-4915-96e2-48d60d5f91a7',
'parent_id': None,
'origin': 'gen1@0a8d0d78cf08',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1}}]
Total pending tasks#
For our scalable architecture, we imagine horizontally scaling the workers in and out depending on the size of the pending tasks which we abstract using the wait variable. Size of pending tasks depend on how fast the workers are able to process the tasks and the rate and size of incoming tasks. We define the following function:
def get_total_pending_size(queue="celery"):
pending_size = 0
response = requests.get(f"{API_URL}/tasks/pending?queue={queue}")
messages = response.json()
for m in messages:
if m["task"] == "dtq.tasks.sleep":
pending_size += m["kwargs"]["wait"]
return pending_size
Below we monitor the pending size while we simulate sleep tasks using threading:
Show code cell source
import threading
import numpy as np
ps = []
ts = []
def tasksim():
n = 100
for _ in range(n):
t = np.random.random() * 4.0
ts.append(t)
requests.post(f"{API_URL}/tasks/sleep?wait={t}&return_value=1&priority=0")
time.sleep(np.random.random())
def monitor():
while True:
time.sleep(0.05)
if response.ok:
ps.append(get_total_pending_size())
else:
continue
if len(ts) == 100:
break
thread_monitor = threading.Thread(target=monitor)
thread_tasksim = threading.Thread(target=tasksim)
thread_monitor.start()
thread_tasksim.start()
thread_monitor.join()
thread_tasksim.join()
Plotting:
Show code cell source
import matplotlib.pyplot as plt
from matplotlib_inline import backend_inline
backend_inline.set_matplotlib_formats("svg")
plt.plot(np.arange(len(ps)) * 0.05, ps)
plt.grid(linestyle="dotted")
plt.ylabel("Total task size (pending)")
plt.xlabel("Time (s)");
Fig. 85 Monitoring the RabbitMQ broker in localhost:15672. The dashboard also shows ready, unacked, and total message counts, as well as message rates. Resource usage metrics can be important when load testing.#
Task abort and revoke#
Remote commands do not work without workers. So we have to check if workers exist:
# ./api/utils.py (continued)
...
def exist_workers():
inspector = celery_app.control.inspect()
return inspector.ping() is not None
...
The following endpoints use the Celery API which allow for proper handling:
# ./api/tasks.py (continued)
...
@router.post("/revoke/{task_id}")
async def revoke_task(task_id: str) -> JSONResponse:
if not exist_workers():
message = f"Task {task_id} revoke failed. No workers found."
status_code = status.HTTP_400_BAD_REQUEST
else:
celery_app.control.revoke(task_id)
message = f"Task {task_id} revoked."
status_code = status.HTTP_200_OK
return JSONResponse({"message": message}, status_code=status_code)
@router.post("/abort/{task_id}")
async def abort_task(task_id: str) -> JSONResponse:
if not exist_workers():
message = f"Task {task_id} abort failed. No workers found."
status_code = status.HTTP_400_BAD_REQUEST
else:
celery_app.control.revoke(task_id, terminate=True)
message = f"Task {task_id} aborted."
status_code = status.HTTP_200_OK
return JSONResponse({"message": message}, status_code=status_code)
Revoke pending test#
Running 6 tasks (4 active tasks and 2 pending):
task_ids = []
for i in range(6):
time.sleep(0.1)
response = requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")
task_id = response.json()["task_id"]
task_ids.append(task_id)
pending = requests.get(f"{API_URL}/tasks/pending").json()
revoke_task_id = pending[-1]["task_id"]
requests.post(f"{API_URL}/tasks/revoke/{revoke_task_id}").json()
{'message': 'Task 2bfe74da-0c1b-49bb-88e2-930ca7d0e9db revoked.'}
Getting results of tasks:
for tid in task_ids:
while not AsyncResult(tid).ready():
time.sleep(0.1)
response = requests.get(f"{API_URL}/results/{tid}")
result = response.json()
print(f"{tid}:", result["status"], result["result"])
1aca6463-f7e3-4df6-ba90-70c875180580: SUCCESS 1
4ac38d1b-964b-43ca-ace5-cf8106e8ccb6: SUCCESS 1
9c3ae1fa-ef7b-4013-a020-76e16262eb82: SUCCESS 1
d390b1c5-90f6-4688-8b25-b3a880cd908d: SUCCESS 1
627b7faa-2cd2-41b6-9c3b-110b6d161cfe: SUCCESS 1
2bfe74da-0c1b-49bb-88e2-930ca7d0e9db: REVOKED revoked
Revoked task from pending list did not succeed:
response = requests.get(f"{API_URL}/results/{revoke_task_id}")
response.json()
{'task_id': '2bfe74da-0c1b-49bb-88e2-930ca7d0e9db',
'status': 'REVOKED',
'successful': False,
'result': 'revoked',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'date_done': '2024-02-22T20:59:43.703656'}
Abort active test#
Running 4 tasks (4 active, 0 pending):
task_ids = []
for i in range(4):
time.sleep(0.1)
response = requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")
task_id = response.json()["task_id"]
task_ids.append(task_id)
active = requests.get(f"{API_URL}/tasks/active").json()
abort_task_id = active[-1]['task_id']
requests.post(f"{API_URL}/tasks/abort/{abort_task_id}").json()
{'message': 'Task 58d2f6b7-8802-4b09-8c0b-7075d6e415ae aborted.'}
Wait for tasks to finish and getting results:
for tid in task_ids:
while not AsyncResult(tid).ready():
time.sleep(0.1)
response = requests.get(f"{API_URL}/results/{tid}")
result = response.json()
print(f"{tid}:", result["status"], result["result"])
2a1044a1-259d-4af0-8f49-520373da50f6: SUCCESS 1
eaabf3e8-47be-4702-a78e-5fc4358b51d5: SUCCESS 1
58d2f6b7-8802-4b09-8c0b-7075d6e415ae: REVOKED terminated
796f88ed-f77e-44e0-a881-90b08e428c32: SUCCESS 1
Cool. Celery assigns task result as terminated:
response = requests.get(f"{API_URL}/results/{abort_task_id}")
response.json()
{'task_id': '58d2f6b7-8802-4b09-8c0b-7075d6e415ae',
'status': 'REVOKED',
'successful': False,
'result': 'terminated',
'args': [],
'kwargs': {'wait': 3.0, 'return_value': 1},
'date_done': '2024-02-22T20:59:49.102432'}
Appendix: Further Celery#
Source: particle1331/distributed-task-queue (⚠ further-celery branch)
!rm -rf ./distributed-task-queue
!git clone --single-branch --branch further-celery https://github.com/particle1331/distributed-task-queue.git ./distributed-task-queue > /dev/null
!pip install -U git+https://github.com/particle1331/distributed-task-queue.git@further-celery --force-reinstall > /dev/null
!echo "\nSuccessfully installed: $(pip list | grep distributed-task-queue)/$(wget --header='Accept: application/vnd.github.VERSION.sha' -qO- commit_id http://api.github.com/repos/particle1331/distributed-task-queue/commits/further-celery | head -c 7)"
Cloning into './distributed-task-queue'...
remote: Enumerating objects: 289, done.
remote: Counting objects: 100% (106/106), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 289 (delta 82), reused 59 (delta 58), pack-reused 183
Receiving objects: 100% (289/289), 1.06 MiB | 1.97 MiB/s, done.
Resolving deltas: 100% (167/167), done.
Running command git clone --filter=blob:none --quiet https://github.com/particle1331/distributed-task-queue.git /private/var/folders/jq/9vsvd9252_349lsng_5gc_jw0000gn/T/pip-req-build-0l0gj037
Running command git checkout -b further-celery --track origin/further-celery
Switched to a new branch 'further-celery'
branch 'further-celery' set up to track 'origin/further-celery'.
Successfully installed: distributed-task-queue 0.0.0/b5aad0c
Rebuilding stack:
!docker compose -f ./distributed-task-queue/docker-compose.yml down
!docker compose -f ./distributed-task-queue/docker-compose.yml up -d --build
Show code cell output
?25l[+] Running 0/0
⠋ Container distributed-task-queue-flower-1 Stopping 0.0s
⠋ Container distributed-task-queue-api-1 Stopping 0.0s
⠋ Container distributed-task-queue-worker-1 Stopping 0.0s
⠋ Container distributed-task-queue-worker-2 Stopping 0.0s
?25h?25l[+] Running 0/4
⠙ Container distributed-task-queue-flower-1 Stopping 0.1s
⠙ Container distributed-task-queue-api-1 Stopping 0.1s
⠙ Container distributed-task-queue-worker-1 Stopping 0.1s
⠙ Container distributed-task-queue-worker-2 Stopping 0.1s
?25h?25l[+] Running 0/4
⠹ Container distributed-task-queue-flower-1 Stopping 0.2s
⠹ Container distributed-task-queue-api-1 Stopping 0.2s
⠹ Container distributed-task-queue-worker-1 Stopping 0.2s
⠹ Container distributed-task-queue-worker-2 Stopping 0.2s
?25h?25l[+] Running 0/4
⠸ Container distributed-task-queue-flower-1 Stopping 0.3s
⠸ Container distributed-task-queue-api-1 Stopping 0.3s
⠸ Container distributed-task-queue-worker-1 Stopping 0.3s
⠸ Container distributed-task-queue-worker-2 Stopping 0.3s
?25h?25l[+] Running 0/4
⠼ Container distributed-task-queue-flower-1 Stopping 0.4s
⠼ Container distributed-task-queue-api-1 Stopping 0.4s
⠼ Container distributed-task-queue-worker-1 Stopping 0.4s
⠼ Container distributed-task-queue-worker-2 Stopping 0.4s
?25h?25l[+] Running 0/4
⠴ Container distributed-task-queue-flower-1 Stopping 0.5s
⠴ Container distributed-task-queue-api-1 Stopping 0.5s
⠴ Container distributed-task-queue-worker-1 Stopping 0.5s
⠴ Container distributed-task-queue-worker-2 Stopping 0.5s
?25h?25l[+] Running 0/4
⠦ Container distributed-task-queue-flower-1 Stopping 0.6s
⠦ Container distributed-task-queue-api-1 Stopping 0.6s
⠦ Container distributed-task-queue-worker-1 Stopping 0.6s
⠦ Container distributed-task-queue-worker-2 Stopping 0.6s
?25h?25l[+] Running 0/4
⠧ Container distributed-task-queue-flower-1 Stopping 0.7s
⠧ Container distributed-task-queue-api-1 Stopping 0.7s
⠧ Container distributed-task-queue-worker-1 Stopping 0.7s
⠧ Container distributed-task-queue-worker-2 Stopping 0.7s
?25h?25l[+] Running 0/4
⠇ Container distributed-task-queue-flower-1 Stopping 0.8s
⠇ Container distributed-task-queue-api-1 Stopping 0.8s
⠇ Container distributed-task-queue-worker-1 Stopping 0.8s
⠇ Container distributed-task-queue-worker-2 Stopping 0.8s
?25h?25l[+] Running 0/4
⠏ Container distributed-task-queue-flower-1 Stopping 0.9s
⠏ Container distributed-task-queue-api-1 Stopping 0.9s
⠏ Container distributed-task-queue-worker-1 Stopping 0.9s
⠏ Container distributed-task-queue-worker-2 Stopping 0.9s
?25h?25l[+] Running 1/4
✔ Container distributed-task-queue-flower-1 Stopped 1.0s
⠋ Container distributed-task-queue-api-1 Stopping 1.0s
⠋ Container distributed-task-queue-worker-1 Stopping 1.0s
⠋ Container distributed-task-queue-worker-2 Stopping 1.0s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠙ Container distributed-task-queue-worker-1 Stopping 1.1s
⠙ Container distributed-task-queue-worker-2 Stopping 1.1s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠹ Container distributed-task-queue-worker-1 Stopping 1.2s
⠹ Container distributed-task-queue-worker-2 Stopping 1.2s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠸ Container distributed-task-queue-worker-1 Stopping 1.3s
⠸ Container distributed-task-queue-worker-2 Stopping 1.3s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠼ Container distributed-task-queue-worker-1 Stopping 1.4s
⠼ Container distributed-task-queue-worker-2 Stopping 1.4s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠴ Container distributed-task-queue-worker-1 Stopping 1.5s
⠴ Container distributed-task-queue-worker-2 Stopping 1.5s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠦ Container distributed-task-queue-worker-1 Stopping 1.6s
⠦ Container distributed-task-queue-worker-2 Stopping 1.6s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠧ Container distributed-task-queue-worker-1 Stopping 1.7s
⠧ Container distributed-task-queue-worker-2 Stopping 1.7s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠇ Container distributed-task-queue-worker-1 Stopping 1.8s
⠇ Container distributed-task-queue-worker-2 Stopping 1.8s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠏ Container distributed-task-queue-worker-1 Stopping 1.9s
⠏ Container distributed-task-queue-worker-2 Stopping 1.9s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠋ Container distributed-task-queue-worker-1 Stopping 2.0s
⠋ Container distributed-task-queue-worker-2 Stopping 2.0s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠙ Container distributed-task-queue-worker-1 Stopping 2.1s
⠙ Container distributed-task-queue-worker-2 Stopping 2.1s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠹ Container distributed-task-queue-worker-1 Stopping 2.2s
⠹ Container distributed-task-queue-worker-2 Stopping 2.2s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠸ Container distributed-task-queue-worker-1 Stopping 2.3s
⠸ Container distributed-task-queue-worker-2 Stopping 2.3s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠼ Container distributed-task-queue-worker-1 Stopping 2.4s
⠼ Container distributed-task-queue-worker-2 Stopping 2.4s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠴ Container distributed-task-queue-worker-1 Stopping 2.5s
⠴ Container distributed-task-queue-worker-2 Stopping 2.5s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠦ Container distributed-task-queue-worker-1 Stopping 2.6s
⠦ Container distributed-task-queue-worker-2 Stopping 2.6s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠧ Container distributed-task-queue-worker-1 Stopping 2.7s
⠧ Container distributed-task-queue-worker-2 Stopping 2.7s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠇ Container distributed-task-queue-worker-1 Stopping 2.8s
⠇ Container distributed-task-queue-worker-2 Stopping 2.8s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠏ Container distributed-task-queue-worker-1 Stopping 2.9s
⠏ Container distributed-task-queue-worker-2 Stopping 2.9s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠋ Container distributed-task-queue-worker-1 Stopping 3.0s
⠋ Container distributed-task-queue-worker-2 Stopping 3.0s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠙ Container distributed-task-queue-worker-1 Stopping 3.1s
⠙ Container distributed-task-queue-worker-2 Stopping 3.1s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠹ Container distributed-task-queue-worker-1 Stopping 3.2s
⠹ Container distributed-task-queue-worker-2 Stopping 3.2s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠸ Container distributed-task-queue-worker-1 Stopping 3.3s
⠸ Container distributed-task-queue-worker-2 Stopping 3.3s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠼ Container distributed-task-queue-worker-1 Stopping 3.4s
⠼ Container distributed-task-queue-worker-2 Stopping 3.4s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠴ Container distributed-task-queue-worker-1 Stopping 3.5s
⠴ Container distributed-task-queue-worker-2 Stopping 3.5s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠦ Container distributed-task-queue-worker-1 Stopping 3.6s
⠦ Container distributed-task-queue-worker-2 Stopping 3.6s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠧ Container distributed-task-queue-worker-1 Stopping 3.7s
⠧ Container distributed-task-queue-worker-2 Stopping 3.7s
?25h?25l[+] Running 2/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
⠇ Container distributed-task-queue-worker-1 Stopping 3.8s
⠇ Container distributed-task-queue-worker-2 Stopping 3.8s
?25h?25l[+] Running 4/4
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
⠋ Container distributed-task-queue-redis-1 Stopping 0.0s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 0.0s
?25h?25l[+] Running 4/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
⠙ Container distributed-task-queue-redis-1 Stopping 0.1s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 0.1s
?25h?25l[+] Running 4/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
⠹ Container distributed-task-queue-redis-1 Stopping 0.2s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 0.2s
?25h?25l[+] Running 4/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
⠸ Container distributed-task-queue-redis-1 Stopping 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 0.3s
?25h?25l[+] Running 4/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
⠼ Container distributed-task-queue-redis-1 Stopping 0.4s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 0.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Stopped 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 0.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 0.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 0.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 0.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 0.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 1.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 1.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 1.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 1.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 1.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 1.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 1.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 1.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 1.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 1.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 2.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 2.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 2.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 2.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 2.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 2.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 2.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 2.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 2.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 2.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 3.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 3.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 3.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 3.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 3.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 3.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 3.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 3.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 3.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 3.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 4.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 4.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 4.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 4.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 4.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 4.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 4.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 4.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 4.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 4.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 5.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 5.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 5.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 5.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 5.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 5.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 5.6s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 5.7s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 5.8s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 5.9s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 6.0s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 6.1s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 6.2s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 6.3s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 6.4s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 6.5s
?25h?25l[+] Running 5/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 6.6s
?25h?25l[+] Running 6/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
✔ Container distributed-task-queue-rabbitmq-1 Removed 6.7s
?25h?25l[+] Running 7/6
✔ Container distributed-task-queue-flower-1 Removed 1.0s
✔ Container distributed-task-queue-api-1 Removed 1.1s
✔ Container distributed-task-queue-worker-1 Removed 3.8s
✔ Container distributed-task-queue-worker-2 Removed 3.8s
✔ Container distributed-task-queue-redis-1 Removed 0.5s
✔ Container distributed-task-queue-rabbitmq-1 Removed 6.7s
✔ Network distributed-task-queue_default Removed 0.1s
?25h?25l[+] Building 0.0s (0/0) docker:desktop-linux
?25h?25l[+] Building 0.0s (4/5) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [worker internal] load metadata for docker.io/library/python:3.9.15-s 0.0s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
?25h?25l[+] Building 0.2s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.2s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 0.3s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.3s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 0.5s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.5s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 0.6s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.6s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 0.8s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.8s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 0.9s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 0.9s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 1.1s (6/7) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [other-worker internal] load metadata for docker.io/library/python:3. 1.1s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
?25h?25l[+] Building 1.2s (26/31) docker:desktop-linux
=> [api internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [api internal] load build definition from Dockerfile.api 0.0s
=> => transferring dockerfile: 340B 0.0s
=> [api internal] load metadata for docker.io/library/python:3.9.15-slim 1.1s
=> [worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load build definition from Dockerfile.worker 0.0s
=> => transferring dockerfile: 271B 0.0s
=> [other-worker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [other-worker 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:f 0.0s
=> [worker internal] load build context 0.0s
=> => transferring context: 3.10kB 0.0s
=> [api internal] load build context 0.0s
=> => transferring context: 8.97kB 0.0s
=> [other-worker internal] load build context 0.0s
=> => transferring context: 3.10kB 0.0s
=> CACHED [api 2/7] WORKDIR /opt 0.0s
=> CACHED [worker 3/7] COPY ./dtq/requirements.txt . 0.0s
=> CACHED [worker 4/7] RUN pip install -r requirements.txt 0.0s
=> CACHED [worker 5/7] RUN pip install setproctitle 0.0s
=> CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps 0.0s
=> CACHED [worker 7/7] COPY /dtq/ ./dtq/ 0.0s
=> [other-worker] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:904fbc852c456db67b1bed754759b15b2507336d29803 0.0s
=> => naming to docker.io/library/distributed-task-queue-other-worker 0.0s
=> [worker] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:ce1e8b2b2e09908b24c96f9d21fc3bc1a7660366bb364 0.0s
=> => naming to docker.io/library/distributed-task-queue-worker 0.0s
=> CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt 0.0s
=> CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt 0.0s
=> CACHED [api 5/8] RUN pip install -r dtq-requirements.txt 0.0s
=> CACHED [api 6/8] RUN pip install -r api-requirements.txt 0.0s
=> CACHED [api 7/8] COPY ./dtq ./dtq 0.0s
=> CACHED [api 8/8] COPY ./api . 0.0s
=> [api] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:8a0de09964cfbd59bd03ff1154eed5f68bec0ad24deed 0.0s
=> => naming to docker.io/library/distributed-task-queue-api 0.0s
?25h?25l[+] Running 0/0
⠋ Network distributed-task-queue_default Creating 0.0s
?25h?25l[+] Running 3/1
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
⠋ Container distributed-task-queue-api-1 Creating 0.0s
⠋ Container distributed-task-queue-worker-2 Creating 0.0s
⠋ Container distributed-task-queue-other-worker-1 Creating 0.0s
⠋ Container distributed-task-queue-worker-1 Creating 0.0s
⠋ Container distributed-task-queue-flower-1 Creating 0.0s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Created 0.1s
✔ Container distributed-task-queue-redis-1 Created 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Created 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Created 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Created 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Created 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Created 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
?25h?25l[+] Running 8/8
✔ Network distributed-task-queue_default Created 0.1s
✔ Container distributed-task-queue-rabbitmq-1 Started 0.1s
✔ Container distributed-task-queue-redis-1 Started 0.1s
✔ Container distributed-task-queue-api-1 Started 0.1s
✔ Container distributed-task-queue-worker-2 Started 0.1s
✔ Container distributed-task-queue-other-worker-1 Started 0.1s
✔ Container distributed-task-queue-worker-1 Started 0.1s
✔ Container distributed-task-queue-flower-1 Started 0.1s
?25h
!docker compose -f ./distributed-task-queue/docker-compose.yml ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
distributed-task-queue-api-1 distributed-task-queue-api "uvicorn main:app --…" api 3 seconds ago Up Less than a second 0.0.0.0:8000->8000/tcp
distributed-task-queue-flower-1 mher/flower "celery --broker=amq…" flower 3 seconds ago Up Less than a second 0.0.0.0:5555->5555/tcp
distributed-task-queue-other-worker-1 distributed-task-queue-other-worker "celery -A dtq.app w…" other-worker 3 seconds ago Up 1 second
distributed-task-queue-rabbitmq-1 rabbitmq:3.12.0-management "docker-entrypoint.s…" rabbitmq 3 seconds ago Up 1 second 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp
distributed-task-queue-redis-1 redis:latest "docker-entrypoint.s…" redis 3 seconds ago Up 1 second 0.0.0.0:6379->6379/tcp
distributed-task-queue-worker-1 distributed-task-queue-worker "celery -A dtq.app w…" worker 3 seconds ago Up 1 second
distributed-task-queue-worker-2 distributed-task-queue-worker "celery -A dtq.app w…" worker 3 seconds ago Up Less than a second
More options#
Some interesting Celery features we can look into for our application is to add retries. Having time limits can prevent tasks from blocking workers. For example, a task that is running 3 hours when the average task runs for about 3 minutes. This task can be cleaned up and logged so that further investigation can be done.
Reworking our previous task definition:
from dtq.tasks import random_fail
random_fail??
Signature: random_fail(prob=0.5)
Call signature: random_fail(*a, **kw)
Type: PromiseProxy
String form: <@task: dtq.tasks.random_fail of distributed-task-queue at 0x1057bc490>
File: ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:
@app.task(
autoretry_for=(ZeroDivisionError,),
max_retries=2,
retry_backoff=3,
retry_jitter=True,
)
def random_fail(prob=0.5):
time.sleep(0.1)
x = random.random()
return 1 / int(x > prob)
Class docstring:
Task base class.
Note:
When called tasks apply the :meth:`run` method. This method must
be defined by all tasks (that is unless the :meth:`__call__` method
is overridden).
This retries twice during a ZeroDivisionError with the first retry occuring between 0 and 3 seconds, and the next between 0 and 6 seconds. Testing:
result = random_fail.apply_async(kwargs={"prob": 1.0})
result
<AsyncResult: f2013e77-f82c-49f0-9d07-a0f6c649b991>
Fig. 86 Task retries. Note that child tasks from retries have the same task ID.#
Setting client-side time limits (in seconds). Note that these can be set at definition in the task decorators in case we have identified a suitable upper bound for the execution time for the task for all workers. The soft time limit raises SoftTimeLimitExceeded which allows handling or cleanup, before the hard time limit kills the process with SIGKILL.
from dtq.tasks import sleep
result = sleep.apply_async(
kwargs=dict(wait=3),
time_limit=2, soft_time_limit=1
)
while not result.ready():
continue
response = requests.get(f"{API_URL}/results/{result.id}")
response.json()
{'task_id': '45823544-da5c-4955-b331-272d968893a4',
'status': 'FAILURE',
'successful': False,
'result': 'SoftTimeLimitExceeded()',
'args': [],
'kwargs': {'wait': 3},
'date_done': '2024-02-22T21:00:48.620936'}
Hard time limit:
result = sleep.apply_async(kwargs=dict(wait=3), time_limit=1)
while not result.ready():
continue
response = requests.get(f"{API_URL}/results/{result.id}")
response.json()
{'task_id': 'ce748246-66ca-4821-b876-9f04ef1b6f04',
'status': 'FAILURE',
'successful': False,
'result': 'TimeLimitExceeded(1,)',
'args': [],
'kwargs': {'wait': 3},
'date_done': '2024-02-22T21:00:49.751109'}
Logging#
Celery automatically sets up logging. The compose file has been modified to write logs to a file:
celery -A dtq.app worker -l INFO -f celery.log
Checking logs for above task that exited with soft time limit:
!docker exec distributed-task-queue-worker-1 tail -14 /opt/celery.log
[2024-02-22 21:00:47,608: INFO/MainProcess] Task dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4] received
[2024-02-22 21:00:48,616: WARNING/MainProcess] Soft time limit (1s) exceeded for dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4]
[2024-02-22 21:00:48,624: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:48,625: ERROR/ForkPoolWorker-1] Task dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4] raised unexpected: SoftTimeLimitExceeded()
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/dtq/tasks.py", line 9, in sleep
time.sleep(wait)
File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
!docker exec distributed-task-queue-worker-2 tail -14 /opt/celery.log
[2024-02-22 21:00:49,750: ERROR/MainProcess] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:49,750: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(1)
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(1,)
"""
[2024-02-22 21:00:49,751: ERROR/MainProcess] Hard time limit (1s) exceeded for dtq.tasks.sleep[ce748246-66ca-4821-b876-9f04ef1b6f04]
[2024-02-22 21:00:49,869: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:7 exited with 'signal 9 (SIGKILL)'
Note worker process which triggered SoftTimeLimitExceeded is still alive. This is not true for worker process which triggered TimeLimitExceeded:
!docker exec distributed-task-queue-worker-1 ps aux | grep celeryd
root 1 4.0 0.9 46248 38424 ? Ss 21:00 0:01 [celeryd: celery@8f52c2c55c86:MainProcess] -active- (-A dtq.app worker --concurrency 2 -Q celery -l INFO -f celery.log)
root 8 0.5 0.8 45972 35340 ? S 21:00 0:00 [celeryd: celery@8f52c2c55c86:ForkPoolWorker-1]
root 9 0.0 0.8 43484 33168 ? S 21:00 0:00 [celeryd: celery@8f52c2c55c86:ForkPoolWorker-2]
!docker exec distributed-task-queue-worker-2 ps aux | grep celeryd
root 1 4.0 0.9 46504 38576 ? Ss 21:00 0:01 [celeryd: celery@4d44321eb801:MainProcess] -active- (-A dtq.app worker --concurrency 2 -Q celery -l INFO -f celery.log)
root 8 0.2 0.8 45976 35216 ? S 21:00 0:00 [celeryd: celery@4d44321eb801:ForkPoolWorker-2]
root 9 0.0 0.8 46508 35064 ? S 21:00 0:00 [celeryd: celery@4d44321eb801:ForkPoolWorker-3]
Workflows#
Task workflows also seems to be very interesting and feature rich. Recall that tasks execute asynchronously. To execute tasks in sequence (i.e. executing a child task once a parent task completes) we can use chains which can be implemented by using the pipe operator | on individual tasks:
from dtq.tasks import random_fail, sleep
chain = (
random_fail.si(prob=1.0)
| sleep.si(wait=3, return_value=0)
| sleep.si(wait=3, return_value=1)
)
result = chain.delay()
while not result.ready():
continue
try:
print(result.parent.parent.id)
print(result.parent.id)
print(result.id)
print(result.get())
except Exception as e:
print(e)
dbcd58f6-6a6a-42c1-ace5-a5d59db6dc3d
d0848984-35c7-4775-9fd7-cb41358851db
1eed767d-d588-4f99-ae76-d8be7fcafcd8
division by zero
Failure cascades to the succeeding tasks:
r = requests.get(f"http://localhost:8000/results/{result.parent.parent.id}")
r.json()
{'task_id': 'dbcd58f6-6a6a-42c1-ace5-a5d59db6dc3d',
'status': 'FAILURE',
'successful': False,
'result': 'division by zero',
'args': [],
'kwargs': {'prob': 1.0},
'date_done': '2024-02-22T21:00:56.204850'}
r = requests.get(f"http://localhost:8000/results/{result.parent.id}")
r.json()
{'task_id': 'd0848984-35c7-4775-9fd7-cb41358851db',
'status': 'FAILURE',
'successful': False,
'result': 'division by zero',
'args': [],
'kwargs': {'wait': 3, 'return_value': 0},
'date_done': '2024-02-22T21:00:56.208908'}
r = requests.get(f"http://localhost:8000/results/{result.id}")
r.json()
{'task_id': '1eed767d-d588-4f99-ae76-d8be7fcafcd8',
'status': 'FAILURE',
'successful': False,
'result': 'division by zero',
'args': [],
'kwargs': {'wait': 3, 'return_value': 1},
'date_done': '2024-02-22T21:00:56.208134'}
Features for parallel task execution and joining parallel task results also exist. Error callbacks for chains is particularly interesting as it helps separating task logic from error handling code (e.g. logging error trace). See the docs.
Fig. 87 A complex graph structure built from chains and groups (23 tasks in total). Source#
Signals#
Several kinds of events trigger signals, you can connect to these signals to perform actions as they trigger. Here we define a worker_init signal that is dispatched before the worker is started, and a task-failure signal dispatched when a task fails. A logger has to be set up since the celery logger does not exist yet at this point.
Show code cell source
!pygmentize -g ./distributed-task-queue/dtq/signals.py
# pylint: disable=unused-argument
import logging
import time
from celery.exceptions import WorkerShutdown
from celery.signals import task_failure, worker_init
from celery.utils.log import get_logger
@worker_init.connect
def setup_workers(sender=None, conf=None, **kwargs):
logging.basicConfig(
filename="worker_init.log",
filemode="w",
format="[%(asctime)s: %(levelname)s/%(name)s] %(message)s",
level=logging.INFO,
)
logger = logging.getLogger("WorkerInit")
try:
logger.info("Starting worker...")
time.sleep(12.0)
logger.info("Worker started successfully.")
except Exception as exc:
logger.error(exc, exc_info=True)
logger.critical("Failed to start worker. Shutting down...")
raise WorkerShutdown() from exc
@task_failure.connect
def task_failure_notifier(sender=None, task_id=None, **kwargs):
logger = get_logger("celery")
message = (
f"[task_failure_notifier]: Task {sender.name} failed successfully!"
)
logger.error(message)
Defining a worker_init signal can be useful when we want to dynamic initialization before the worker picks up tasks. Pinging some external dependency such as a background prediction server can be done here. Or a worker can shut down whenever an exception is encountered during initialization.
!docker exec distributed-task-queue-worker-1 cat worker_init.log
[2024-02-22 21:00:26,091: INFO/WorkerInit] Starting worker...
[2024-02-22 21:00:38,105: INFO/WorkerInit] Worker started successfully.
The task_failure_notifier simply logs a message in the event of a failure.
!docker exec distributed-task-queue-worker-1 cat celery.log | grep task_failure_notifier
[2024-02-22 21:00:43,255: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.random_fail failed successfully!
[2024-02-22 21:00:48,624: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:56,209: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.random_fail failed successfully!
Fig. 88 Notice that worker logging starts after worker initialization. Task failure notifier occurs exactly when a failure occurs.#
Broker config#
Here we look at configuring RabbitMQ for our use-case of having long-running tasks. RabbitMQ has a default timeout of 30 mins before a received task is acknowledged. Otherwise, it triggers a PreconditionFailed error redelivering the task to another worker. RabbitMQ also closes the channel for this worker, eventually killing all of our workers. We modify this behavior and instead rely on soft and hard time limits to ensure that workers are not hanging.
Fig. 89 Sleep task with wait time exceeding 1800s triggers PRECONDITION_FAILED. RabbitMQ closes the connection.#
To fix this we created the following file:
# rabbitmq/rabbitmq.conf
consumer_timeout = 31622400000
This is in milliseconds which is equivalent to roughly 1 year. Then, the compose file is modified as follows. The change should be visible in the Queues/Consumers tab of the RabbitMQ management console.
# docker-compose.yaml (further-celery)
version: "3"
services:
...
rabbitmq:
image: rabbitmq:3.12.0-management
ports:
- "5672:5672"
- "15672:15672"
volumes:
- ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
...
Appendix: Task routing#
We can also implement task routing. This can be useful, for example, if we want specific machines (e.g. an expensive instance with a GPU) to focus on specific tasks. Note that our implementation of ActiveTask model hides routing keys from tasks. The configuration can be modified by defining another queue called other:
# ./dtq/app.py (further-celery)
...
CELERY_CONFIG = {
"task_acks_late": True,
"worker_prefetch_multiplier": 1,
"task_default_priority": 0,
"task_queue_max_priority": 10,
"task_create_missing_queues": True, # (!)
"result_expires": None,
"result_extended": True,
}
app.conf.update(CELERY_CONFIG)
app.conf.task_routes = { # (!)
"dtq.tasks.*": {"queue": "celery"},
"dtq.other_tasks.*": {"queue": "other"}
}
...
This automatically creates queues which can be bad. But this is the simplest way to do this.
Routing tasks based on their module path to certain queues is defined in task_routes above.
Then, the specific worker for that queue is started (along with other arguments) using the -Q flag.
Here we start a worker on the new queue:
# ./docker-compose.yml
version: "3"
services:
...
other-worker:
build:
dockerfile: Dockerfile.worker
depends_on:
- redis
- rabbitmq
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_PORT=5672
deploy:
replicas: 1
command: [
"-A", "dtq.app", "worker", "-n", "other@%h", "--concurrency", "1", "-Q", "other",
"-l", "INFO", "-f", "celery.log"
]
...
Our new task:
from dtq.other_tasks import hello
hello??
Signature: hello()
Call signature: hello(*a, **kw)
Type: PromiseProxy
String form: <@task: dtq.other_tasks.hello of distributed-task-queue at 0x131b33e80>
File: ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/other_tasks.py
Source:
@app.task
def hello():
return "Hello, from other queue."
Class docstring:
Task base class.
Note:
When called tasks apply the :meth:`run` method. This method must
be defined by all tasks (that is unless the :meth:`__call__` method
is overridden).
Recall that we have 4 workers on the celery queue and 1 worker on the other queue.
The other tasks are not distributed to the default workers:
for i in range(4):
hello.delay()
time.sleep(1.0)
Similarly, when two tasks are pending in the default queue, the other worker does not pick up:
for i in range(6):
sleep.delay(5)
time.sleep(6.5)
Overriding task routes. Configured task routes can be overridden using the queue parameter:
# swapping out the queues
# ⚠⚠⚠ Typo in queue name creates a new queue (without workers)! Fix: global vars
hello.apply_async(queue="celery")
sleep.apply_async(kwargs=dict(wait=3), queue="other")
Show code cell output
<AsyncResult: e7b9a385-d621-40e9-9a60-b8ad7c72a897>
Fig. 90 Workers only pick up tasks routed to their configured queues (even when free). This is overridden in the last two tasks where the queues are swapped.#
Remark. Multiple queues can be used to A/B test different code or ML model versions. Or do things like canary deployment. The task API can be internally modified to dynamically route to certain queues (e.g. based on argument values).
Appendix: Periodic tasks#
Periodic schedules using celery beat only need to be added in the application code:
# ./dtq/app.py
from celery.schedules import crontab
...
app.conf.beat_schedule = {
"random-fail-every-10-seconds": {
"task": "dtq.tasks.random_fail",
"schedule": 10,
"args": (),
},
"sleep-every-1-minute": {
"task": "dtq.tasks.sleep",
"schedule": crontab(),
"args": (1, 0),
"options": {"priority": 3},
},
}
Running the schedule:
celery -A dtq.app beat
Fig. 91 Tasks scheduled using celery beat.#
Fig. 92 Celery beat triggers the task execution without passing through a queue.#
!docker compose -f ./distributed-task-queue/docker-compose.yml down
Show code cell output
?25l[+] Running 0/0
⠋ Container distributed-task-queue-other-worker-1 Stopping 0.1s
⠋ Container distributed-task-queue-flower-1 Stopping 0.1s
⠋ Container distributed-task-queue-worker-1 Stopping 0.1s
⠋ Container distributed-task-queue-api-1 Stopping 0.1s
⠋ Container distributed-task-queue-worker-2 Stopping 0.1s
?25h?25l[+] Running 0/5
⠙ Container distributed-task-queue-other-worker-1 Stopping 0.2s
⠙ Container distributed-task-queue-flower-1 Stopping 0.2s
⠙ Container distributed-task-queue-worker-1 Stopping 0.2s
⠙ Container distributed-task-queue-api-1 Stopping 0.2s
⠙ Container distributed-task-queue-worker-2 Stopping 0.2s
?25h?25l[+] Running 0/5
⠹ Container distributed-task-queue-other-worker-1 Stopping 0.3s
⠹ Container distributed-task-queue-flower-1 Stopping 0.3s
⠹ Container distributed-task-queue-worker-1 Stopping 0.3s
⠹ Container distributed-task-queue-api-1 Stopping 0.3s
⠹ Container distributed-task-queue-worker-2 Stopping 0.3s
?25h?25l[+] Running 1/5
⠸ Container distributed-task-queue-other-worker-1 Stopping 0.4s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠸ Container distributed-task-queue-worker-1 Stopping 0.4s
⠸ Container distributed-task-queue-api-1 Stopping 0.4s
⠸ Container distributed-task-queue-worker-2 Stopping 0.4s
?25h?25l[+] Running 1/5
⠼ Container distributed-task-queue-other-worker-1 Stopping 0.5s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠼ Container distributed-task-queue-worker-1 Stopping 0.5s
⠼ Container distributed-task-queue-api-1 Stopping 0.5s
⠼ Container distributed-task-queue-worker-2 Stopping 0.5s
?25h?25l[+] Running 2/5
⠴ Container distributed-task-queue-other-worker-1 Stopping 0.6s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠴ Container distributed-task-queue-worker-1 Stopping 0.6s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠴ Container distributed-task-queue-worker-2 Stopping 0.6s
?25h?25l[+] Running 2/5
⠦ Container distributed-task-queue-other-worker-1 Stopping 0.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠦ Container distributed-task-queue-worker-1 Stopping 0.7s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠦ Container distributed-task-queue-worker-2 Stopping 0.7s
?25h?25l[+] Running 2/5
⠧ Container distributed-task-queue-other-worker-1 Stopping 0.8s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠧ Container distributed-task-queue-worker-1 Stopping 0.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠧ Container distributed-task-queue-worker-2 Stopping 0.8s
?25h?25l[+] Running 2/5
⠇ Container distributed-task-queue-other-worker-1 Stopping 0.9s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠇ Container distributed-task-queue-worker-1 Stopping 0.9s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠇ Container distributed-task-queue-worker-2 Stopping 0.9s
?25h?25l[+] Running 2/5
⠏ Container distributed-task-queue-other-worker-1 Stopping 1.0s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠏ Container distributed-task-queue-worker-1 Stopping 1.0s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠏ Container distributed-task-queue-worker-2 Stopping 1.0s
?25h?25l[+] Running 2/5
⠋ Container distributed-task-queue-other-worker-1 Stopping 1.1s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠋ Container distributed-task-queue-worker-1 Stopping 1.1s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠋ Container distributed-task-queue-worker-2 Stopping 1.1s
?25h?25l[+] Running 2/5
⠙ Container distributed-task-queue-other-worker-1 Stopping 1.2s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠙ Container distributed-task-queue-worker-1 Stopping 1.2s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠙ Container distributed-task-queue-worker-2 Stopping 1.2s
?25h?25l[+] Running 2/5
⠹ Container distributed-task-queue-other-worker-1 Stopping 1.3s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠹ Container distributed-task-queue-worker-1 Stopping 1.3s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠹ Container distributed-task-queue-worker-2 Stopping 1.3s
?25h?25l[+] Running 2/5
⠸ Container distributed-task-queue-other-worker-1 Stopping 1.4s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠸ Container distributed-task-queue-worker-1 Stopping 1.4s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠸ Container distributed-task-queue-worker-2 Stopping 1.4s
?25h?25l[+] Running 2/5
⠼ Container distributed-task-queue-other-worker-1 Stopping 1.5s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠼ Container distributed-task-queue-worker-1 Stopping 1.5s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠼ Container distributed-task-queue-worker-2 Stopping 1.5s
?25h?25l[+] Running 2/5
⠴ Container distributed-task-queue-other-worker-1 Stopping 1.6s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠴ Container distributed-task-queue-worker-1 Stopping 1.6s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠴ Container distributed-task-queue-worker-2 Stopping 1.6s
?25h?25l[+] Running 2/5
⠦ Container distributed-task-queue-other-worker-1 Stopping 1.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠦ Container distributed-task-queue-worker-1 Stopping 1.7s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠦ Container distributed-task-queue-worker-2 Stopping 1.7s
?25h?25l[+] Running 2/5
⠧ Container distributed-task-queue-other-worker-1 Stopping 1.8s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠧ Container distributed-task-queue-worker-1 Stopping 1.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠧ Container distributed-task-queue-worker-2 Stopping 1.8s
?25h?25l[+] Running 2/5
⠇ Container distributed-task-queue-other-worker-1 Stopping 1.9s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠇ Container distributed-task-queue-worker-1 Stopping 1.9s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠇ Container distributed-task-queue-worker-2 Stopping 1.9s
?25h?25l[+] Running 2/5
⠏ Container distributed-task-queue-other-worker-1 Stopping 2.0s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠏ Container distributed-task-queue-worker-1 Stopping 2.0s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠏ Container distributed-task-queue-worker-2 Stopping 2.0s
?25h?25l[+] Running 2/5
⠋ Container distributed-task-queue-other-worker-1 Stopping 2.1s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠋ Container distributed-task-queue-worker-1 Stopping 2.1s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠋ Container distributed-task-queue-worker-2 Stopping 2.1s
?25h?25l[+] Running 2/5
⠙ Container distributed-task-queue-other-worker-1 Stopping 2.2s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠙ Container distributed-task-queue-worker-1 Stopping 2.2s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠙ Container distributed-task-queue-worker-2 Stopping 2.2s
?25h?25l[+] Running 2/5
⠹ Container distributed-task-queue-other-worker-1 Stopping 2.3s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠹ Container distributed-task-queue-worker-1 Stopping 2.3s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠹ Container distributed-task-queue-worker-2 Stopping 2.3s
?25h?25l[+] Running 2/5
⠸ Container distributed-task-queue-other-worker-1 Stopping 2.4s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠸ Container distributed-task-queue-worker-1 Stopping 2.4s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠸ Container distributed-task-queue-worker-2 Stopping 2.4s
?25h?25l[+] Running 2/5
⠼ Container distributed-task-queue-other-worker-1 Stopping 2.5s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠼ Container distributed-task-queue-worker-1 Stopping 2.5s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠼ Container distributed-task-queue-worker-2 Stopping 2.5s
?25h?25l[+] Running 2/5
⠴ Container distributed-task-queue-other-worker-1 Stopping 2.6s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠴ Container distributed-task-queue-worker-1 Stopping 2.6s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠴ Container distributed-task-queue-worker-2 Stopping 2.6s
?25h?25l[+] Running 2/5
⠦ Container distributed-task-queue-other-worker-1 Stopping 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠦ Container distributed-task-queue-worker-1 Stopping 2.7s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠦ Container distributed-task-queue-worker-2 Stopping 2.7s
?25h?25l[+] Running 3/5
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
⠧ Container distributed-task-queue-worker-1 Stopping 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
⠧ Container distributed-task-queue-worker-2 Stopping 2.8s
?25h?25l[+] Running 5/5
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Stopped 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
?25h?25l[+] Running 5/5
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
⠋ Container distributed-task-queue-redis-1 Stopping 0.1s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 0.1s
?25h?25l[+] Running 5/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
⠙ Container distributed-task-queue-redis-1 Stopping 0.2s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 0.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 0.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 0.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 0.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 0.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 0.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 0.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 0.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 1.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 1.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 1.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 1.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 1.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 1.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 1.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 1.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 1.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 1.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 2.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 2.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 2.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 2.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 2.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 2.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 2.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 2.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 2.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 2.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 3.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 3.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 3.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 3.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 3.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 3.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 3.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 3.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 3.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 3.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 4.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 4.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 4.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 4.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 4.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 4.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 4.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 4.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 4.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 4.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 5.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 5.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 5.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 5.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 5.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 5.5s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠴ Container distributed-task-queue-rabbitmq-1 Stopping 5.6s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠦ Container distributed-task-queue-rabbitmq-1 Stopping 5.7s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠧ Container distributed-task-queue-rabbitmq-1 Stopping 5.8s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠇ Container distributed-task-queue-rabbitmq-1 Stopping 5.9s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠏ Container distributed-task-queue-rabbitmq-1 Stopping 6.0s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠋ Container distributed-task-queue-rabbitmq-1 Stopping 6.1s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠙ Container distributed-task-queue-rabbitmq-1 Stopping 6.2s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠹ Container distributed-task-queue-rabbitmq-1 Stopping 6.3s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠸ Container distributed-task-queue-rabbitmq-1 Stopping 6.4s
?25h?25l[+] Running 6/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
⠼ Container distributed-task-queue-rabbitmq-1 Stopping 6.5s
?25h?25l[+] Running 7/7
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
✔ Container distributed-task-queue-rabbitmq-1 Removed 6.5s
⠋ Network distributed-task-queue_default Removing 0.0s
?25h?25l[+] Running 8/8
✔ Container distributed-task-queue-other-worker-1 Removed 2.7s
✔ Container distributed-task-queue-flower-1 Removed 0.3s
✔ Container distributed-task-queue-worker-1 Removed 2.8s
✔ Container distributed-task-queue-api-1 Removed 0.5s
✔ Container distributed-task-queue-worker-2 Removed 2.8s
✔ Container distributed-task-queue-redis-1 Removed 0.3s
✔ Container distributed-task-queue-rabbitmq-1 Removed 6.5s
✔ Network distributed-task-queue_default Removed 0.1s
?25h
■