Distributed Task Queues#

Status Source Stars


Source: particle1331/distributed-task-queue

Introduction#

Prediction systems (e.g. offline or batch) can be modeled as workers picking up on a task queue. In this notebook, we will look at an implementation of such a system. The workers will be distributed in the sense that each worker runs in its own container. We will use the open-source framework Celery:

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.

An overview of the system is depicted in Fig. 88:

../../_images/04-celery-arch.png

Fig. 88 Four components of the distributed task system. A task server produces standard tasks which are consumed by worker instances. A fault-tolerant message broker distributes the tasks reliably to the workers. The results backend is used for storing results which can be queried asynchronously.#

In particular, we will consider the scenario where we have (1) instances running long-running tasks on two processes such that (2) the instance can terminate any time (e.g. with autoscaling). Any time-consuming or resource-intensive computation that can be offloaded from the main application as background task such as ML model training and batch inference can be orchestrated using this approach.

Celery application#

Any machine with a copy of the Celery application code and a connection to the broker is automatically a part of the distributed system. This is demonstrated below by connecting our local machine to the broker and pushing tasks to the queue. So first we have to install the app library particle1331/distributed-task-queue:

!rm -rf ./task-queue/distributed-task-queue
!git clone --single-branch --branch main https://github.com/particle1331/distributed-task-queue.git ./task-queue/distributed-task-queue > /dev/null
!pip install -U git+https://github.com/particle1331/distributed-task-queue.git@main --force-reinstall > /dev/null
!echo "\nSuccessfully installed: $(pip list | grep distributed-task-queue)/$(wget --header='Accept: application/vnd.github.VERSION.sha' -qO- commit_id  http://api.github.com/repos/particle1331/distributed-task-queue/commits/main | head -c 7)"
Cloning into './task-queue/distributed-task-queue'...
remote: Enumerating objects: 222, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 222 (delta 85), reused 60 (delta 60), pack-reused 114
Receiving objects: 100% (222/222), 1.05 MiB | 1.99 MiB/s, done.
Resolving deltas: 100% (125/125), done.
  Running command git clone --filter=blob:none --quiet https://github.com/particle1331/distributed-task-queue.git /private/var/folders/jq/9vsvd9252_349lsng_5gc_jw0000gn/T/pip-req-build-5c0aoav6

Successfully installed: distributed-task-queue        0.0.0/60efa9b

The code is also copied for building the app locally later:

import os; os.chdir("./task-queue/")
!tree ./distributed-task-queue
./distributed-task-queue
├── Dockerfile.api
├── Dockerfile.worker
├── LICENSE
├── Makefile
├── README.md
├── api
│   ├── __init__.py
│   ├── main.py
│   ├── models.py
│   ├── requirements.txt
│   ├── results.py
│   ├── tasks.py
│   └── utils.py
├── architecture.png
├── docker-compose.yml
├── dtq
│   ├── __init__.py
│   ├── app.py
│   ├── requirements.txt
│   └── tasks.py
├── pyproject.toml
└── setup.cfg

3 directories, 20 files

Finally, we configure the connection:

import os

os.environ["REDIS_HOST"] = "localhost"
os.environ["REDIS_PORT"] = "6379"
os.environ["RABBITMQ_HOST"] = "localhost"
os.environ["RABBITMQ_PORT"] = "5672"

Remark. One caveat is that task producers and deployed workers can have different versions of the code. This can be fixed by containerization and orchestrating rebuilds.

Services#

In this section, we will send tasks to the workers directly using our copy of the dtq code. This contains the configured Celery app which at this point is already connected to the broker. See diagram above (Fig. 88) with the Tasks API replaced by a terminal. The ensemble of services will be built and run using Docker compose:

# ./docker-compose.yml
version: "3"
services:
  api:
    build:
      dockerfile: Dockerfile.api
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - rabbitmq
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_PORT=5672
    command: ["--port", "8000", "--host", "0.0.0.0"]

  worker:
    build:
      dockerfile: Dockerfile.worker
    depends_on:
      - redis
      - rabbitmq
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_PORT=5672
    deploy:
      replicas: 2
    command: ["-A", "dtq.app", "worker", "--concurrency", "2", "-l", "INFO"]

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

  rabbitmq:
    image: rabbitmq:3.12.0-management
    ports:
      - "5672:5672"
      - "15672:15672"

  flower:
    image: mher/flower
    ports:
      - "5555:5555"
    command:
      - "celery"
      - "--broker=amqp://guest@rabbitmq:5672//"
      - "flower"
      - "--broker_api=http://guest:guest@rabbitmq:15672/api//"
    depends_on:
      - rabbitmq

Running the stack in the background:

!docker compose -f ./distributed-task-queue/docker-compose.yml up -d --build
Hide code cell output
?25l[+] Building 0.0s (0/0)                                    docker:desktop-linux
?25h?25l[+] Building 0.0s (0/0)                                    docker:desktop-linux
?25h?25l[+] Building 0.0s (0/2)                                    docker:desktop-linux
?25h?25l[+] Building 0.2s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.1s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.3s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.3s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.5s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.4s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.6s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.6s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.8s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.7s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.9s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.9s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.1s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.0s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.2s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.2s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.4s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.3s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.5s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.5s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.7s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.6s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 1.8s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.8s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.0s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  1.9s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.1s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.1s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.3s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.2s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.4s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.4s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.6s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.5s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.7s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.7s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 2.9s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  2.8s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.0s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.0s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.2s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.1s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.3s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.3s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.5s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.4s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.6s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.6s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.8s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  3.7s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 3.9s (14/20)                                  docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [api internal] load metadata for docker.io/library/python:3.9.15-slim  3.8s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [worker internal] load build context                                   0.0s
 => => transferring context: 1.21kB                                        0.0s
 => [api 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:ffc6cb648d  0.0s
 => [api internal] load build context                                      0.0s
 => => transferring context: 7.08kB                                        0.0s
 => CACHED [worker 2/7] WORKDIR /opt                                       0.0s
 => CACHED [worker 3/7] COPY ./dtq/requirements.txt .                      0.0s
 => CACHED [worker 4/7] RUN pip install -r requirements.txt                0.0s
 => CACHED [worker 5/7] RUN pip install setproctitle                       0.0s
 => CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps    0.0s
 => CACHED [worker 7/7] COPY /dtq/ ./dtq/                                  0.0s
?25h?25l[+] Building 4.0s (22/22)                                  docker:desktop-linux
 => [api internal] load build context                                      0.0s
 => => transferring context: 7.08kB                                        0.0s
 => CACHED [api 2/7] WORKDIR /opt                                          0.0s
 => CACHED [worker 3/7] COPY ./dtq/requirements.txt .                      0.0s
 => CACHED [worker 4/7] RUN pip install -r requirements.txt                0.0s
 => CACHED [worker 5/7] RUN pip install setproctitle                       0.0s
 => CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps    0.0s
 => CACHED [worker 7/7] COPY /dtq/ ./dtq/                                  0.0s
 => [worker] exporting to image                                            0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:15aacea1a3c26b4263b28bf091223528cc5e3a4741206  0.0s
 => => naming to docker.io/library/distributed-task-queue-worker           0.0s
 => CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt      0.0s
 => CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt      0.0s
 => CACHED [api 5/8] RUN pip install -r dtq-requirements.txt               0.0s
 => CACHED [api 6/8] RUN pip install -r api-requirements.txt               0.0s
 => CACHED [api 7/8] COPY ./dtq ./dtq                                      0.0s
 => CACHED [api 8/8] COPY ./api .                                          0.0s
 => [api] exporting to image                                               0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:233ecd18c4b7096d01414d847d731453fa972188b0bc2  0.0s
 => => naming to docker.io/library/distributed-task-queue-api              0.0s
?25h?25l[+] Building 4.0s (22/22) FINISHED                         docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [api internal] load metadata for docker.io/library/python:3.9.15-slim  3.8s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [worker internal] load build context                                   0.0s
 => => transferring context: 1.21kB                                        0.0s
 => [api 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:ffc6cb648d  0.0s
 => [api internal] load build context                                      0.0s
 => => transferring context: 7.08kB                                        0.0s
 => CACHED [api 2/7] WORKDIR /opt                                          0.0s
 => CACHED [worker 3/7] COPY ./dtq/requirements.txt .                      0.0s
 => CACHED [worker 4/7] RUN pip install -r requirements.txt                0.0s
 => CACHED [worker 5/7] RUN pip install setproctitle                       0.0s
 => CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps    0.0s
 => CACHED [worker 7/7] COPY /dtq/ ./dtq/                                  0.0s
 => [worker] exporting to image                                            0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:15aacea1a3c26b4263b28bf091223528cc5e3a4741206  0.0s
 => => naming to docker.io/library/distributed-task-queue-worker           0.0s
 => CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt      0.0s
 => CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt      0.0s
 => CACHED [api 5/8] RUN pip install -r dtq-requirements.txt               0.0s
 => CACHED [api 6/8] RUN pip install -r api-requirements.txt               0.0s
 => CACHED [api 7/8] COPY ./dtq ./dtq                                      0.0s
 => CACHED [api 8/8] COPY ./api .                                          0.0s
 => [api] exporting to image                                               0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:233ecd18c4b7096d01414d847d731453fa972188b0bc2  0.0s
 => => naming to docker.io/library/distributed-task-queue-api              0.0s
?25h?25l[+] Running 0/0
 ⠋ Network distributed-task-queue_default  Creating                        0.0s 
?25h?25l[+] Running 1/1
  Network distributed-task-queue_default       Created                    0.1s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Creating                   0.1s 
 ⠋ Container distributed-task-queue-redis-1     Creating                   0.1s 
?25h?25l[+] Running 3/3
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
 ⠋ Container distributed-task-queue-flower-1    Creating                   0.1s 
 ⠋ Container distributed-task-queue-worker-2    Creating                   0.1s 
 ⠋ Container distributed-task-queue-worker-1    Creating                   0.1s 
 ⠋ Container distributed-task-queue-api-1       Creating                   0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Created                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Created                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Created                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Created                    0.1s 
  Container distributed-task-queue-worker-2    Started                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Created                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Started                    0.1s 
  Container distributed-task-queue-worker-2    Started                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Started                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Started                    0.1s 
  Container distributed-task-queue-worker-2    Started                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Started                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Started                    0.1s 
  Container distributed-task-queue-worker-2    Started                    0.1s 
  Container distributed-task-queue-worker-1    Created                    0.1s 
  Container distributed-task-queue-api-1       Started                    0.1s 
?25h?25l[+] Running 7/7
  Network distributed-task-queue_default       Created                    0.1s 
  Container distributed-task-queue-rabbitmq-1  Started                    0.1s 
  Container distributed-task-queue-redis-1     Started                    0.1s 
  Container distributed-task-queue-flower-1    Started                    0.1s 
  Container distributed-task-queue-worker-2    Started                    0.1s 
  Container distributed-task-queue-worker-1    Started                    0.1s 
  Container distributed-task-queue-api-1       Started                    0.1s 
?25h
!docker compose -f ./distributed-task-queue/docker-compose.yml ps | sort
NAME                                IMAGE                           COMMAND                  SERVICE    CREATED         STATUS                  PORTS
distributed-task-queue-api-1        distributed-task-queue-api      "uvicorn main:app --…"   api        3 seconds ago   Up Less than a second   0.0.0.0:8000->8000/tcp
distributed-task-queue-flower-1     mher/flower                     "celery --broker=amq…"   flower     3 seconds ago   Up Less than a second   0.0.0.0:5555->5555/tcp
distributed-task-queue-rabbitmq-1   rabbitmq:3.12.0-management      "docker-entrypoint.s…"   rabbitmq   3 seconds ago   Up 1 second             4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp
distributed-task-queue-redis-1      redis:latest                    "docker-entrypoint.s…"   redis      3 seconds ago   Up 1 second             0.0.0.0:6379->6379/tcp
distributed-task-queue-worker-1     distributed-task-queue-worker   "celery -A dtq.app w…"   worker     3 seconds ago   Up Less than a second   
distributed-task-queue-worker-2     distributed-task-queue-worker   "celery -A dtq.app w…"   worker     3 seconds ago   Up Less than a second   

Before proceeding we check whether all system components are available:

import time
from redis import StrictRedis
from dtq.app import app

def healthcheck(app):
    """Ping and wait up to one second for responses."""
    r = StrictRedis(
        host=os.environ["REDIS_HOST"], 
        port=os.environ["REDIS_PORT"],
        socket_timeout=3
    )
    workers = app.control.inspect().ping()  # True if at least one response
    return bool(workers) and r.ping()


MAX_RETRIES = 5
for i in range(MAX_RETRIES):
    try:
        if healthcheck(app):
            print("Celery app connection is established.")
            time.sleep(1.0)
            break
    except Exception as e:
        if i == MAX_RETRIES - 1:
            raise Exception("Celery results backend or workers unavailable.")
Celery app connection is established.

Workers#

Building a worker simply involves providing a copy of the Celery application code:

# ./Dockerfile.worker
FROM python:3.9.15-slim

WORKDIR /opt

COPY ./dtq/requirements.txt .

RUN pip install -r requirements.txt
RUN pip install setproctitle
RUN apt-get update && apt-get install -y procps

COPY /dtq/ ./dtq/

ENTRYPOINT ["celery"]

From the compose file, two replicas of the worker service is built. Each worker runs the ff. command at startup:

celery -A dtq.app worker --concurrency 2 -l INFO

This runs two worker processes for each worker with log level INFO. It is recommended to set concurrency equal to the CPU count of the worker machine. Here we set to 2 for simplicity. Distinguishing between the worker and its processes is important when reading the documentation.

Installing setproctitle allows us to see the type of process in ps listings. The worker runs two child processes and a main worker process:

!docker exec distributed-task-queue-worker-1 ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  8.9  0.9  45876 38032 ?        Ss   20:56   0:01 [celeryd: celery@364128f8e977:MainProcess] -active- (-A dtq.app worker --concurrency 2 -l INFO)
root         8  0.0  0.8  43484 32996 ?        S    20:56   0:00 [celeryd: celery@364128f8e977:ForkPoolWorker-1]
root         9  0.0  0.8  43488 32868 ?        S    20:56   0:00 [celeryd: celery@364128f8e977:ForkPoolWorker-2]
root        10  0.0  0.0   8336  2688 ?        Rs   20:57   0:00 ps aux

Tasks#

Tasks are the building blocks of Celery applications. Every task class has a unique name, and this name is referenced in broker messages so a worker can find the right function to execute. Ideally task functions should be idempotent, i.e. can be called multiple times with the same arguments with no side-effects.

from dtq.tasks import sleep

??sleep
Signature:       sleep(wait=0, return_value=1)
Call signature:  sleep(*a, **kw)
Type:            PromiseProxy
String form:     <@task: dtq.tasks.sleep of distributed-task-queue at 0x10554cbb0>
File:            ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:         
@app.task
def sleep(wait=0, return_value=1):
    time.sleep(wait)
    return return_value
Class docstring:
Task base class.

Note:
    When called tasks apply the :meth:`run` method.  This method must
    be defined by all tasks (that is unless the :meth:`__call__` method
    is overridden).

Tasks pushed to the queue will have not have results until it has been fully processed by a worker, so its return value is an celery.result.AsyncResult object:

result = sleep.delay(wait=3)
result
<AsyncResult: ada4cc70-9b55-4da5-a8e3-4385ce98b8f6>

Getting the actual result. This will block execution until result is ready:

import time

start = time.time()
print(result.get())
print(f"Time elapsed: {time.time() - start:.3f}s")
1
Time elapsed: 3.046s

A task that has an exception is acknowledged (i.e. removed from the queue) with FAILURE state:

from dtq.tasks import random_fail

while True:
    result = random_fail.delay()
    try:
        result.get()
    except ZeroDivisionError:
        break

result.id, result.status, result.result
('c8e5beaa-8da6-442e-8c94-2276d9c6051a',
 'FAILURE',
 ZeroDivisionError('division by zero'))
??random_fail
Signature:       random_fail()
Call signature:  random_fail(*a, **kw)
Type:            PromiseProxy
String form:     <@task: dtq.tasks.random_fail of distributed-task-queue at 0x10554cbb0>
File:            ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:         
@app.task
def random_fail():
    time.sleep(0.1)
    x = random.choice([0, 1])
    return 1 / x
Class docstring:
Task base class.

Note:
    When called tasks apply the :meth:`run` method.  This method must
    be defined by all tasks (that is unless the :meth:`__call__` method
    is overridden).

Flower UI#

Flower UI can be used to visualize task results. Flower has an extensive API for sending and querying tasks and changing worker configurations, but as mentioned above we will develop our own tasks and results API for our use-case.

Hide code cell content
from selenium import webdriver
TAKE_SCREENSHOT = True

def take_screenshot(url: str, zoom: str, savename: str, wait=4.0):
    """Automate taking screenshot of our services' UI / docs. This requires chromedriver:
    https://www.geeksforgeeks.org/how-to-run-selenium-running-test-on-chrome-using-webdriver/"""
    
    if os.path.isfile("/usr/local/bin/chromedriver") and TAKE_SCREENSHOT:
        options = webdriver.ChromeOptions()
        options.add_argument("--headless")
        options.add_argument("--window-size=1200,660")

        driver = webdriver.Chrome(options=options)
        driver.get(url)
        time.sleep(wait)
        driver.refresh()
        viewport_height = int(driver.execute_script("return window.innerHeight"))
        viewport_width  = int(driver.execute_script("return window.innerWidth"))
        new_viewport_height = int(viewport_height * (100 / zoom))
        new_viewport_width  = int(viewport_width  * (100 / zoom))
        driver.set_window_size(new_viewport_width, new_viewport_height)
        driver.save_screenshot(f"{savename}.png")
        driver.quit()


# Getting failed task example
FLOWER_URL = "http://localhost:5555"
take_screenshot(FLOWER_URL + "/tasks", 67, "./img/04-celery-tasks", wait=3)
take_screenshot(FLOWER_URL + f"/task/{result.id}", 75, "./img/04-celery-stacktrace")
take_screenshot(FLOWER_URL + "/workers", 100, "./img/04-celery-workers-tasks")
../../_images/04-celery-tasks.png

Fig. 89 Tasks shown in the Flower UI. Observe that tasks are automatically distributed to each worker:#

../../_images/04-celery-stacktrace.png

Fig. 90 Stack trace of failed task is also included in the UI.#

../../_images/04-celery-workers-tasks.png

Fig. 91 Counts for successful and failed tasks by worker are shown in Flower.#

Celery config#

Depending on the use-case and program behavior that you want to achieve, you will have to spend some time reading the docs to get the correct config variables and testing if the settings achieve the desired behavior. In this section, we will focus on the contents of CELERY_CONFIG. Each key of this dictionary is a configuration variable. All configuration variables can be accessed as a dict object in app.conf.

# ./dtq/app.py
import os

from celery import Celery

REDIS_HOST = os.environ["REDIS_HOST"]
REDIS_PORT = os.environ["REDIS_PORT"]
RABBITMQ_HOST = os.environ["RABBITMQ_HOST"]
RABBITMQ_PORT = os.environ["RABBITMQ_PORT"]

PROJECT_NAME = "distributed-task-queue"
BACKEND_URL = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
BROKER_URL = f"pyamqp://guest:guest@{RABBITMQ_HOST}:{RABBITMQ_PORT}//"

app = Celery(
    PROJECT_NAME,
    broker=BROKER_URL,
    backend=BACKEND_URL,
    include=("dtq.tasks",),
)


CELERY_CONFIG = {
    "task_acks_late": True,
    "worker_prefetch_multiplier": 1,
    "task_default_priority": 0,
    "task_queue_max_priority": 10,
    "task_create_missing_queues": False,
    "result_expires": None,
    "result_extended": True,
}

app.conf.update(CELERY_CONFIG)

Remark. Results expire in 1 day by default, so we set 'results_expires' to None. This is bad in practice, as we can easily run out of storage. This highlights the fact that Celery config has a lot of sharp corners. Similarly, the broker needs specific configuration. There is really no way around it but reading the docs and experimenting certain scenarios.

Acknowledgement#

A task message is not removed from the queue until it has been acknowledged by a worker. A worker can reserve many messages in advance, and even if the worker is killed (e.g. when autoscaling) the message may be redelivered to another worker. The default behavior is that a task is acknowledged once a worker starts processing a task:

Since the worker cannot detect if your tasks are idempotent, the default behavior is to acknowledge the message in advance, just before it’s executed, so that a task invocation that already started is never executed again.

For our use-case, we set task_acks_late to True so that received tasks are not acknowledged until it is completed. This is nice since it allows tasks to remain in the queue so that other workers can pick it up whenever one worker is terminated. Messages in the broker which are not yet received by a worker are called ready.

Remark. Note that worker shutdown is different from simply killing a worker process.

tasks = []
for i in range(6):
    task = sleep.delay(6.0)
    tasks.append(task)
../../_images/04-celery-unacked.png

Fig. 92 Message counts in the broker tab showing the count of unacklowleged tasks in the queue. This shows 2 ready and 4 unacknowledged messages. This makes sense since we are queueing 6 tasks, while we only have 4 worker processes.#

Remark. Messages equals the sum of unacked and ready (Fig. 92). This is not true by default. We configured Celery so that a task is acknowledged only after a worker is done processing it. The default behavior is that a task is acknowledged once a worker starts processing a task:

Since the worker cannot detect if your tasks are idempotent, the default behavior is to acknowledge the message in advance, just before it’s executed, so that a task invocation that already started is never executed again.


Let us demonstrate redelivery of tasks when a worker is terminated:

import pandas as pd
from dtq.app import app

def get_active_tasks() -> pd.DataFrame:
    time.sleep(0.5)
    inspector = app.control.inspect()
    active = inspector.active() or {}
    output = []
    for worker in active.keys():
        for metadata in active[worker]:
            delivery_info = metadata.pop("delivery_info")
            metadata["task_id"] = metadata.pop("id")
            metadata.update(delivery_info)
            output.append(metadata)

    return pd.DataFrame(output)


result = sleep.delay(wait=1200)
get_active_tasks()
name args kwargs type hostname time_start acknowledged worker_pid task_id exchange routing_key priority redelivered
0 dtq.tasks.sleep [] {'wait': 1200} dtq.tasks.sleep celery@364128f8e977 1.708635e+09 False 9 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 celery 0 False

Shutting down this worker:

active_tasks = get_active_tasks().query(f"task_id=='{result.task_id}'")
worker_id = active_tasks.loc[0, "hostname"].replace("celery@", "")
!docker restart {worker_id}

Task should be picked up by the other worker. Notice redelivered is now True:

get_active_tasks()
name args kwargs type hostname time_start acknowledged worker_pid task_id exchange routing_key priority redelivered
0 dtq.tasks.sleep [] {'wait': 1200} dtq.tasks.sleep celery@bdfaad682753 1.708635e+09 False 9 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 celery 0 True

Revoking tasks#

Tasks can be revoked using app.control.revoke. This sets task status to REVOKED such that any worker will not pick up this task from the queue. But if the task has already started processing, revoking the task will not cancel it:

while len(app.control.ping()) == 0: # See remark below
    time.sleep(1.0)

app.control.revoke(result.task_id)
print(f"Task {result.task_id} revoked.")
print(f"Remaining tasks: {len(get_active_tasks())}")
get_active_tasks()
Task 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 revoked.
Remaining tasks: 1
name args kwargs type hostname time_start acknowledged worker_pid task_id exchange routing_key priority redelivered
0 dtq.tasks.sleep [] {'wait': 1200} dtq.tasks.sleep celery@bdfaad682753 1.708635e+09 False 9 66c4f6df-96bd-4f3b-85ab-3be2427fc7c7 celery 0 True

Restarting the worker. Note that task is not picked up by the other available worker:

while len(app.control.ping()) < 2: # Need always one worker present (i.e. 1 after restarting the other)
    time.sleep(1.0)

active_tasks = get_active_tasks().query(f"task_id=='{result.task_id}'")
worker_id = active_tasks.loc[0, "hostname"].replace("celery@", "")
print(f"Restarting worker celery@{worker_id}...")
!docker restart {worker_id} > /dev/null
print(f"Remaining tasks: {len(get_active_tasks())}")
Restarting worker celery@bdfaad682753...
Remaining tasks: 0

Remark. Note that revoke works by sending a broadcast message to all the workers, the workers then keep a list of revoked tasks in memory. When a worker starts up it will synchronize revoked tasks with other workers in the cluster. If all workers restart (or are terminated), the list of revoked ids will also vanish! ⚠

This is bad for our setup since it only has two workers. Indeed, an earlier version of the code had this bug (Fig. 94). A non-fragile setup is therefore to have at least 3 workers. To check worker availability, we do:

app.control.ping()
[{'celery@364128f8e977': {'ok': 'pong'}},
 {'celery@bdfaad682753': {'ok': 'pong'}}]

Stopping a worker process#

To abort an active task without having to restart or terminate a worker, we can use revoke with terminate=True. Note that this will terminate the worker process not the task itself! This distinction is important — the process may have already started processing another task at the point when the signal is sent. Never call this programmatically! ⚠

while len(app.control.ping()) == 0: # See remark above
    time.sleep(1.0)

result = sleep.delay(wait=300)
print(f"Task {result.id} started.")
app.control.revoke(result.id, terminate=True)
print(f"Task {result.id} revoked with SIGTERM.")
print(f"Remaining tasks: {len(get_active_tasks())}")
Task c1e1324c-19a5-44f3-ae19-e55129ee5e1a started.
Task c1e1324c-19a5-44f3-ae19-e55129ee5e1a revoked with SIGTERM.
Remaining tasks: 0

Remark. The revoke function has an optional signal argument which defaults to 'SIGTERM'. This can be the uppercase name of any signal defined in the signal module from the standard library. This allows time for cleanup. If the process does not stop after a considerable amount of time, then a 'SIGKILL' signal may be sent. But this comes with some complications.


../../_images/04-celery-revoked-tasks.png

Fig. 93 Revoked tasks.#

../../_images/04-celery-revoke-broadcast-failed.png

Fig. 94 Task revoke fails when remote broadcast is done improperly. To summarize, only revoke when there is at least one worker. And be careful when shutting down all workers, as data shared by the workers is lost.#

Task prioritization#

Suppose we have priority levels for our tasks with higher priority tasks to be processed first once a worker is free. The following config keys seek to accomplish this:

CELERY_CONFIG = {
    ...,
    "worker_prefetch_multiplier": 1,
    "task_default_priority": 0,
    "task_queue_max_priority": 10,
    "task_create_missing_queues": False,
    ...
}

The number of reserved messages by each worker process is equal to the prefetch multiplier which defaults to 4. For long-running tasks the network overhead of picking up from the queue is negligible, so setting the prefetch multiplier to 1 is fine. Moreover, this allows messages to be ordered in the queue which is supported by RabbitMQ. Here we have to match the max priority with the correct queue arguments of RabbitMQ (i.e. x-max-priority of 10.) This allows us to set a priority for each task.

The last variable is for raising QueueNotFound when a task is sent to an undefined queue. Without this, client-side logic could send tasks into a queue that is automatically created, but does not have a consuming worker. This can be hard to spot in production.

Sending tasks with priority:

sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)
sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)
sleep.apply_async(kwargs={"wait": 1, "return_value": 0}, priority=0)

sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)
sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)
sleep.apply_async(kwargs={"wait": 1, "return_value": 1}, priority=1)

sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)
sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)
sleep.apply_async(kwargs={"wait": 1, "return_value": 2}, priority=2)

sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3)
sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3)
sleep.apply_async(kwargs={"wait": 1, "return_value": 3}, priority=3);

Here we use apply_async which is just delay but allows us to set task parameters such as priority which as configured defaults to 0. Tasks with higher integer priority value should be picked up first when using a RabbitMQ broker:

../../_images/04-celery-task-priority.png

Fig. 95 Tasks ordered by started. Notice that tasks with priority 0 and 1 start first, because they are called first. But once these tasks block the workers, tasks with higher priority are processed first.#

Tasks and Results API#

Endpoints for creating tasks are exposed to other parts of the business which input specific metadata that it can use as arguments to the request during downstream processing. This also standardizes task creation and retrieval in one place. Also, using an API ensures that all data communicated between services are validated.

# ./docker-compose.yml
version: "3"
services:
  api:
    build:
      dockerfile: Dockerfile.api
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - rabbitmq
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_PORT=5672
    command: ["--port", "8000", "--host", "0.0.0.0"]

    # ... other services

Environmental variables are loaded which identify the broker and backend URLs. The container for this is built as follows. It copies both the celery application code and the API code and installs their requirements. Note that the requirements file of the Celery app is used to install its respective dependencies preventing version drift (i.e. celery appears once in the two requirements files).

# ./Dockerfile.api
FROM python:3.9.15-slim

WORKDIR /opt

COPY ./dtq/requirements.txt dtq-requirements.txt
COPY ./api/requirements.txt api-requirements.txt
RUN pip install -r dtq-requirements.txt
RUN pip install -r api-requirements.txt

COPY ./dtq ./dtq
COPY ./api .

EXPOSE 8000

ENTRYPOINT ["uvicorn", "main:app"]

The main module simply loads tasks are results routers:

# ./api/main.py
import results
import tasks
from fastapi import FastAPI
from fastapi.responses import RedirectResponse

app = FastAPI()
app.include_router(results.router)
app.include_router(tasks.router)


@app.get("/", include_in_schema=False)
async def docs_redirect():
    return RedirectResponse(url="/docs")

Below pending (queued) and active tasks (processing) have their specific schemas:

# ./api/models.py
...

class Task(BaseModel):
    task_id: str
    status: str


class ActiveTask(BaseModel):
    task_id: str
    task: str
    args: list
    kwargs: dict
    worker: str
    time_start: datetime
    acknowledged: bool
    priority: int
    redelivered: bool
    worker_pid: int


class PendingTask(BaseModel):
    task_id: str
    task: str
    retries: int
    timelimit: list
    root_id: str
    parent_id: Optional[str]
    origin: str
    args: list
    kwargs: dict

...

../../_images/04-celery-fastapi-app.png

Fig. 96 FastAPI app port forwarded to localhost:8000.#

Tasks endpoints#

First we look at the first two task endpoints. These push the corresponding tasks to the queue. Here we can add validation for query parameters such as constraining wait to be nonnegative. Data validation is one of the primary benefits of having an API for producing tasks. Note that the we can now set priorities for each task:

# ./api/tasks.py
from fastapi import APIRouter, status
from fastapi.responses import JSONResponse
from models import ActiveTask, PendingTask, Task
from utils import exist_workers, poll_messages

from dtq import tasks
from dtq.app import app as celery_app

router = APIRouter(prefix="/tasks", tags=["tasks"])


@router.post("/sleep", response_model=Task)
async def sleep(wait: float, return_value: int = 1, priority: int = 0) -> str:
    result = tasks.sleep.apply_async(
        kwargs={"wait": wait, "return_value": return_value},
        priority=priority,
    )
    task_id = result.task_id
    return {"task_id": task_id, "status": result.status}


@router.post("/random_fail", response_model=Task)
async def random_fail():
    result = tasks.random_fail.delay()
    task_id = result.task_id
    return {"task_id": task_id, "status": result.status}

...

Trying these out:

import requests

API_URL = "http://localhost:8000"
sleep_response = requests.post(f"{API_URL}/tasks/sleep?wait=5&return_value=1&priority=0")
sleep_response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551', 'status': 'PENDING'}
fail_response = requests.post(f"{API_URL}/tasks/random_fail")
fail_response.json()
{'task_id': '62fc3ab0-3fca-424e-acb3-5376082a7b8b', 'status': 'PENDING'}

Results endpoints#

The results/{task_id} endpoint returns the task result and status. It returns null result when the task is still processing or completed but failed. See the docs for built-in states. Note that Celery assigns the exception for failed tasks in the .result attribute of the AsyncResult object.

# ./api/results.py
from celery.result import AsyncResult
from fastapi import APIRouter
from models import Result

router = APIRouter(prefix="/results", tags=["results"])


@router.get("/{task_id}", response_model=Result)
async def fetch_result(task_id):
    result = AsyncResult(task_id)
    return {
        "task_id": result.task_id,
        "status": result.status,
        "successful": result.successful(),
        "args": result.args,
        "kwargs": result.kwargs,
        "date_done": result.date_done,
        "result": result.result,
    }

The results data model is as follows. Note that result is optional since since a task has delayed result (or None when failed). Recall in the configuration we have results_extended set to True. This is so that task args and kwargs are sent to the results backend. Here result has type str since it can be an exception or some return value (e.g. float or int).

# ./api/models.py (continued)
...

class Result(BaseModel):
    task_id: str
    status: str
    successful: bool
    result: Optional[str]
    args: Optional[list]
    kwargs: Optional[dict]
    date_done: Optional[datetime]

    @validator("result", pre=True)
    def validate_result(cls, value):
        if value is None:
            return None
        return str(value)

Sleep task above still running. The successful field defaults to False while others are null.

task_id = sleep_response.json()["task_id"]
response = requests.get(f"{API_URL}/results/{task_id}")
response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551',
 'status': 'PENDING',
 'successful': False,
 'result': None,
 'args': None,
 'kwargs': None,
 'date_done': None}

Result of the random_fail endpoint:

time.sleep(0.5)
task_id = fail_response.json()["task_id"]
response = requests.get(f"{API_URL}/results/{task_id}")
response.json()
{'task_id': '62fc3ab0-3fca-424e-acb3-5376082a7b8b',
 'status': 'FAILURE',
 'successful': False,
 'result': 'division by zero',
 'args': [],
 'kwargs': {},
 'date_done': '2024-02-22T20:58:38.894050'}

Note result is the exception as string. Getting the result once the long-running task is done:

from celery.result import AsyncResult

task_id = sleep_response.json()["task_id"]
result = AsyncResult(task_id)
while not result.ready():
    continue

response = requests.get(f"http://127.0.0.1:8000/results/{task_id}")
response.json()
{'task_id': '304e87be-449c-47c3-8ecb-9b9a21695551',
 'status': 'SUCCESS',
 'successful': True,
 'result': '1',
 'args': [],
 'kwargs': {'wait': 5.0, 'return_value': 1},
 'date_done': '2024-02-22T20:58:43.771632'}

Pending and active tasks#

The next endpoints list active and pending tasks. Recall our workers are configured to prefetch one task from the queue. A good mental model is that Celery can only see what the workers can see. For example, revoke does not work when there are no workers! Pending tasks therefore have to be queried directly from RabbitMQ using a library such as pika:

# ./api/utils.py
...

def poll_messages(queue="celery"):
    parameters = pika.ConnectionParameters(
        host=os.environ["RABBITMQ_HOST"],
        port=os.environ["RABBITMQ_PORT"],
    )
    conn = pika.BlockingConnection(parameters)
    queue_args = {"x-max-priority": 10}
    channel = conn.channel()
    channel.queue_declare(queue, durable=True, arguments=queue_args)

    messages = []
    while True:
        response = channel.basic_get(queue=queue, auto_ack=False)
        method_frame, properties, _ = response
        if method_frame is None:
            break

        metadata = properties.headers
        fields = [
            "id",
            "task",
            "retries",
            "timelimit",
            "root_id",
            "parent_id",
            "origin",
        ]

        msg = {k: metadata[k] for k in fields}
        msg["task_id"] = msg.pop("id")
        msg["args"] = ast.literal_eval(metadata["argsrepr"])
        msg["kwargs"] = ast.literal_eval(metadata["kwargsrepr"])
        messages.append(msg)

    conn.close()
    return messages

Remark. Queue declare above should match the queue configuration in the main app. Best practice is to have a unified configuration which we import. But here we just hard code.

Getting active tasks (i.e. started by a worker and removed from the queue):

# tasks.py (continued)
...

@router.get("/active")
async def active_tasks() -> list[ActiveTask]:
    inspector = celery_app.control.inspect()
    active = inspector.active() or {}
    result = []
    for worker in active.keys():
        for metadata in active[worker]:
            delivery_info = metadata.pop("delivery_info")
            metadata["task_id"] = metadata.pop("id")
            metadata["task"] = metadata.pop("name")
            metadata["worker"] = metadata.pop("hostname")
            metadata["priority"] = delivery_info["priority"]
            metadata["redelivered"] = delivery_info["redelivered"]
            result.append(metadata)
    return result


@router.get("/pending")
async def pending_tasks(queue: str = "celery") -> list[PendingTask]:
    messages = poll_messages(queue)
    return messages

...

Remark. For querying individual active tasks, we can define an endpoint that uses:

inspector.query_task(task_id)

Queueing up some tasks to try out the endpoints:

for i in range(6):
    requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")

Expecting a list of length four corresponding to four worker processes:

active = requests.get(f"{API_URL}/tasks/active").json()
active
Hide code cell output
[{'task_id': '7b0dee69-aab7-4033-ada7-c9f7f84f2437',
  'task': 'dtq.tasks.sleep',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1},
  'worker': 'celery@bdfaad682753',
  'time_start': '2024-02-22T20:58:43.799137Z',
  'acknowledged': False,
  'priority': 0,
  'redelivered': False,
  'worker_pid': 9},
 {'task_id': '044238f6-277b-4c1d-824c-65529e27ae04',
  'task': 'dtq.tasks.sleep',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1},
  'worker': 'celery@bdfaad682753',
  'time_start': '2024-02-22T20:58:43.839229Z',
  'acknowledged': False,
  'priority': 0,
  'redelivered': False,
  'worker_pid': 8},
 {'task_id': '6cb3d744-6b32-4bf6-94b7-9dd7d6975a33',
  'task': 'dtq.tasks.sleep',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1},
  'worker': 'celery@364128f8e977',
  'time_start': '2024-02-22T20:58:43.818486Z',
  'acknowledged': False,
  'priority': 0,
  'redelivered': False,
  'worker_pid': 10},
 {'task_id': '7fe15dbe-ac70-4382-a89d-b61d2887e30a',
  'task': 'dtq.tasks.sleep',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1},
  'worker': 'celery@364128f8e977',
  'time_start': '2024-02-22T20:58:43.850899Z',
  'acknowledged': False,
  'priority': 0,
  'redelivered': False,
  'worker_pid': 8}]

Expecting a list of length two:

pending = requests.get(f"{API_URL}/tasks/pending").json()
pending
Hide code cell output
[{'task_id': 'a5c75cc8-0113-44cc-979a-146fb9b66c45',
  'task': 'dtq.tasks.sleep',
  'retries': 0,
  'timelimit': [None, None],
  'root_id': 'a5c75cc8-0113-44cc-979a-146fb9b66c45',
  'parent_id': None,
  'origin': 'gen1@0a8d0d78cf08',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1}},
 {'task_id': 'a0cf96cd-61cf-4915-96e2-48d60d5f91a7',
  'task': 'dtq.tasks.sleep',
  'retries': 0,
  'timelimit': [None, None],
  'root_id': 'a0cf96cd-61cf-4915-96e2-48d60d5f91a7',
  'parent_id': None,
  'origin': 'gen1@0a8d0d78cf08',
  'args': [],
  'kwargs': {'wait': 3.0, 'return_value': 1}}]

Total pending tasks#

For our scalable architecture, we imagine horizontally scaling the workers in and out depending on the size of the pending tasks which we abstract using the wait variable. Size of pending tasks depend on how fast the workers are able to process the tasks and the rate and size of incoming tasks. We define the following function:

def get_total_pending_size(queue="celery"):
    pending_size = 0
    response = requests.get(f"{API_URL}/tasks/pending?queue={queue}")
    messages = response.json()
    for m in messages:
        if m["task"] == "dtq.tasks.sleep":
            pending_size += m["kwargs"]["wait"]
    return pending_size

Below we monitor the pending size while we simulate sleep tasks using threading:

Hide code cell source
import threading
import numpy as np

ps = []
ts = []

def tasksim():
    n = 100
    for _ in range(n):
        t = np.random.random() * 4.0
        ts.append(t)
        requests.post(f"{API_URL}/tasks/sleep?wait={t}&return_value=1&priority=0")
        time.sleep(np.random.random())

def monitor():
    while True:
        time.sleep(0.05)
        if response.ok:
            ps.append(get_total_pending_size())
        else:
            continue
        
        if len(ts) == 100:
            break


thread_monitor = threading.Thread(target=monitor)
thread_tasksim = threading.Thread(target=tasksim)

thread_monitor.start()
thread_tasksim.start()

thread_monitor.join()
thread_tasksim.join()

Plotting:

Hide code cell source
import matplotlib.pyplot as plt
from matplotlib_inline import backend_inline
backend_inline.set_matplotlib_formats("svg")

plt.plot(np.arange(len(ps)) * 0.05, ps)
plt.grid(linestyle="dotted")
plt.ylabel("Total task size (pending)")
plt.xlabel("Time (s)");
../../_images/6e72a65156acc0676a82c00b7f98213015d597fca1d92542444560b7708903e8.svg

../../_images/04-celery-rabbit-mq-monitor.png

Fig. 97 Monitoring the RabbitMQ broker in localhost:15672. The dashboard also shows ready, unacked, and total message counts, as well as message rates. Resource usage metrics can be important when load testing.#

Task abort and revoke#

Remote commands do not work without workers. So we have to check if workers exist:

# ./api/utils.py (continued)
...

def exist_workers():
    inspector = celery_app.control.inspect()
    return inspector.ping() is not None

...

The following endpoints use the Celery API which allow for proper handling:

# ./api/tasks.py (continued)
...

@router.post("/revoke/{task_id}")
async def revoke_task(task_id: str) -> JSONResponse:
    if not exist_workers():
        message = f"Task {task_id} revoke failed. No workers found."
        status_code = status.HTTP_400_BAD_REQUEST
    else:
        celery_app.control.revoke(task_id)
        message = f"Task {task_id} revoked."
        status_code = status.HTTP_200_OK

    return JSONResponse({"message": message}, status_code=status_code)


@router.post("/abort/{task_id}")
async def abort_task(task_id: str) -> JSONResponse:
    if not exist_workers():
        message = f"Task {task_id} abort failed. No workers found."
        status_code = status.HTTP_400_BAD_REQUEST
    else:
        celery_app.control.revoke(task_id, terminate=True)
        message = f"Task {task_id} aborted."
        status_code = status.HTTP_200_OK

    return JSONResponse({"message": message}, status_code=status_code)

Revoke pending test#

Running 6 tasks (4 active tasks and 2 pending):

task_ids = []
for i in range(6):
    time.sleep(0.1)
    response = requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")
    task_id = response.json()["task_id"]
    task_ids.append(task_id)


pending = requests.get(f"{API_URL}/tasks/pending").json()
revoke_task_id = pending[-1]["task_id"]
requests.post(f"{API_URL}/tasks/revoke/{revoke_task_id}").json()
{'message': 'Task 2bfe74da-0c1b-49bb-88e2-930ca7d0e9db revoked.'}

Getting results of tasks:

for tid in task_ids:
    while not AsyncResult(tid).ready():
        time.sleep(0.1)

    response = requests.get(f"{API_URL}/results/{tid}")
    result = response.json()
    print(f"{tid}:", result["status"], result["result"])
1aca6463-f7e3-4df6-ba90-70c875180580: SUCCESS 1
4ac38d1b-964b-43ca-ace5-cf8106e8ccb6: SUCCESS 1
9c3ae1fa-ef7b-4013-a020-76e16262eb82: SUCCESS 1
d390b1c5-90f6-4688-8b25-b3a880cd908d: SUCCESS 1
627b7faa-2cd2-41b6-9c3b-110b6d161cfe: SUCCESS 1
2bfe74da-0c1b-49bb-88e2-930ca7d0e9db: REVOKED revoked

Revoked task from pending list did not succeed:

response = requests.get(f"{API_URL}/results/{revoke_task_id}")
response.json()
{'task_id': '2bfe74da-0c1b-49bb-88e2-930ca7d0e9db',
 'status': 'REVOKED',
 'successful': False,
 'result': 'revoked',
 'args': [],
 'kwargs': {'wait': 3.0, 'return_value': 1},
 'date_done': '2024-02-22T20:59:43.703656'}

Abort active test#

Running 4 tasks (4 active, 0 pending):

task_ids = []
for i in range(4):
    time.sleep(0.1)
    response = requests.post(f"{API_URL}/tasks/sleep?wait=3&return_value=1&priority=0")
    task_id = response.json()["task_id"]
    task_ids.append(task_id)

active = requests.get(f"{API_URL}/tasks/active").json()
abort_task_id = active[-1]['task_id']
requests.post(f"{API_URL}/tasks/abort/{abort_task_id}").json()
{'message': 'Task 58d2f6b7-8802-4b09-8c0b-7075d6e415ae aborted.'}

Wait for tasks to finish and getting results:

for tid in task_ids:
    while not AsyncResult(tid).ready():
        time.sleep(0.1)

    response = requests.get(f"{API_URL}/results/{tid}")
    result = response.json()
    print(f"{tid}:", result["status"], result["result"])
2a1044a1-259d-4af0-8f49-520373da50f6: SUCCESS 1
eaabf3e8-47be-4702-a78e-5fc4358b51d5: SUCCESS 1
58d2f6b7-8802-4b09-8c0b-7075d6e415ae: REVOKED terminated
796f88ed-f77e-44e0-a881-90b08e428c32: SUCCESS 1

Cool. Celery assigns task result as terminated:

response = requests.get(f"{API_URL}/results/{abort_task_id}")
response.json()
{'task_id': '58d2f6b7-8802-4b09-8c0b-7075d6e415ae',
 'status': 'REVOKED',
 'successful': False,
 'result': 'terminated',
 'args': [],
 'kwargs': {'wait': 3.0, 'return_value': 1},
 'date_done': '2024-02-22T20:59:49.102432'}

Appendix: Further Celery#

Source: particle1331/distributed-task-queue (⚠ further-celery branch)

!rm -rf ./distributed-task-queue
!git clone --single-branch --branch further-celery https://github.com/particle1331/distributed-task-queue.git ./distributed-task-queue > /dev/null
!pip install -U git+https://github.com/particle1331/distributed-task-queue.git@further-celery --force-reinstall > /dev/null
!echo "\nSuccessfully installed: $(pip list | grep distributed-task-queue)/$(wget --header='Accept: application/vnd.github.VERSION.sha' -qO- commit_id  http://api.github.com/repos/particle1331/distributed-task-queue/commits/further-celery | head -c 7)"
Cloning into './distributed-task-queue'...
remote: Enumerating objects: 289, done.
remote: Counting objects: 100% (106/106), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 289 (delta 82), reused 59 (delta 58), pack-reused 183
Receiving objects: 100% (289/289), 1.06 MiB | 1.97 MiB/s, done.
Resolving deltas: 100% (167/167), done.
  Running command git clone --filter=blob:none --quiet https://github.com/particle1331/distributed-task-queue.git /private/var/folders/jq/9vsvd9252_349lsng_5gc_jw0000gn/T/pip-req-build-0l0gj037
  Running command git checkout -b further-celery --track origin/further-celery
  Switched to a new branch 'further-celery'
  branch 'further-celery' set up to track 'origin/further-celery'.

Successfully installed: distributed-task-queue        0.0.0/b5aad0c

Rebuilding stack:

!docker compose -f ./distributed-task-queue/docker-compose.yml down
!docker compose -f ./distributed-task-queue/docker-compose.yml up -d --build
Hide code cell output
?25l[+] Running 0/0
 ⠋ Container distributed-task-queue-flower-1  Stopping                     0.0s 
 ⠋ Container distributed-task-queue-api-1     Stopping                     0.0s 
 ⠋ Container distributed-task-queue-worker-1  Stopping                     0.0s 
 ⠋ Container distributed-task-queue-worker-2  Stopping                     0.0s 
?25h?25l[+] Running 0/4
 ⠙ Container distributed-task-queue-flower-1  Stopping                     0.1s 
 ⠙ Container distributed-task-queue-api-1     Stopping                     0.1s 
 ⠙ Container distributed-task-queue-worker-1  Stopping                     0.1s 
 ⠙ Container distributed-task-queue-worker-2  Stopping                     0.1s 
?25h?25l[+] Running 0/4
 ⠹ Container distributed-task-queue-flower-1  Stopping                     0.2s 
 ⠹ Container distributed-task-queue-api-1     Stopping                     0.2s 
 ⠹ Container distributed-task-queue-worker-1  Stopping                     0.2s 
 ⠹ Container distributed-task-queue-worker-2  Stopping                     0.2s 
?25h?25l[+] Running 0/4
 ⠸ Container distributed-task-queue-flower-1  Stopping                     0.3s 
 ⠸ Container distributed-task-queue-api-1     Stopping                     0.3s 
 ⠸ Container distributed-task-queue-worker-1  Stopping                     0.3s 
 ⠸ Container distributed-task-queue-worker-2  Stopping                     0.3s 
?25h?25l[+] Running 0/4
 ⠼ Container distributed-task-queue-flower-1  Stopping                     0.4s 
 ⠼ Container distributed-task-queue-api-1     Stopping                     0.4s 
 ⠼ Container distributed-task-queue-worker-1  Stopping                     0.4s 
 ⠼ Container distributed-task-queue-worker-2  Stopping                     0.4s 
?25h?25l[+] Running 0/4
 ⠴ Container distributed-task-queue-flower-1  Stopping                     0.5s 
 ⠴ Container distributed-task-queue-api-1     Stopping                     0.5s 
 ⠴ Container distributed-task-queue-worker-1  Stopping                     0.5s 
 ⠴ Container distributed-task-queue-worker-2  Stopping                     0.5s 
?25h?25l[+] Running 0/4
 ⠦ Container distributed-task-queue-flower-1  Stopping                     0.6s 
 ⠦ Container distributed-task-queue-api-1     Stopping                     0.6s 
 ⠦ Container distributed-task-queue-worker-1  Stopping                     0.6s 
 ⠦ Container distributed-task-queue-worker-2  Stopping                     0.6s 
?25h?25l[+] Running 0/4
 ⠧ Container distributed-task-queue-flower-1  Stopping                     0.7s 
 ⠧ Container distributed-task-queue-api-1     Stopping                     0.7s 
 ⠧ Container distributed-task-queue-worker-1  Stopping                     0.7s 
 ⠧ Container distributed-task-queue-worker-2  Stopping                     0.7s 
?25h?25l[+] Running 0/4
 ⠇ Container distributed-task-queue-flower-1  Stopping                     0.8s 
 ⠇ Container distributed-task-queue-api-1     Stopping                     0.8s 
 ⠇ Container distributed-task-queue-worker-1  Stopping                     0.8s 
 ⠇ Container distributed-task-queue-worker-2  Stopping                     0.8s 
?25h?25l[+] Running 0/4
 ⠏ Container distributed-task-queue-flower-1  Stopping                     0.9s 
 ⠏ Container distributed-task-queue-api-1     Stopping                     0.9s 
 ⠏ Container distributed-task-queue-worker-1  Stopping                     0.9s 
 ⠏ Container distributed-task-queue-worker-2  Stopping                     0.9s 
?25h?25l[+] Running 1/4
  Container distributed-task-queue-flower-1  Stopped                      1.0s 
 ⠋ Container distributed-task-queue-api-1     Stopping                     1.0s 
 ⠋ Container distributed-task-queue-worker-1  Stopping                     1.0s 
 ⠋ Container distributed-task-queue-worker-2  Stopping                     1.0s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠙ Container distributed-task-queue-worker-1  Stopping                     1.1s 
 ⠙ Container distributed-task-queue-worker-2  Stopping                     1.1s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠹ Container distributed-task-queue-worker-1  Stopping                     1.2s 
 ⠹ Container distributed-task-queue-worker-2  Stopping                     1.2s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠸ Container distributed-task-queue-worker-1  Stopping                     1.3s 
 ⠸ Container distributed-task-queue-worker-2  Stopping                     1.3s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠼ Container distributed-task-queue-worker-1  Stopping                     1.4s 
 ⠼ Container distributed-task-queue-worker-2  Stopping                     1.4s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠴ Container distributed-task-queue-worker-1  Stopping                     1.5s 
 ⠴ Container distributed-task-queue-worker-2  Stopping                     1.5s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠦ Container distributed-task-queue-worker-1  Stopping                     1.6s 
 ⠦ Container distributed-task-queue-worker-2  Stopping                     1.6s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠧ Container distributed-task-queue-worker-1  Stopping                     1.7s 
 ⠧ Container distributed-task-queue-worker-2  Stopping                     1.7s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠇ Container distributed-task-queue-worker-1  Stopping                     1.8s 
 ⠇ Container distributed-task-queue-worker-2  Stopping                     1.8s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠏ Container distributed-task-queue-worker-1  Stopping                     1.9s 
 ⠏ Container distributed-task-queue-worker-2  Stopping                     1.9s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠋ Container distributed-task-queue-worker-1  Stopping                     2.0s 
 ⠋ Container distributed-task-queue-worker-2  Stopping                     2.0s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠙ Container distributed-task-queue-worker-1  Stopping                     2.1s 
 ⠙ Container distributed-task-queue-worker-2  Stopping                     2.1s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠹ Container distributed-task-queue-worker-1  Stopping                     2.2s 
 ⠹ Container distributed-task-queue-worker-2  Stopping                     2.2s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠸ Container distributed-task-queue-worker-1  Stopping                     2.3s 
 ⠸ Container distributed-task-queue-worker-2  Stopping                     2.3s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠼ Container distributed-task-queue-worker-1  Stopping                     2.4s 
 ⠼ Container distributed-task-queue-worker-2  Stopping                     2.4s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠴ Container distributed-task-queue-worker-1  Stopping                     2.5s 
 ⠴ Container distributed-task-queue-worker-2  Stopping                     2.5s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠦ Container distributed-task-queue-worker-1  Stopping                     2.6s 
 ⠦ Container distributed-task-queue-worker-2  Stopping                     2.6s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠧ Container distributed-task-queue-worker-1  Stopping                     2.7s 
 ⠧ Container distributed-task-queue-worker-2  Stopping                     2.7s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠇ Container distributed-task-queue-worker-1  Stopping                     2.8s 
 ⠇ Container distributed-task-queue-worker-2  Stopping                     2.8s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠏ Container distributed-task-queue-worker-1  Stopping                     2.9s 
 ⠏ Container distributed-task-queue-worker-2  Stopping                     2.9s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠋ Container distributed-task-queue-worker-1  Stopping                     3.0s 
 ⠋ Container distributed-task-queue-worker-2  Stopping                     3.0s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠙ Container distributed-task-queue-worker-1  Stopping                     3.1s 
 ⠙ Container distributed-task-queue-worker-2  Stopping                     3.1s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠹ Container distributed-task-queue-worker-1  Stopping                     3.2s 
 ⠹ Container distributed-task-queue-worker-2  Stopping                     3.2s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠸ Container distributed-task-queue-worker-1  Stopping                     3.3s 
 ⠸ Container distributed-task-queue-worker-2  Stopping                     3.3s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠼ Container distributed-task-queue-worker-1  Stopping                     3.4s 
 ⠼ Container distributed-task-queue-worker-2  Stopping                     3.4s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠴ Container distributed-task-queue-worker-1  Stopping                     3.5s 
 ⠴ Container distributed-task-queue-worker-2  Stopping                     3.5s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠦ Container distributed-task-queue-worker-1  Stopping                     3.6s 
 ⠦ Container distributed-task-queue-worker-2  Stopping                     3.6s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠧ Container distributed-task-queue-worker-1  Stopping                     3.7s 
 ⠧ Container distributed-task-queue-worker-2  Stopping                     3.7s 
?25h?25l[+] Running 2/4
  Container distributed-task-queue-flower-1  Removed                      1.0s 
  Container distributed-task-queue-api-1     Removed                      1.1s 
 ⠇ Container distributed-task-queue-worker-1  Stopping                     3.8s 
 ⠇ Container distributed-task-queue-worker-2  Stopping                     3.8s 
?25h?25l[+] Running 4/4
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
 ⠋ Container distributed-task-queue-redis-1     Stopping                   0.0s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   0.0s 
?25h?25l[+] Running 4/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
 ⠙ Container distributed-task-queue-redis-1     Stopping                   0.1s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   0.1s 
?25h?25l[+] Running 4/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
 ⠹ Container distributed-task-queue-redis-1     Stopping                   0.2s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   0.2s 
?25h?25l[+] Running 4/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
 ⠸ Container distributed-task-queue-redis-1     Stopping                   0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   0.3s 
?25h?25l[+] Running 4/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
 ⠼ Container distributed-task-queue-redis-1     Stopping                   0.4s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   0.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Stopped                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   0.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   0.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   0.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   0.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   0.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   1.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   1.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   1.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   1.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   1.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   1.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   1.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   1.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   1.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   1.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   2.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   2.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   2.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   2.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   2.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   2.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   2.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   2.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   2.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   2.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   3.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   3.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   3.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   3.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   3.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   3.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   3.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   3.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   3.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   3.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   4.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   4.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   4.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   4.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   4.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   4.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   4.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   4.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   4.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   4.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   5.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   5.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   5.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   5.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   5.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   5.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   5.6s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠧ Container distributed-task-queue-rabbitmq-1  Stopping                   5.7s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠇ Container distributed-task-queue-rabbitmq-1  Stopping                   5.8s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠏ Container distributed-task-queue-rabbitmq-1  Stopping                   5.9s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠋ Container distributed-task-queue-rabbitmq-1  Stopping                   6.0s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠙ Container distributed-task-queue-rabbitmq-1  Stopping                   6.1s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠹ Container distributed-task-queue-rabbitmq-1  Stopping                   6.2s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠸ Container distributed-task-queue-rabbitmq-1  Stopping                   6.3s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠼ Container distributed-task-queue-rabbitmq-1  Stopping                   6.4s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠴ Container distributed-task-queue-rabbitmq-1  Stopping                   6.5s 
?25h?25l[+] Running 5/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
 ⠦ Container distributed-task-queue-rabbitmq-1  Stopping                   6.6s 
?25h?25l[+] Running 6/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
  Container distributed-task-queue-rabbitmq-1  Removed                    6.7s 
?25h?25l[+] Running 7/6
  Container distributed-task-queue-flower-1    Removed                    1.0s 
  Container distributed-task-queue-api-1       Removed                    1.1s 
  Container distributed-task-queue-worker-1    Removed                    3.8s 
  Container distributed-task-queue-worker-2    Removed                    3.8s 
  Container distributed-task-queue-redis-1     Removed                    0.5s 
  Container distributed-task-queue-rabbitmq-1  Removed                    6.7s 
  Network distributed-task-queue_default       Removed                    0.1s 
?25h?25l[+] Building 0.0s (0/0)                                    docker:desktop-linux
?25h?25l[+] Building 0.0s (4/5)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [worker internal] load metadata for docker.io/library/python:3.9.15-s  0.0s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
?25h?25l[+] Building 0.2s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.2s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 0.3s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.3s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 0.5s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.5s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 0.6s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.6s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 0.8s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.8s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 0.9s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  0.9s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 1.1s (6/7)                                    docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [other-worker internal] load metadata for docker.io/library/python:3.  1.1s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
?25h?25l[+] Building 1.2s (26/31)                                  docker:desktop-linux
 => [api internal] load .dockerignore                                      0.0s
 => => transferring context: 2B                                            0.0s
 => [api internal] load build definition from Dockerfile.api               0.0s
 => => transferring dockerfile: 340B                                       0.0s
 => [api internal] load metadata for docker.io/library/python:3.9.15-slim  1.1s
 => [worker internal] load .dockerignore                                   0.0s
 => => transferring context: 2B                                            0.0s
 => [worker internal] load build definition from Dockerfile.worker         0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load build definition from Dockerfile.worker   0.0s
 => => transferring dockerfile: 271B                                       0.0s
 => [other-worker internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                            0.0s
 => [other-worker 1/7] FROM docker.io/library/python:3.9.15-slim@sha256:f  0.0s
 => [worker internal] load build context                                   0.0s
 => => transferring context: 3.10kB                                        0.0s
 => [api internal] load build context                                      0.0s
 => => transferring context: 8.97kB                                        0.0s
 => [other-worker internal] load build context                             0.0s
 => => transferring context: 3.10kB                                        0.0s
 => CACHED [api 2/7] WORKDIR /opt                                          0.0s
 => CACHED [worker 3/7] COPY ./dtq/requirements.txt .                      0.0s
 => CACHED [worker 4/7] RUN pip install -r requirements.txt                0.0s
 => CACHED [worker 5/7] RUN pip install setproctitle                       0.0s
 => CACHED [worker 6/7] RUN apt-get update && apt-get install -y procps    0.0s
 => CACHED [worker 7/7] COPY /dtq/ ./dtq/                                  0.0s
 => [other-worker] exporting to image                                      0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:904fbc852c456db67b1bed754759b15b2507336d29803  0.0s
 => => naming to docker.io/library/distributed-task-queue-other-worker     0.0s
 => [worker] exporting to image                                            0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:ce1e8b2b2e09908b24c96f9d21fc3bc1a7660366bb364  0.0s
 => => naming to docker.io/library/distributed-task-queue-worker           0.0s
 => CACHED [api 3/8] COPY ./dtq/requirements.txt dtq-requirements.txt      0.0s
 => CACHED [api 4/8] COPY ./api/requirements.txt api-requirements.txt      0.0s
 => CACHED [api 5/8] RUN pip install -r dtq-requirements.txt               0.0s
 => CACHED [api 6/8] RUN pip install -r api-requirements.txt               0.0s
 => CACHED [api 7/8] COPY ./dtq ./dtq                                      0.0s
 => CACHED [api 8/8] COPY ./api .                                          0.0s
 => [api] exporting to image                                               0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:8a0de09964cfbd59bd03ff1154eed5f68bec0ad24deed  0.0s
 => => naming to docker.io/library/distributed-task-queue-api              0.0s
?25h?25l[+] Running 0/0
 ⠋ Network distributed-task-queue_default  Creating                        0.0s 
?25h?25l[+] Running 3/1
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
 ⠋ Container distributed-task-queue-api-1           Creating               0.0s 
 ⠋ Container distributed-task-queue-worker-2        Creating               0.0s 
 ⠋ Container distributed-task-queue-other-worker-1  Creating               0.0s 
 ⠋ Container distributed-task-queue-worker-1        Creating               0.0s 
 ⠋ Container distributed-task-queue-flower-1        Creating               0.0s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Created                0.1s 
  Container distributed-task-queue-redis-1         Created                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Created                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Created                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Created                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Created                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Started                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Started                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Started                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Started                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Started                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Started                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Started                0.1s 
  Container distributed-task-queue-worker-2        Created                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Started                0.1s 
?25h?25l[+] Running 8/8
  Network distributed-task-queue_default           Created                0.1s 
  Container distributed-task-queue-rabbitmq-1      Started                0.1s 
  Container distributed-task-queue-redis-1         Started                0.1s 
  Container distributed-task-queue-api-1           Started                0.1s 
  Container distributed-task-queue-worker-2        Started                0.1s 
  Container distributed-task-queue-other-worker-1  Started                0.1s 
  Container distributed-task-queue-worker-1        Started                0.1s 
  Container distributed-task-queue-flower-1        Started                0.1s 
?25h
!docker compose -f ./distributed-task-queue/docker-compose.yml ps
NAME                                    IMAGE                                 COMMAND                  SERVICE        CREATED         STATUS                  PORTS
distributed-task-queue-api-1            distributed-task-queue-api            "uvicorn main:app --…"   api            3 seconds ago   Up Less than a second   0.0.0.0:8000->8000/tcp
distributed-task-queue-flower-1         mher/flower                           "celery --broker=amq…"   flower         3 seconds ago   Up Less than a second   0.0.0.0:5555->5555/tcp
distributed-task-queue-other-worker-1   distributed-task-queue-other-worker   "celery -A dtq.app w…"   other-worker   3 seconds ago   Up 1 second             
distributed-task-queue-rabbitmq-1       rabbitmq:3.12.0-management            "docker-entrypoint.s…"   rabbitmq       3 seconds ago   Up 1 second             4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp
distributed-task-queue-redis-1          redis:latest                          "docker-entrypoint.s…"   redis          3 seconds ago   Up 1 second             0.0.0.0:6379->6379/tcp
distributed-task-queue-worker-1         distributed-task-queue-worker         "celery -A dtq.app w…"   worker         3 seconds ago   Up 1 second             
distributed-task-queue-worker-2         distributed-task-queue-worker         "celery -A dtq.app w…"   worker         3 seconds ago   Up Less than a second   

More options#

Some interesting Celery features we can look into for our application is to add retries. Having time limits can prevent tasks from blocking workers. For example, a task that is running 3 hours when the average task runs for about 3 minutes. This task can be cleaned up and logged so that further investigation can be done.

Reworking our previous task definition:

from dtq.tasks import random_fail

random_fail??
Signature:       random_fail(prob=0.5)
Call signature:  random_fail(*a, **kw)
Type:            PromiseProxy
String form:     <@task: dtq.tasks.random_fail of distributed-task-queue at 0x1057bc490>
File:            ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/tasks.py
Source:         
@app.task(
    autoretry_for=(ZeroDivisionError,),
    max_retries=2,
    retry_backoff=3,
    retry_jitter=True,
)
def random_fail(prob=0.5):
    time.sleep(0.1)
    x = random.random()
    return 1 / int(x > prob)
Class docstring:
Task base class.

Note:
    When called tasks apply the :meth:`run` method.  This method must
    be defined by all tasks (that is unless the :meth:`__call__` method
    is overridden).

This retries twice during a ZeroDivisionError with the first retry occuring between 0 and 3 seconds, and the next between 0 and 6 seconds. Testing:

result = random_fail.apply_async(kwargs={"prob": 1.0})
result
<AsyncResult: f2013e77-f82c-49f0-9d07-a0f6c649b991>
../../_images/04-celery-retries.png

Fig. 98 Task retries. Note that child tasks from retries have the same task ID.#

Setting client-side time limits (in seconds). Note that these can be set at definition in the task decorators in case we have identified a suitable upper bound for the execution time for the task for all workers. The soft time limit raises SoftTimeLimitExceeded which allows handling or cleanup, before the hard time limit kills the process with SIGKILL.

from dtq.tasks import sleep

result = sleep.apply_async(
    kwargs=dict(wait=3), 
    time_limit=2, soft_time_limit=1
)

while not result.ready():
    continue

response = requests.get(f"{API_URL}/results/{result.id}")
response.json()
{'task_id': '45823544-da5c-4955-b331-272d968893a4',
 'status': 'FAILURE',
 'successful': False,
 'result': 'SoftTimeLimitExceeded()',
 'args': [],
 'kwargs': {'wait': 3},
 'date_done': '2024-02-22T21:00:48.620936'}

Hard time limit:

result = sleep.apply_async(kwargs=dict(wait=3), time_limit=1)
while not result.ready():
    continue

response = requests.get(f"{API_URL}/results/{result.id}")
response.json()
{'task_id': 'ce748246-66ca-4821-b876-9f04ef1b6f04',
 'status': 'FAILURE',
 'successful': False,
 'result': 'TimeLimitExceeded(1,)',
 'args': [],
 'kwargs': {'wait': 3},
 'date_done': '2024-02-22T21:00:49.751109'}

Logging#

Celery automatically sets up logging. The compose file has been modified to write logs to a file:

celery -A dtq.app worker -l INFO -f celery.log

Checking logs for above task that exited with soft time limit:

!docker exec distributed-task-queue-worker-1 tail -14 /opt/celery.log
[2024-02-22 21:00:47,608: INFO/MainProcess] Task dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4] received
[2024-02-22 21:00:48,616: WARNING/MainProcess] Soft time limit (1s) exceeded for dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4]
[2024-02-22 21:00:48,624: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:48,625: ERROR/ForkPoolWorker-1] Task dtq.tasks.sleep[45823544-da5c-4955-b331-272d968893a4] raised unexpected: SoftTimeLimitExceeded()
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/dtq/tasks.py", line 9, in sleep
    time.sleep(wait)
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
    raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
!docker exec distributed-task-queue-worker-2 tail -14 /opt/celery.log
[2024-02-22 21:00:49,750: ERROR/MainProcess] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:49,750: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(1)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(1,)
"""
[2024-02-22 21:00:49,751: ERROR/MainProcess] Hard time limit (1s) exceeded for dtq.tasks.sleep[ce748246-66ca-4821-b876-9f04ef1b6f04]
[2024-02-22 21:00:49,869: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:7 exited with 'signal 9 (SIGKILL)'

Note worker process which triggered SoftTimeLimitExceeded is still alive. This is not true for worker process which triggered TimeLimitExceeded:

!docker exec distributed-task-queue-worker-1 ps aux | grep celeryd
root         1  4.0  0.9  46248 38424 ?        Ss   21:00   0:01 [celeryd: celery@8f52c2c55c86:MainProcess] -active- (-A dtq.app worker --concurrency 2 -Q celery -l INFO -f celery.log)
root         8  0.5  0.8  45972 35340 ?        S    21:00   0:00 [celeryd: celery@8f52c2c55c86:ForkPoolWorker-1]
root         9  0.0  0.8  43484 33168 ?        S    21:00   0:00 [celeryd: celery@8f52c2c55c86:ForkPoolWorker-2]
!docker exec distributed-task-queue-worker-2 ps aux | grep celeryd
root         1  4.0  0.9  46504 38576 ?        Ss   21:00   0:01 [celeryd: celery@4d44321eb801:MainProcess] -active- (-A dtq.app worker --concurrency 2 -Q celery -l INFO -f celery.log)
root         8  0.2  0.8  45976 35216 ?        S    21:00   0:00 [celeryd: celery@4d44321eb801:ForkPoolWorker-2]
root         9  0.0  0.8  46508 35064 ?        S    21:00   0:00 [celeryd: celery@4d44321eb801:ForkPoolWorker-3]

Workflows#

Task workflows also seems to be very interesting and feature rich. Recall that tasks execute asynchronously. To execute tasks in sequence (i.e. executing a child task once a parent task completes) we can use chains which can be implemented by using the pipe operator | on individual tasks:

from dtq.tasks import random_fail, sleep

chain = (
    random_fail.si(prob=1.0)
    | sleep.si(wait=3, return_value=0)
    | sleep.si(wait=3, return_value=1)
)

result = chain.delay()
while not result.ready():
    continue

try:
    print(result.parent.parent.id)
    print(result.parent.id)
    print(result.id)
    print(result.get())
except Exception as e:
    print(e)
dbcd58f6-6a6a-42c1-ace5-a5d59db6dc3d
d0848984-35c7-4775-9fd7-cb41358851db
1eed767d-d588-4f99-ae76-d8be7fcafcd8
division by zero

Failure cascades to the succeeding tasks:

r = requests.get(f"http://localhost:8000/results/{result.parent.parent.id}")
r.json()
{'task_id': 'dbcd58f6-6a6a-42c1-ace5-a5d59db6dc3d',
 'status': 'FAILURE',
 'successful': False,
 'result': 'division by zero',
 'args': [],
 'kwargs': {'prob': 1.0},
 'date_done': '2024-02-22T21:00:56.204850'}
r = requests.get(f"http://localhost:8000/results/{result.parent.id}")
r.json()
{'task_id': 'd0848984-35c7-4775-9fd7-cb41358851db',
 'status': 'FAILURE',
 'successful': False,
 'result': 'division by zero',
 'args': [],
 'kwargs': {'wait': 3, 'return_value': 0},
 'date_done': '2024-02-22T21:00:56.208908'}
r = requests.get(f"http://localhost:8000/results/{result.id}")
r.json()
{'task_id': '1eed767d-d588-4f99-ae76-d8be7fcafcd8',
 'status': 'FAILURE',
 'successful': False,
 'result': 'division by zero',
 'args': [],
 'kwargs': {'wait': 3, 'return_value': 1},
 'date_done': '2024-02-22T21:00:56.208134'}

Features for parallel task execution and joining parallel task results also exist. Error callbacks for chains is particularly interesting as it helps separating task logic from error handling code (e.g. logging error trace). See the docs.

../../_images/04-celery-wfgraph.png

Fig. 99 A complex graph structure built from chains and groups (23 tasks in total). Source#

Signals#

Several kinds of events trigger signals, you can connect to these signals to perform actions as they trigger. Here we define a worker_init signal that is dispatched before the worker is started, and a task-failure signal dispatched when a task fails. A logger has to be set up since the celery logger does not exist yet at this point.

Hide code cell source
!pygmentize -g ./distributed-task-queue/dtq/signals.py
# pylint: disable=unused-argument
import logging
import time

from celery.exceptions import WorkerShutdown
from celery.signals import task_failure, worker_init
from celery.utils.log import get_logger


@worker_init.connect
def setup_workers(sender=None, conf=None, **kwargs):
    logging.basicConfig(
        filename="worker_init.log",
        filemode="w",
        format="[%(asctime)s: %(levelname)s/%(name)s] %(message)s",
        level=logging.INFO,
    )
    logger = logging.getLogger("WorkerInit")

    try:
        logger.info("Starting worker...")
        time.sleep(12.0)
        logger.info("Worker started successfully.")

    except Exception as exc:
        logger.error(exc, exc_info=True)
        logger.critical("Failed to start worker. Shutting down...")
        raise WorkerShutdown() from exc


@task_failure.connect
def task_failure_notifier(sender=None, task_id=None, **kwargs):
    logger = get_logger("celery")
    message = (
        f"[task_failure_notifier]: Task {sender.name} failed successfully!"
    )
    logger.error(message)

Defining a worker_init signal can be useful when we want to dynamic initialization before the worker picks up tasks. Pinging some external dependency such as a background prediction server can be done here. Or a worker can shut down whenever an exception is encountered during initialization.

!docker exec distributed-task-queue-worker-1 cat worker_init.log
[2024-02-22 21:00:26,091: INFO/WorkerInit] Starting worker...
[2024-02-22 21:00:38,105: INFO/WorkerInit] Worker started successfully.

The task_failure_notifier simply logs a message in the event of a failure.

!docker exec distributed-task-queue-worker-1 cat celery.log | grep task_failure_notifier
[2024-02-22 21:00:43,255: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.random_fail failed successfully!
[2024-02-22 21:00:48,624: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.sleep failed successfully!
[2024-02-22 21:00:56,209: ERROR/ForkPoolWorker-1] [task_failure_notifier]: Task dtq.tasks.random_fail failed successfully!

../../_images/04-celery-signals.png

Fig. 100 Notice that worker logging starts after worker initialization. Task failure notifier occurs exactly when a failure occurs.#

Broker config#

Here we look at configuring RabbitMQ for our use-case of having long-running tasks. RabbitMQ has a default timeout of 30 mins before a received task is acknowledged. Otherwise, it triggers a PreconditionFailed error redelivering the task to another worker. RabbitMQ also closes the channel for this worker, eventually killing all of our workers. We modify this behavior and instead rely on soft and hard time limits to ensure that workers are not hanging.

../../_images/04-celery-precondition-fail.png

Fig. 101 Sleep task with wait time exceeding 1800s triggers PRECONDITION_FAILED. RabbitMQ closes the connection.#

To fix this we created the following file:

# rabbitmq/rabbitmq.conf
consumer_timeout = 31622400000

This is in milliseconds which is equivalent to roughly 1 year. Then, the compose file is modified as follows. The change should be visible in the Queues/Consumers tab of the RabbitMQ management console.

# docker-compose.yaml (further-celery)
version: "3"
services:
    ...

    rabbitmq:
        image: rabbitmq:3.12.0-management
        ports:
        - "5672:5672"
        - "15672:15672"
        volumes:
        - ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf

    ...

Appendix: Task routing#

We can also implement task routing. This can be useful, for example, if we want specific machines (e.g. an expensive instance with a GPU) to focus on specific tasks. Note that our implementation of ActiveTask model hides routing keys from tasks. The configuration can be modified by defining another queue called other:

# ./dtq/app.py (further-celery)
...

CELERY_CONFIG = {
    "task_acks_late": True,
    "worker_prefetch_multiplier": 1,
    "task_default_priority": 0,
    "task_queue_max_priority": 10,
    "task_create_missing_queues": True,     # (!)
    "result_expires": None,
    "result_extended": True,
}

app.conf.update(CELERY_CONFIG)
app.conf.task_routes = {                    # (!)
    "dtq.tasks.*": {"queue": "celery"},
    "dtq.other_tasks.*": {"queue": "other"}
}
...

This automatically creates queues which can be bad. But this is the simplest way to do this. Routing tasks based on their module path to certain queues is defined in task_routes above. Then, the specific worker for that queue is started (along with other arguments) using the -Q flag. Here we start a worker on the new queue:

# ./docker-compose.yml
version: "3"
services:
  ...

  other-worker:
    build:
      dockerfile: Dockerfile.worker
    depends_on:
      - redis
      - rabbitmq
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_PORT=5672
    deploy:
      replicas: 1
    command: [
      "-A", "dtq.app", "worker", "-n", "other@%h", "--concurrency", "1", "-Q", "other",
      "-l", "INFO", "-f", "celery.log"
    ]

    ...

Our new task:

from dtq.other_tasks import hello

hello??
Signature:       hello()
Call signature:  hello(*a, **kw)
Type:            PromiseProxy
String form:     <@task: dtq.other_tasks.hello of distributed-task-queue at 0x131b33e80>
File:            ~/opt/miniconda3/envs/tasks/lib/python3.9/site-packages/dtq/other_tasks.py
Source:         
@app.task
def hello():
    return "Hello, from other queue."
Class docstring:
Task base class.

Note:
    When called tasks apply the :meth:`run` method.  This method must
    be defined by all tasks (that is unless the :meth:`__call__` method
    is overridden).

Recall that we have 4 workers on the celery queue and 1 worker on the other queue. The other tasks are not distributed to the default workers:

for i in range(4):
    hello.delay()

time.sleep(1.0)

Similarly, when two tasks are pending in the default queue, the other worker does not pick up:

for i in range(6):
    sleep.delay(5)

time.sleep(6.5)

Overriding task routes. Configured task routes can be overridden using the queue parameter:

# swapping out the queues
# ⚠⚠⚠ Typo in queue name creates a new queue (without workers)! Fix: global vars
hello.apply_async(queue="celery")
sleep.apply_async(kwargs=dict(wait=3), queue="other")
Hide code cell output
<AsyncResult: e7b9a385-d621-40e9-9a60-b8ad7c72a897>
../../_images/04-celery-queue-other.png

Fig. 102 Workers only pick up tasks routed to their configured queues (even when free). This is overridden in the last two tasks where the queues are swapped.#

Remark. Multiple queues can be used to A/B test different code or ML model versions. Or do things like canary deployment. The task API can be internally modified to dynamically route to certain queues (e.g. based on argument values).

Appendix: Periodic tasks#

Periodic schedules using celery beat only need to be added in the application code:

# ./dtq/app.py
from celery.schedules import crontab

...

app.conf.beat_schedule = {
    "random-fail-every-10-seconds": {
        "task": "dtq.tasks.random_fail",
        "schedule": 10,
        "args": (),
    },
    "sleep-every-1-minute": {
        "task": "dtq.tasks.sleep",
        "schedule": crontab(),
        "args": (1, 0),
        "options": {"priority": 3},
    },
}

Running the schedule:

celery -A dtq.app beat

../../_images/04-celery-beat.png

Fig. 103 Tasks scheduled using celery beat.#

../../_images/04-celery-beat-rabbitmq.png

Fig. 104 Celery beat triggers the task execution without passing through a queue.#

!docker compose -f ./distributed-task-queue/docker-compose.yml down
Hide code cell output
?25l[+] Running 0/0
 ⠋ Container distributed-task-queue-other-worker-1  Stopping               0.1s 
 ⠋ Container distributed-task-queue-flower-1        Stopping               0.1s 
 ⠋ Container distributed-task-queue-worker-1        Stopping               0.1s 
 ⠋ Container distributed-task-queue-api-1           Stopping               0.1s 
 ⠋ Container distributed-task-queue-worker-2        Stopping               0.1s 
?25h?25l[+] Running 0/5
 ⠙ Container distributed-task-queue-other-worker-1  Stopping               0.2s 
 ⠙ Container distributed-task-queue-flower-1        Stopping               0.2s 
 ⠙ Container distributed-task-queue-worker-1        Stopping               0.2s 
 ⠙ Container distributed-task-queue-api-1           Stopping               0.2s 
 ⠙ Container distributed-task-queue-worker-2        Stopping               0.2s 
?25h?25l[+] Running 0/5
 ⠹ Container distributed-task-queue-other-worker-1  Stopping               0.3s 
 ⠹ Container distributed-task-queue-flower-1        Stopping               0.3s 
 ⠹ Container distributed-task-queue-worker-1        Stopping               0.3s 
 ⠹ Container distributed-task-queue-api-1           Stopping               0.3s 
 ⠹ Container distributed-task-queue-worker-2        Stopping               0.3s 
?25h?25l[+] Running 1/5
 ⠸ Container distributed-task-queue-other-worker-1  Stopping               0.4s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠸ Container distributed-task-queue-worker-1        Stopping               0.4s 
 ⠸ Container distributed-task-queue-api-1           Stopping               0.4s 
 ⠸ Container distributed-task-queue-worker-2        Stopping               0.4s 
?25h?25l[+] Running 1/5
 ⠼ Container distributed-task-queue-other-worker-1  Stopping               0.5s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠼ Container distributed-task-queue-worker-1        Stopping               0.5s 
 ⠼ Container distributed-task-queue-api-1           Stopping               0.5s 
 ⠼ Container distributed-task-queue-worker-2        Stopping               0.5s 
?25h?25l[+] Running 2/5
 ⠴ Container distributed-task-queue-other-worker-1  Stopping               0.6s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠴ Container distributed-task-queue-worker-1        Stopping               0.6s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠴ Container distributed-task-queue-worker-2        Stopping               0.6s 
?25h?25l[+] Running 2/5
 ⠦ Container distributed-task-queue-other-worker-1  Stopping               0.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠦ Container distributed-task-queue-worker-1        Stopping               0.7s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠦ Container distributed-task-queue-worker-2        Stopping               0.7s 
?25h?25l[+] Running 2/5
 ⠧ Container distributed-task-queue-other-worker-1  Stopping               0.8s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠧ Container distributed-task-queue-worker-1        Stopping               0.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠧ Container distributed-task-queue-worker-2        Stopping               0.8s 
?25h?25l[+] Running 2/5
 ⠇ Container distributed-task-queue-other-worker-1  Stopping               0.9s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠇ Container distributed-task-queue-worker-1        Stopping               0.9s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠇ Container distributed-task-queue-worker-2        Stopping               0.9s 
?25h?25l[+] Running 2/5
 ⠏ Container distributed-task-queue-other-worker-1  Stopping               1.0s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠏ Container distributed-task-queue-worker-1        Stopping               1.0s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠏ Container distributed-task-queue-worker-2        Stopping               1.0s 
?25h?25l[+] Running 2/5
 ⠋ Container distributed-task-queue-other-worker-1  Stopping               1.1s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠋ Container distributed-task-queue-worker-1        Stopping               1.1s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠋ Container distributed-task-queue-worker-2        Stopping               1.1s 
?25h?25l[+] Running 2/5
 ⠙ Container distributed-task-queue-other-worker-1  Stopping               1.2s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠙ Container distributed-task-queue-worker-1        Stopping               1.2s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠙ Container distributed-task-queue-worker-2        Stopping               1.2s 
?25h?25l[+] Running 2/5
 ⠹ Container distributed-task-queue-other-worker-1  Stopping               1.3s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠹ Container distributed-task-queue-worker-1        Stopping               1.3s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠹ Container distributed-task-queue-worker-2        Stopping               1.3s 
?25h?25l[+] Running 2/5
 ⠸ Container distributed-task-queue-other-worker-1  Stopping               1.4s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠸ Container distributed-task-queue-worker-1        Stopping               1.4s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠸ Container distributed-task-queue-worker-2        Stopping               1.4s 
?25h?25l[+] Running 2/5
 ⠼ Container distributed-task-queue-other-worker-1  Stopping               1.5s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠼ Container distributed-task-queue-worker-1        Stopping               1.5s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠼ Container distributed-task-queue-worker-2        Stopping               1.5s 
?25h?25l[+] Running 2/5
 ⠴ Container distributed-task-queue-other-worker-1  Stopping               1.6s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠴ Container distributed-task-queue-worker-1        Stopping               1.6s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠴ Container distributed-task-queue-worker-2        Stopping               1.6s 
?25h?25l[+] Running 2/5
 ⠦ Container distributed-task-queue-other-worker-1  Stopping               1.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠦ Container distributed-task-queue-worker-1        Stopping               1.7s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠦ Container distributed-task-queue-worker-2        Stopping               1.7s 
?25h?25l[+] Running 2/5
 ⠧ Container distributed-task-queue-other-worker-1  Stopping               1.8s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠧ Container distributed-task-queue-worker-1        Stopping               1.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠧ Container distributed-task-queue-worker-2        Stopping               1.8s 
?25h?25l[+] Running 2/5
 ⠇ Container distributed-task-queue-other-worker-1  Stopping               1.9s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠇ Container distributed-task-queue-worker-1        Stopping               1.9s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠇ Container distributed-task-queue-worker-2        Stopping               1.9s 
?25h?25l[+] Running 2/5
 ⠏ Container distributed-task-queue-other-worker-1  Stopping               2.0s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠏ Container distributed-task-queue-worker-1        Stopping               2.0s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠏ Container distributed-task-queue-worker-2        Stopping               2.0s 
?25h?25l[+] Running 2/5
 ⠋ Container distributed-task-queue-other-worker-1  Stopping               2.1s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠋ Container distributed-task-queue-worker-1        Stopping               2.1s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠋ Container distributed-task-queue-worker-2        Stopping               2.1s 
?25h?25l[+] Running 2/5
 ⠙ Container distributed-task-queue-other-worker-1  Stopping               2.2s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠙ Container distributed-task-queue-worker-1        Stopping               2.2s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠙ Container distributed-task-queue-worker-2        Stopping               2.2s 
?25h?25l[+] Running 2/5
 ⠹ Container distributed-task-queue-other-worker-1  Stopping               2.3s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠹ Container distributed-task-queue-worker-1        Stopping               2.3s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠹ Container distributed-task-queue-worker-2        Stopping               2.3s 
?25h?25l[+] Running 2/5
 ⠸ Container distributed-task-queue-other-worker-1  Stopping               2.4s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠸ Container distributed-task-queue-worker-1        Stopping               2.4s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠸ Container distributed-task-queue-worker-2        Stopping               2.4s 
?25h?25l[+] Running 2/5
 ⠼ Container distributed-task-queue-other-worker-1  Stopping               2.5s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠼ Container distributed-task-queue-worker-1        Stopping               2.5s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠼ Container distributed-task-queue-worker-2        Stopping               2.5s 
?25h?25l[+] Running 2/5
 ⠴ Container distributed-task-queue-other-worker-1  Stopping               2.6s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠴ Container distributed-task-queue-worker-1        Stopping               2.6s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠴ Container distributed-task-queue-worker-2        Stopping               2.6s 
?25h?25l[+] Running 2/5
 ⠦ Container distributed-task-queue-other-worker-1  Stopping               2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠦ Container distributed-task-queue-worker-1        Stopping               2.7s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠦ Container distributed-task-queue-worker-2        Stopping               2.7s 
?25h?25l[+] Running 3/5
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
 ⠧ Container distributed-task-queue-worker-1        Stopping               2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
 ⠧ Container distributed-task-queue-worker-2        Stopping               2.8s 
?25h?25l[+] Running 5/5
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Stopped                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
?25h?25l[+] Running 5/5
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
 ⠋ Container distributed-task-queue-redis-1         Stopping               0.1s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               0.1s 
?25h?25l[+] Running 5/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
 ⠙ Container distributed-task-queue-redis-1         Stopping               0.2s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               0.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               0.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               0.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               0.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               0.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               0.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               0.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               0.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               1.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               1.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               1.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               1.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               1.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               1.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               1.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               1.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               1.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               1.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               2.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               2.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               2.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               2.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               2.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               2.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               2.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               2.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               2.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               2.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               3.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               3.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               3.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               3.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               3.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               3.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               3.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               3.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               3.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               3.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               4.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               4.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               4.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               4.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               4.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               4.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               4.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               4.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               4.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               4.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               5.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               5.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               5.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               5.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               5.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               5.5s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠴ Container distributed-task-queue-rabbitmq-1      Stopping               5.6s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠦ Container distributed-task-queue-rabbitmq-1      Stopping               5.7s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠧ Container distributed-task-queue-rabbitmq-1      Stopping               5.8s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠇ Container distributed-task-queue-rabbitmq-1      Stopping               5.9s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠏ Container distributed-task-queue-rabbitmq-1      Stopping               6.0s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠋ Container distributed-task-queue-rabbitmq-1      Stopping               6.1s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠙ Container distributed-task-queue-rabbitmq-1      Stopping               6.2s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠹ Container distributed-task-queue-rabbitmq-1      Stopping               6.3s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠸ Container distributed-task-queue-rabbitmq-1      Stopping               6.4s 
?25h?25l[+] Running 6/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
 ⠼ Container distributed-task-queue-rabbitmq-1      Stopping               6.5s 
?25h?25l[+] Running 7/7
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
  Container distributed-task-queue-rabbitmq-1      Removed                6.5s 
 ⠋ Network distributed-task-queue_default           Removing               0.0s 
?25h?25l[+] Running 8/8
  Container distributed-task-queue-other-worker-1  Removed                2.7s 
  Container distributed-task-queue-flower-1        Removed                0.3s 
  Container distributed-task-queue-worker-1        Removed                2.8s 
  Container distributed-task-queue-api-1           Removed                0.5s 
  Container distributed-task-queue-worker-2        Removed                2.8s 
  Container distributed-task-queue-redis-1         Removed                0.3s 
  Container distributed-task-queue-rabbitmq-1      Removed                6.5s 
  Network distributed-task-queue_default           Removed                0.1s 
?25h