Weave at the compose stage

Docker compose is an efficient way to locally assemble - or compose hence da name - a bunch of Docker containers. You usually end up with a single file that spawns and stitches the different actors, dealing with network and environment configuration. But this raw setup does not scale very well as the ecosystem expands. Not only the number of services grows but the inherent coupling is likely to become more complex, increasing the declarative syntax and noising the overall intent. Each service may end up with a bunch of heterogenous environment variables to cope with, such as those specific to observability (o11y) or Identity Access Management (IAM), or be adorned with an ever growing depends_on section to materialize dependencies. It becomes both hard to read and hard to maintain. Moreover, it becomes harder to operate. Sometimes we would like to work on a subset of the whole ecosystem. One may rely on profile facility to do so but it is not that straightforward, as it complicates both the Docker compose file structure while noising the downstream command line. And imagine we need a couple of satellite containers to ease development. They will in turn grow Docker compose file content. And be instantiated every time unless you also provide dedicated policy and thus burden the whole thing.

I was exposed to some Docker compose files involving 25+ services. My first move was to understand the underlying structure, trying to materialize the stack we are deploying. Doing so by simply parsing the raw Docker compose file is hard. At least for me. As a lazy developer ™, I crafted a dedicated feature to enrich Cornifer with, moving from hundred of lines to tens of shapes. Far easier to cope with. But still difficult to mentally arrange. Reducing complexity comes with the losing information price. You may be smart but GIGO - Garbage In Garbage Out - paradigm rarely lies.

As always, when situation seems to be locked, step back. Change your point of view. Challenge the context. Should I really consider original Docker compose file as frozen, focusing to surface meaningful knowledge from it? Or should I first reshape this original Docker compose file to achieve my goals? Is there a way to end up with something both easy to read, maintain, and operate? Let’s see how one could engage this journey…

Pattern

SOLID principles are there for a while now in the software engineering world, standing for:

Single Responsibility Principle (SRP): A class should have only one reason to change.
Open/Closed Principle (OCP): Software entities (classes, modules, functions, …) should be open for extension but closed for modification.
Liskov Substitution Principle (LSP): Objects of a superclass should be able to be replaced with objects of a subclass without affecting the correctness of the program.
Interface Segregation Principle (ISP): Clients should not be forced to depend on interfaces they do not use.
Dependency Inversion Principle (DI): High-level modules should not depend on low-level modules. Both should depend on abstractions.

Let’s see how we can apply these principles to Docker compose files to achieve a more modular, maintainable, and scalable architecture.

SRP can be applied to Docker compose files by separating concerns into different files. Each file can focus on a specific aspect of the ecosystem, such as the main application, o11y, or IAM. This way, each file has a single responsibility and can be managed independently.

OCP can be applied by allowing the Docker compose files to be extended without modifying the original file. New files can be created to add new services or configurations, while the original file remains unchanged. This allows for easy updates and modifications without affecting the core functionality.

LSP can be applied by ensuring that new services or configurations can be added without breaking the existing functionality. This means that new files should be compatible with the original pattern and should not introduce breaking changes.

ISP can be applied by creating smaller, focused Docker compose files that only include the services and configurations needed for a specific aspect of the ecosystem. This way, Docker compose consumers can depend on only the files they use, reducing complexity and improving maintainability.

DI can be applied by ensuring that high-level modules (such as the main application) do not depend on low-level modules (such as specific services). Instead, coupling occurs when we assemble the compose files. This allows for greater flexibility and easier updates.

Implementation

Assuming we need to introduce a new dimension to our ecosystem, such as observability.

We can easily do so by creating a brand-new compose.o11y.yaml file to gather o11y related services and configuration. This file will be used in conjunction with the main compose.yaml file to keep the original stack intact while adding new functionality.

1
2
3
4


docker compose `
  -f compose.yaml `       # Main stack introducing services
  -f compose.o11y.yaml `  # Add o11y stack with Aspire dashboard
  up

The new file contains both new services, here the Aspire dashboard:

1
2
3
4
5
6


aspire:
  image: mcr.microsoft.com/dotnet/aspire-dashboard:9.1.0
  ports:
    - 18888:18888 # UI
  environment:
    - DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true

And extra configuration for the existing services:

1
2
3
4
5
6


backend:
  environment:
    OTEL_EXPORTER_OTLP_ENDPOINT: http://aspire:18889
    OTEL_SERVICE_NAME: backend
  depends_on:
    - aspire

It leverages Docker compose built-in facility to merge multiple files and consolidate the configuration.

Scaling

Introduce new dimension

Assuming we need to introduce another new dimension to our ecosystem, such as Identity Access Management.

One can easily doing so by creating a brand-new compose.iam.yaml file to gather IAM related services and configuration. As for o11y one, this file will be used in conjunction with the main compose.yaml file to keep the original stack intact while adding new functionality.

1
2
3
4


docker compose `
  -f compose.yaml `
  -f compose.iam.yaml `
  up

Applying pattern, the new file contains both new services, here Keycloak:

1
2


keycloak:
  image: quay.io/keycloak/keycloak:26.2.5

And extra configuration for the existing services:

1
2
3
4
5


frontend:
  environment:
    OD-IDC_REALM: archicionado
    OIDC_CLIENT_ID: daedalus
    OIDC_CLIENT_SECRET: xxxxxxxx

Introduce existing dimension twin

Adding new dimensions to our ecosystem is one way to scale but we may also want to swap existing ones. For instance, we may want to replace the o11y stack with a different one, such as OpenTelemetry (OTEL).

Assuming we need to enrich existing o11y dimension with a new OTEL flavor, leveraging OTEL collector.

Once again, we end up with a brand-new compose file. We may call it compose.o11y.prod.yaml to indicate that it is a production-ready version of the o11y stack. Or we may decide to name it compose.otlp.yaml, renaming the previous compose.o11y.yaml file to compose.aspire.yaml for consistency. It is up to you to decide the naming convention that fits your needs but please choose one.

1
2
3
4


docker compose `
  -f compose.yaml `             # Main stack introducing services
  -f compose.o11y.prod.yaml `   # Add o11y stack with OTEL collector flavor
  up

Introduce new services, here the OpenTelemetry Collector:

1
2
3
4
5


otel-collector:
  image: otel/opentelemetry-collector-contrib:0.127.0
  command: ["--config=/etc/otel-config.yaml"]
  volumes:
    - ./otel-config.yml:/etc/otel-config.yaml:ro

And enrich existing services with the matching configuration:

1
2
3
4
5
6


backend:
  environment:
    OTLPEXPORTER__ENDPOINT: http://otel-collector:4317
    OTEL_SERVICE_NAME: daedalus/backend
  depends_on:
    - otel-collector

Introduce porous dimension

We may also encounter situations where the new dimension is not orthogonal to the existing ones. For instance, Keycloak recently introduced native support of OpenTelemetry, allowing us to collect metrics and traces from Keycloak itself. This means we end up now with interleaved dimensions, where the o11y dimension is connected to both the main one and the o11y one(s).

To achieve this, one could create new compose.aspire.keycloak.yaml and compose.otel.keycloak.yaml files that bridge both dimensions. This approach may fit your needs if combinatorial explosion is not an issue.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Development
docker compose `
  -f compose.yaml `             # Main stack introducing services
  -f compose.aspire.yaml `      # Add o11y stack with Aspire dashboard
  -f compose.iam.yaml `         # Add IAM stack with Keycloak
  -f compose.iam.aspire.yaml `  # Enrich IAM with o11y leveraging Aspire dashboard
  up

# Production
docker compose `
  -f compose.yaml `           # Main stack introducing services
  -f compose.otel.yaml `      # Add o11y stack with OTEL collector
  -f compose.iam.yaml `       # Add IAM stack with Keycloak
  -f compose.iam.otel.yaml `  # Enrich IAM with o11y leveraging OTEL collector
  up

Instead, you may want to consider a different approach. Once again, step back. We made the choice to leverage Aspire dashboard for o11y during development and OTEL collector for production. But we could have chosen to use OTEL collector for both development and production, in conjunction with Aspire dashboard for development to ease telemetry consumption, ending up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Development
docker compose `
  -f compose.yaml `             # Main stack introducing services
  -f compose.o11y.yaml `        # Add o11y stack with OTEL collector
  -f compose.iam.yaml `         # Add IAM stack with Keycloak
  -f compose.iam.o11y.yaml `    # Enrich IAM with o11y leveraging OTEL collector
  -f compose.o11y.aspire.yaml ` # Enrich o11y stack by adding Aspire dashboard export to existing OTEL collector
  up

# Production
docker compose `
  -f compose.yaml `             # Main stack introducing services
  -f compose.o11y.yaml `        # Add o11y stack with OTEL collector
  -f compose.iam.yaml `         # Add IAM stack with Keycloak
  -f compose.iam.o11y.yaml `    # Enrich IAM with o11y leveraging OTEL collector
  up

For sure, we slightly complicate the o11y stack as we are now using OTEL collector as a telemetry sink every time. But we also simplify related services such as Keycloak, as from their point of view they only have to deal with a single o11y flavor. As always, there is no one-size-fits-all solution. It is all about tradeoff and, at the end of the day, up to you to decide the best approach that fits your needs.

Closing

Leveraging tools such as containerization (Docker) and orchestration (Docker compose) is the first move towards Infrastructure as Code (IaC). And, as name implies, pattern that applies to plain old code base can be extended here. And we may safely swap this previous can with a solid must be. The SOLID principles are not only applicable to software development but also to infrastructure management. By applying these principles to Docker compose files, we can achieve a more modular, maintainable, and scalable architecture. This approach allows us to easily extend our ecosystem with new services and configurations while keeping the original stack intact. By separating concerns into different files, we can reduce complexity and improve readability. Each file can focus on a specific aspect of the ecosystem, making it easier to manage and understand. This modular approach also allows for easier updates and modifications without affecting the core functionality. In conclusion, applying the SOLID principles to Docker compose files can greatly enhance the maintainability and scalability of our infrastructure. By embracing these principles, we can create a more organized and efficient ecosystem that is easier to manage and extend.