Manifests in Mexico

January, 2026

Microservices

About 8 years ago, I heard about microservices for the first time. It was something that my employer was using. Since then, they've appeared at nearly every company I've worked at, and I've grown to be largely against them.

How does it happen, then, that I'm now 3 (micro?)services deep on a product with no active users!?

I guess it's true. I am the meme... I have more services/servers than users.

But it has happened naturally, and this time, I feel good about it.

As I mentioned in a previous dev log update, I recently migrated off of the Pulumi Cloud as my state backend. This process was very smooth and pretty easy to reason about, largely because of the strict separation of concerns that I created between my services.

I heard a quote once that was roughly:

Any innate complexity in your system must be expressed somewhere. If it is not in your language, it will be in your application logic. If not in your application, it will be in the architecture.

I've absolutely noticed this trade-off. When I want to trigger a Pulumi action, the flow is roughly:

This is a more complicated flow than exposing everything over a single, public API. For example, I could just have the Console API verify the user's request, read from the database, and write the manifest. This would all be in the context of a single process.

Instead, I'm intentionally opting to use a separate service to handle Pulumi or Kubernetes-related tasks.

This adds some architectural complexity, but now I know that I can completely forget about Pulumi, AWS resources, and Kubernetes if I'm operating in the Console API. Instead, I can just focus on the experience of the end user, verifying their requests, and reading from the database to render HTML responses.

If I'm working on the Internal API, I don't have to think about users. I can just query for the configuration that's relevant to the current request and generate any relevant manifest(s).

The Internal API also exposes some endpoints for writebacks/webhooks from Github App installations or from the Pulumi Kubernetes Operator. The PKO writebacks are my current solution for syncing cloud resource data back to the Dorsal database. For example, a Domain's nameservers are known by Pulumi when they are created. The Pulumi Program then posts these back to the Internal API, which writes them to the database.

This inter-service communication is obviously one of the core features of a Kubernetes Cluster, but with a simple environment variable to set the base URL for the Internal API, I can run the exact some network locally with a simple Docker Compose definition.

At the database level, the Console API and Internal API have mutually exclusive read/write permissions for the different database schemas. This explicitly maps their scope of responsibility into the data access patterns.

The data flow and access patterns are baked into the architecture of the system.

All that is to say... I'm building Dorsal using microservices, and so Dorsal will be natively supporting a microservice architecture eventually.

Don't Write A Manifest

Above, I offhandedly mentioned syncing manifests based on database state. This has been the primary task of January so far (one week left to go at the time of writing).

After last month, I was fully generating the Pulumi Kubernetes Operator Stack manifests. This month, I expanded that manifest generation to ArgoCD Application and Karpenter NodePool manifests.

The end result: no more manually updating manifests when I want to deploy something.

So, what am I actually building?

I'm still working on the terminology here, but let's go with "Compute" and "Workload" for now.

A "Compute" resource will define the underlying NodePool that will be created, and the "Workload" will be any unit of an application that will be deployed to the Compute.

A Workload may wind up mapping 1:1 with a Kubernetes Pod, but I'm not sure yet, as I'd like to make Init Containers and Sidecars composable with main containers. This may mean having something like "infinite" and "finite" Workloads, which can be combined to produce the underlying Kubernetes Pod.

For example, a public API could be deployed as a Workload, running on a Pod's port 5001. The same pod could also expose an internal API on port 4001. I'm not sure if I should abstract ports into something like "Services." I think I'll have to actually build it, dogfood it for a bit, and then I'll have a better idea of what makes the most sense to the end user.

Exposing the entire Kubernetes API is obviously not a useful abstraction, so I'll need to simplify things.

Last week, I finished the manifest generation for Compute and wired-up a full end-to-end experience of defining Compute in the Console and having that compute be automatically deployed to the Kubernetes Cluster.

Compute Table

The "tiers" are fake for now, but the size is fully working! Dorsal is now running on Compute defined in Dorsal.

The most satisfying part of this was creating the Compute in the Dorsal Console, deleting the manually-created NodePool manifests, and watching as all of my pods automatically migrated from the old NodePools to the ones that I had just created with Dorsal.

It was a bit of a Doctor Frankenstein "it's alive" moment for me.

Kubernetes

So, I know that I'm late to the party on appreciating Kubernetes, but I feel like this past year has taught me the real reasons why it is so powerful - even after 8+ years of working with it.

Rather than choosing to adopt Kubernetes and then fitting everything into it, I've found myself asking "how could I accomplish _____?"

After evaluating, and even building out, things like serverless Lambda functions, playing with ECS (yuck), and considering a standard VPS, I'm finding myself coming back to the realization that the dynamic environment of a Kubernetes Cluster is tremendously well-suited to my needs on this project.

The fact that my Pulumi Kubernetes Operator workspace pods spin up immediately, on new instances provisioned by Karpenter (if necessary), and then disappear when they're complete means that I am not footing a massive compute bill each month.

I find it very intuitive to have these different subsets of the system doing their job, declaratively, and as long as they know how to leave outputs or how to expose ports the way that another subset of the system expects, everything can work together.

For example, pulling in secrets with External Secrets Operator, and using the Kubernetes patterns like valueFrom in Pods or Deployments makes it easy to think about secrets management and secrets access independently. Fundamentally, this pattern of simple(r) subsystems working together really makes a lot of sense to me.

By combining that with GitOps, I can also use git diffs to see the state of my system change overtime. This makes debugging, reverting to previous states, or migrating to new tools much easier than if I were making raw Kubernetes API calls.

About 6 years ago, I had this notion of an idempotent state-management system being very powerful.

Imagine, I thought, if I could fire off events with at-least-once delivery, process duplicates without concern, and always wind up in the correct state. That's, obviously, not a ground-breaking revelation, but I'd somehow only ever seen that sort of thing on a tiny subset of a system I'd worked on up to that point - something like an API endpoint that accepts PUT requests.

But what about a whole system? Even better would be if that system was declarative.

Well, I think that's when I finally realized why Kubernetes is the right tool for this project.

If I always just have to reconcile the database state (as set by the end user) into manifests, and Kubernetes always reconciles the manifests into the active state of the system... It's just declarative, idempotent turtles all the way down.

Until next time,

Nathan