You can follow every enterprise best practice. You can use Clean Architecture, some type of layered architecture, event-driven architecture, microservices, or whatever else is popular.

It can still end up in the same place.

You have a system that is really hard to change.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

When you do make a change, you are afraid you are going to break something. Eventually, the same comment keeps coming up:

We would be better off rewriting this whole thing.

The codebase might not even look that bad. It might be organized and relatively easy to follow. Except it is not easy to follow when you actually need to change it. That is when everything becomes convoluted and complicated.

One of the reasons is probably one of these five architectural decisions. In each case, you are making an expensive decision without having enough information.

That is the common thread between all five. You are making expensive decisions before you truly understand the business or the problems you are trying to solve. Rather than solving an actual problem, you are adding technical nonsense as a solution to a problem you might not even have.

The shift is in the question.

It is not:

What architecture should we use? Should we use Clean Architecture or event driven architecture?

The question should be:

What do we understand about the problem that justifies the architectural decision we are about to make?

1. Choosing an Architecture Before Understanding the Domain

The first bad decision is choosing your architecture, tooling, frameworks, and infrastructure without really understanding the business domain.

If the start of a project conversation is about how you need microservices, event sourcing, CQRS, Kafka, and Kubernetes, but nobody can explain the business processes or workflows, you are doing it backwards.

Can anybody explain the constraints? What consistency issues might exist? What are the possible failure modes? Which parts of the system change frequently? Where are delays acceptable?

If nobody can answer those questions, why are you already deciding on the solution?

Architectural patterns and tooling are not a bingo card. They are not a checklist. Every one of them comes with tradeoffs and added complexity.

You want independent deployability? Now you have distributed operations.

You want to scale parts of the system independently? Now you have network failures to deal with.

You want team autonomy, with different teams managing their own services and deployments? You are going to have cross service and cross team communication. Even if that communication is asynchronous and uses events, there is still a contract that needs to be managed.

You want isolation between boundaries? You are going to have consistency challenges across those boundaries.

There is always a tradeoff. There is always added complexity. You might be getting a solution, but that solution has a cost.

The conversation should not start with whether microservices are good or bad. That is irrelevant. The focus should be on the forces acting against your system and the problems you are trying to solve.

Do you have consistency issues that need to be addressed? Is there a part of the system where the rules change quickly and should be isolated? Does a workflow contain delays that downstream processes need to account for?

Those are the questions you should be asking.

Do not start with, “Should we use this pattern or that pattern?” Use the pattern that fits as a solution to the problem you actually have.

If you are making technical decisions first, you are guessing that you are eventually going to have the problem those decisions are supposed to solve.

2. Building Entity Services Driven by CRUD

The second bad decision is what I call entity services. This is where the system is completely driven by its data model rather than its behavior.

At first glance, the code can look great. It might be well organized. Maybe you have Clean Architecture or some type of layered architecture. You have controllers, services, and repositories that interact with the database.

But the entities are really just tables or object graphs representing customers, orders, products, and other records.

Ultimately, you just have CRUD.

Consider a simple order with an ID, a customer ID, a status, and a total. What does that model tell us?

It tells us that some data exists. It does not tell us anything about how an order behaves.

Can you cancel an order? You would assume so, but how does that happen? Do you just change the status? At what point can the order be shipped? Does it need to be in a particular status? Can it be changed after inventory has been reserved?

The model tells us none of that. It only gives us data.

Where does all the business logic go when the system is driven by CRUD and entity services?

Often, it stays in the end users’ heads. They are the ones who actually understand the processes and workflows.

Some of that logic will make its way into the system, but it is usually sprinkled everywhere. It might be in controllers, message handlers, stored procedures, or increasingly, frontends that contain most of the actual business logic.

CRUD itself is not bad. There are always parts of a system that are inherently CRUD. Referential data and simple lists might have no meaningful behavior behind them. They are CRUD, and they should be treated that way.

The issue is assuming everything is CRUD. It is developing everything as if it were just an updatable record when it is not.

There is a significant difference between an operation called UpdateOrder, where you change properties on an order, and an operation called CancelOrder, where you explicitly communicate that the user intends to cancel an order and provide the reason.

That is not simply CRUD. It expresses behavior and business intent.

The question you need to ask is:

Does the operation I am performing communicate business intent?

In this example, the intent is to cancel the order. Once that intent is explicit, you can start asking useful questions.

Can the order still be cancelled based on how long ago it was placed? Has inventory already been reserved? Has the order been shipped? Does the customer need a refund?

Those questions are derived from the intent of the operation.

When everything is treated as a generic update, you lose that intent. You also lose the natural place where business rules, validation, and workflow decisions should exist.

3. Using the Database Schema as an Integration Point

The third bad decision follows directly from the second. It comes from thinking only about the data model and using the database schema as an integration point.

Imagine two parts of a system: sales and warehouse.

They use a single database instance, but the database is somewhat separated into data related to sales and data related to the warehouse.

Sales interacts with the data it owns. But then it also reaches over and queries data that appears to belong to the warehouse. Potentially worse, it writes directly to warehouse data.

Who actually owns that data?

Is it sales, or is it the warehouse?

At that point, there is no real separation and no clear ownership. The database schema has become the integration point.

Some people will say they do not care. They have one large database, and the different parts of the system need consistency constraints. They want to use the database directly.

That can be fine, as long as you understand the implications. You have no explicit contract. You are treating the database as shared storage.

The problems usually start when you need ownership.

When data is written, who controls how it is written? If a table, collection, or stream needs to change, who owns that change? Who needs to be involved? Which other parts of the system could break?

Without clear ownership, you do not know.

One solution is to have the warehouse expose an API. That API becomes a contract that sales can use to send requests or retrieve information.

A database view can also be a contract. A view can explicitly define how other boundaries are allowed to access data. If you create a view for that purpose and treat it as a contract, it can be a completely valid alternative.

The important distinction is that sharing a database instance is not automatically a problem. Different boundaries can own separate schemas within the same database instance.

The problem is the lack of ownership.

When the database becomes a free for all and anything can read or change any data, you eventually start breaking things.

Suppose an order gets into an invalid state. How did it get there?

You have no idea.

Anything could have changed it. The update might not have gone through the actual process, business rules, and validation required to enter that state correctly.

The order ended up in an invalid state because nobody clearly owned the process of changing it.

4. Creating Abstractions Before You Know What Varies

The fourth bad decision, and arguably my biggest pet peeve, is building abstractions when you do not know what varies.

It usually starts with a reasonable sounding idea.

Maybe we will need to swap something out later.

You begin with a shared service. Then you notice some patterns, so you create a generic workflow engine. After that, you start thinking about what happens if you change the database or messaging library, so you create abstractions around those as well.

Only after all of that do you start building the actual application.

It sounds like it makes sense. We are all taught to value reuse. There is also the constant “what if” question.

What if the underlying provider changes? We will have an abstraction. We can create a new implementation behind it without rewriting large parts of the application.

For example, maybe we want to replace our messaging library. If everything is behind an abstraction, that should be easy.

I understand the reasoning. The problem is that if you only have one implementation of the abstraction you are creating, you probably do not have an abstraction.

You do not have enough information.

You do not understand the other concrete implementations, where they overlap, or where they differ. You are trying to generalize something when you only understand one example.

Messaging libraries are a good example. RabbitMQ and Azure Service Bus have significant differences. Kafka is different again. They might all appear to involve sending and receiving messages, but they have different semantics.

If you start with one of them and immediately build an abstraction around it, that abstraction will be shaped entirely by the only implementation you know.

You did not remove the coupling. You hid it behind an interface.

Are abstractions bad? Of course not.

But a bad abstraction pretends to remove coupling when it does not. It often makes the system harder to understand because developers now need to understand both the abstraction and the concrete technology hidden behind it.

You should create abstractions based on actual variation that you understand, not variation you are imagining might exist someday.

5. Building for Scale You Do Not Have Yet

The fifth bad decision is building for scale you do not have yet.

“Yet” is the important word.

This means paying all the upfront complexity to build for a level or type of scale that you do not know you will ever have.

That does not mean you should design a system that cannot scale if the need arrives. It means you should not pay the entire cost upfront based on hypotheticals.

We might have millions of users.

We might need to replace the database.

This might eventually become a global system.

What do any of those statements actually mean?

When you say the system needs to scale, are you talking about more users, more data, more transactions, or more geographical regions?

“It needs to scale” is not a requirement.

This is not about ignoring scale. It is not about assuming growth will never happen. It is about defining clear boundaries and managing the coupling between those boundaries.

You also need to understand that logical boundaries and physical boundaries are not the same thing.

When you define logical boundaries and avoid coupling them unnecessarily, you give yourself a much better chance of scaling different parts of the system if that need actually appears.

This is also why the conversation about modular monoliths versus microservices is often misleading.

People assume microservices automatically scale better because everything is independent. But a modular monolith and microservices can often be scaled in the same ways.

Once you understand that logical and physical boundaries are different things, the idea that these architectural styles are complete opposites starts to fall apart.

You can define meaningful boundaries without immediately turning every boundary into a separately deployed service.

Start with the boundaries. Decide on the physical deployment model when you have enough information to justify it.

Expensive Decisions Require Real Information

These five decisions cause a lot of pain in software systems.

You make technical decisions first without understanding the domain.

You treat your data model as if it were a domain model, while the actual workflows remain in users’ heads or are scattered throughout the system.

You have no clear ownership, turning the database into a free for all where anything can read or change data anywhere.

You create abstractions without understanding what they are supposed to abstract.

You build for hypothetical scale and pay the cost of complexity before you know whether you will ever need it.

In every case, the problem is the same. You are making an expensive decision without enough information.

Architecture should not start with patterns, frameworks, or infrastructure. It should start with understanding the business, the workflows, the constraints, and the actual problems the system needs to solve.

Then you can make the architectural decision that fits.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Multi-Tenancy Isn’t About Databases
• Stop Blaming Event-Driven Architecture
• Break that big ball of mud

The post 5 Software Architecture Mistakes That Make Systems Hard to Change appeared first on CodeOpinion.

You start off with what seems like the obvious solution to a multi-tenant SaaS application.

We have tenant A. We have tenant B. We have one application and one database. Within that database, for every structure, whether that is a table, collection, or stream, we segregate things by a tenant ID.

It is simple. It is easy. It works.

Until it does not.

What happens when one tenant imports five million records? Or they run a bunch of reports, and some of those reports are massive? That shared infrastructure which seemed simple is also what is coupling everything together.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Shared Infrastructure Means Shared Problems

Let us use another basic example. We have multiple instances of our application in a scaled out environment, but now we need to make a schema change.

The first thing that happens is we make our schema change to the database. Then we do a rolling deployment where we update one of the application instances first. That instance is now updated. Its code matches what the schema is. That is what it is expecting.

Everything is fine.

Uh oh.

What about the second instance that has not been updated yet? It is still making requests at runtime while everything is being deployed. Of course, it is going to fail. It has zero expectation of what that schema change is. It does not even know about it.

If your schema changes are not backwards compatible, your old code and your new code are looking at the database in a different way. They are expecting a certain shape, and the database is not giving both of them what they expect.

Can you go down the road of blue green deployments? Yes.

Could you have canary deployments? Sure.

But you are still under the same circumstance. You need to understand the coupling involved and the trade offs being made.

The First Question Should Not Be Database Per Tenant

Almost always, when people talk about multi-tenancy, they immediately ask:

Should we have a shared database?
Should we have a database per tenant?
Should we have a schema per tenant?

Those are valid questions, but they should not be the first question.

A better question is this:

What are you actually trying to isolate?

Are you trying to isolate data? Compute? Deployments? Schema changes? Performance? Different types of failures?

Because multi-tenancy is not just about databases. It is about creating isolation through boundaries. A database is just one of those boundaries.

Every Shared Resource Creates Coupling

Every time tenants share something, they are coupling through it.

In the database example, yes, you are sharing the schema. You have to deal with shared migrations. You are also sharing the performance of that database.

With your API, tenants are sharing the compute that is actually running it. They are sharing deployments. They are sharing failures happening within that instance or those instances.

The same thing applies if you are using queues and background processing. The same thing applies to cache, what is in memory, invalidation, and the keys used in that cache. It applies to regions, where things are deployed, residency, and latency.

If you are sharing something between tenants, they are ultimately coupled through it.

A Shared Database Can Be Perfectly Fine

Back to the simple example: one application, one shared database, and tenant ID on the rows.

There is nothing inherently wrong with this.

If we have a single instance and a single shared database with segregation by tenant ID, it is simple. It works. When we add a new tenant, it is easy. We do not have to set up a new database. We do not have to roll out some new infrastructure.

We are sharing the schema, and that is a lot easier to manage. It is simpler to deal with. It is probably more cost effective.

Within our code, every time we deal with queries, we have some type of filtering. Depending on the libraries and frameworks you are using, that might be handled by query filters or something similar.

Again, there is nothing inherently wrong with this.

As long as you understand the trade offs.

The Tenant ID Has To Leak Everywhere

The trade off is that this information has to leak everywhere through your system.

It is not a bad thing. It is just the trade off.

If you are doing database access at the row level, the tenant ID has to be in every possible query, or you have to force it through some type of query filter.

Every report that you have anywhere in your system has to filter by tenant. It might not even be application code. It might be somewhere else entirely, but it still has to filter.

If you are using queues and messaging, tenant ID has to be part of every message because you are not segregating everything physically.

Your tenant ID has to be part of everything.

Again, not bad. Just the trade off.

Shared Schema Means Shared Migrations

If you are sharing the database, that means you are sharing the schema. You are sharing migrations.

That means if you upgrade your database, make some type of schema change, and are doing a rolling deployment, it might work for one instance, but it might not work for another if everything is not backwards compatible.

So you need to think about that.

The trade off is that you have to make schema changes backwards compatible. That means having a process. You expand the schema, make the change in a compatible way, deploy the new code, then backfill the data.

If you add columns that are nullable, and the new application code is going to start sending that data, you still have to assume that existing data needs to be backfilled.

That becomes part of your deployment pipeline.

Is it more complexity? Yes.

Is it more work? Yes.

Is it a trade off? Yes.

That is the whole point. You get the simplicity of maintaining one physical database, but the complexity moves into dealing with shared schema changes and rolling deployments.

Deployment Isolation Is Not Schema Isolation

There is also isolation at the application level.

You could have tenant A using version one of your application and tenant B using version two. That gives you isolation at the deployment level, almost like a canary deployment.

This can be totally fine.

Maybe you have a change coming in that you are unsure about. You give it to tenant B because they are a little more relaxed about it. Once you know it is solid and there are no issues, you roll it out further to tenant A or to more tenants.

But if they are still using the same underlying schema, you still have to make everything backwards compatible.

You are providing deployment isolation. You are not necessarily providing schema isolation.

Multi Tenant Architecture Is A Spectrum

When you hear somebody talk about multi tenant architecture, it is really about isolation.

But what kind of isolation?

It is not just the database.

At the database level, you have data isolation. At the API level, you have compute and deployment isolation. If you are using queues and messaging, you have background processing isolation. You might have different queues by priority or by tenant. The same thing applies to caching.

If you are using cloud regions, now you are also talking about residency and latency depending on where things are deployed relative to your tenants.

It is not just about data isolation.

All of these choices are on a spectrum. Depending on your needs, you can be on one side, in the middle, or on the other.

For data isolation, you could have shared tables with a tenant ID. You could have a shared database, but separate schemas. You could have different pools where some tenants are on one database instance and some tenants are on another. Or you could go as far as having a database per tenant.

The same goes for compute. You could have shared instances where everybody is using everything. You could have pooled instances where a subset of tenants uses a subset of compute. Or you could have completely dedicated compute.

You do not necessarily have to fit on one side or the other. It is often a mix and match depending on your needs.

Efficiency Versus Control

The trade off is efficiency versus control.

On one side, you have efficiency. A shared database is simple to maintain. You have one instance. You have one place for backups, migrations, upgrades, and operational concerns.

On the other side, you have control. A database per tenant gives you more control, but that control is not free. There is cost and complexity in maintaining multiple databases, different pools, dedicated compute, or dedicated infrastructure.

The same thing applies to compute.

Shared compute can be very efficient and cost effective. But if you have pooled or dedicated compute for particular tenants, now you are dealing with every particular instance for every tenant or group of tenants.

There is just more complexity involved.

Noisy Neighbors

Here is one reason you might not want to share compute.

Let us say we have tenant A and tenant B.

Tenant A is making requests. Everything is working fine. Happy path. Everything is good.

Tenant B starts making a massive number of requests. Maybe they are importing data. Maybe they are running requests against the database that are really heavy. Maybe what they are doing is CPU or memory intensive.

Now tenant B is affecting the database. They might also be affecting the API.

That is the noisy neighbor problem.

Tenant B is flooding requests to the app or to the database, and tenant A is affected by what tenant B is doing.

Is that a reason to isolate or pool compute differently? It could be.

But there are also alternatives.

Rate Limiting Is One Tool

One alternative is rate limiting.

That means every tenant has their own scope. It does not necessarily mean we have to separate compute and have some compute sitting idle, not doing much.

Both tenants could still use the exact same underlying instance of the app or API, but we rate limit by tenant ID as part of the request.

That can help solve the noisy neighbor problem, but it does not necessarily solve it entirely. Depending on the request being made, you might save the API, but that does not mean you are saving the database.

Rate limiting is just one tool.

You could do the same kind of thing with database connection pools and group those by tenant. Again, they are just tools.

What you really need to identify is what resources are being exhausted.

Tenants Are Not All The Same

The type of system you are building and the customer base you have matters.

At the beginning, tenants might seem like they are all the same, but they probably are not.

Some tenants might have a small number of users. Larger tenants might have a much larger set of users. Some tenants might run heavier workloads. Some might import more data. Some might run more reports.

Over time, you might realize that you need to pool small tenants together, pool large tenants together, or move very large tenants into dedicated infrastructure.

Again, it goes back to mix and match.

At first, you might think all tenants are the same, but they are not.

When Should You Isolate?

This leads to the actual decision you need to make: what should you isolate, and when?

You isolate when you need a smaller blast radius. You isolate when you do not want noisy neighbors at a given level, whether that is compute, database, background processing, or something else.

You isolate when you have tenant specific requirements. That could be compliance. You might have a requirement that a tenant needs to be isolated at the database level for compliance reasons.

You isolate when a tenant needs dedicated capacity. Maybe they do not want to share resources because they care about availability or performance guarantees.

You isolate when you want safer rollouts. Maybe you want to provide one version to a subset of tenants, and then roll it out later to more sensitive tenants or a larger pool.

It is isolation at various levels because you want to avoid noisy neighbors, reduce the blast radius, and have protection against specific types of failures.

Multi-Tenancy Is About Boundaries

Multi-tenant architecture is not a database pattern.

It is about isolation.

The real question is: what should be isolated?

Sharing gives you efficiency. It is usually simpler and more cost effective.

Isolation gives you more control. But that control comes with more cost and more complexity.

So if you are building or living in a multi-tenant system, do not start with “shared database or database per tenant?”

Start with what you are actually trying to isolate.

Then decide where sharing makes sense, where isolation makes sense, and what trade offs you are willing to take.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Resilience Patterns Can Make Your System Less Resilient
• Stop Blaming Event-Driven Architecture
• Building Mutlitenant Systems with CosmosDB

The post Multi-Tenancy Isn’t About Databases appeared first on CodeOpinion.

You wanted your system to be resilient, so you followed the standard advice.

You added retries.
You added circuit breakers.
You added fallbacks.

But now your system is less resilient.

Not because those patterns are bad. They are not. The problem is they are doing exactly what you told them to do.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Retries are great for network blips and transient issues. But the tradeoff is that you are adding more load to a system that might already be struggling.

Circuit breakers are great because you do not have to keep hammering something that is already down. But in a distributed system, everything might be sitting in its own silo on each instance.

Same thing goes for fallbacks. They can provide a better experience because you fall back to something that you know is reliable. But they can also hide the underlying issues that are occurring.

All of these patterns are useful. So are the libraries that implement them. But you do not get resilience just by applying them or using some library. It does not work that way.

In the wrong context, you can actually make things worse.

The question is not, “Can I apply this pattern?” or even, “Should I apply this pattern?”

The question is, “What is the failure mode?”

The Example

Here is the example I am going to use.

We have some type of message broker queue interacting with a worker. That worker ends up making an HTTP call to some third party API.

But this has nothing to do with queues specifically. It has nothing to do with HTTP APIs or brokers specifically. It is just an example to illustrate that one pattern for resilience can work great in one context and be a total disaster in another.

Imagine a worker consuming a message from a queue. The message is something like ShipOrder.

The worker needs to make an HTTP call to a shipments route or some third party service. Maybe it is UPS, FedEx, some carrier, whatever the case may be. You need to create the shipping label in their system and get the response back.

Once you know that is good, maybe you do some internal state change in your own database, marking the shipment label as created.

That is the happy path.

Consume a message.
Call a third party API.
Update some internal state.

That code would be incredibly simple on purpose, because it does not show what really matters.

What really matters is understanding things operationally.

How many messages are you processing? What is your throughput? Is everything in the same queue? How many worker instances do you have concurrently processing messages? What happens if that third party HTTP API is down? What happens if it is just really slow compared to what it normally is?

The code only illustrates the happy path. It does not illustrate what you want to deal with when failures occur or what is acceptable in your context.

Standard Advice Is Incomplete

If that HTTP API was a little flaky, the standard advice might be to just slap on a retry and a circuit breaker. Maybe even a fallback.

So you add a retry policy around the HTTP call. You add a circuit breaker. Good to go.

That is not necessarily bad advice. It is just incomplete.

And it can have consequences that affect your overall system negatively. Not more resilient. Less resilient.

The resilience patterns or policies you build up in code can only happen after you understand the failure mode and all the different parts of your system that are affected when a failure happens.

But “failure” is not really one thing. I am not even sure it is a good name for it.

Something could be down. Straight up down. It is not responsive. Something could be slow or degraded. You could have intermittent issues that are transient, where one request works and one does not. A service could be overloaded because you are hammering it, or because other services are.

It could be a partial issue where you make a request, maybe it times out, but it actually is working on the other side. You are just not getting a response. Or you might have no idea what the heck is going on with it.

But all of that gets put under the label of “failure.”

That is the problem. Not every failure type deserves the same resilience pattern. One pattern can work for one type of failure and be a disaster for another.

Down Is Not The Same As Slow

Take the difference between something being straight up down versus something being really slow.

If something is down, it usually fails very quickly.

You try to make the HTTP call and you get a 503. Or connection refused. It happens immediately. It is easy to detect because it is happening right away.

In the case of our worker, the worker is released quickly, so it can start performing other work. Maybe it removes that message, throws it into a dead letter queue, however you have that configured, and then it can start processing another message.

If you are using a circuit breaker, it can open rather quickly.

But what if the call is not failing fast? What if it is just slow? What if it is slower than you expect on the happy path and it just hangs there?

That is much harder to classify. Is it degraded? Is it slow? Is it partial? You are not really sure yet.

And there is a big difference for our workers.

You are occupying them.

If something that normally takes milliseconds to process is now taking 10 seconds or 30 seconds, your worker is sitting there waiting. It is occupied. It cannot process other work.

Because of that delay, it may take you much longer to realize that failures are actually happening.

Retries Are Not Free

Now take the standard practice of adding retries.

Maybe you have a shared policy that you use for all HTTP calls to third party services. There is a little bit of backoff to it.

If there is a failure, maybe you wait 500 milliseconds and retry. If the failure continues, you wait 2 seconds. Then maybe 10 seconds. Then 30 seconds. Finally 100 seconds. After that, you give up.

But retries are not free.

If this was not a transient issue, not just a blip, that retry is ultimately costing every job more time to complete. That means every worker processing messages that needs to make that call is now taking longer.

And it can get worse. Because not all failures are the same.

What happens if the third party API is just really slow? You process the message. You call the API. It takes 30 seconds. Finally, it fails. Maybe you have a timeout. It returns some error.

Guess what you do now? You wait 500 milliseconds and try again.

How long is that going to take? Potentially another 30 seconds. And you keep going with this.

So something that you thought might take a couple of seconds because of retries can now take minutes, depending on how long it takes that third party API to time out.

Another way of putting this is that retry policies are not just about reliability. That is why you use them, but everything has tradeoffs. The tradeoff is that you are now consuming a resource.

In this example, that resource is your worker. You are consuming that worker for the length of time that you are applying the retry policy.

The Resource You Are Actually Consuming

With our retry policy, what resources are we affecting? How does it affect them? Is that something we can deal with?

Say we have four instances processing messages off our queue. We have retries configured, but the third party API is slow.

Each instance is now stuck sitting there for seconds or minutes, depending on the policy. Each one is processing a message that requires that HTTP API, but the call is not succeeding.

Now the broker is getting backlogged with queued messages because nothing can process anything else.

In that exact example, a lack of isolation is the problem.

If we have a shared pool of workers, maybe we do not want a shared pool. Maybe we want specific queues. Maybe our queue design should determine which workers are working on which queues.

Is retry the answer? It depends on the type of error. If it is transient, maybe you do want to retry. If it is timing out, maybe that is a different failure type and you do not want to retry.

Local Decisions In A Distributed System

What can be problematic is trying to make that distinction when you are also in a distributed environment.

Maybe you have multiple workers. Each one has its own local view of the world.

A great way of dealing with this is metrics. Publish metrics about the failure types so you can react to them accordingly across workers and build your policies around that.

You want to know what is happening. Say there is a temporary issue with the third party HTTP API and it is failing. You might not want everything to be localized to each instance.

You can publish a metric that says, “Hey, this failed,” or “This timed out,” or whatever the case may be.

Then you can get those metrics pushed back to your workers, or your workers can look them up to decide what to do.

Should we continue calling?
Should we adjust our retry limits?
Should we stop calling that dependency entirely for now?

You can be very runtime specific about how you want to handle failures.

Because if you have four different workers and there is a timeout with that third party system, they all have to experience it enough times to hit their own circuit breaker.

But if you are recording metrics like failure rate, timeout rate, retry rate, and all of your instances have a global view, they can react faster.

They can decide, “I am not going to that third party service because it is timing out.”. That helps messages move through your system more predictably.

The tool does not matter. It could be CloudWatch. It could be Prometheus. That is not the point. The point is that you are signaling to your system that something is degraded and how you want to handle it across all instances.

What Should You Actually Do?

Whether something is local or systemwide is part of the process of figuring out what you should do.

If there is a failure, did you reach some kind of failure rate? Timeout rate? Latency threshold?

What is the risk? What is the impact on other resources? There are always other resources involved that are going to be affected by it.

If it is localized, maybe you retry with backoff. Maybe you time out and fail fast. Maybe you develop your own timeout because the built in timeout for the HTTP client is not what you want. In C#, for example, the default HttpClient timeout is 100 seconds.

Do you really want to use that? Maybe you use a local circuit breaker that only applies to that particular instance.

If it is systemwide, maybe you want a shared circuit breaker based on metrics. Maybe you want a global fallback. Maybe you want to fail over to another service. Maybe you want to rate limit load or change how you scale things.

And if it is workflow or data related, what are the implications of something failing?

Do you need to move it to a dead letter queue?

Can you kick this off later?

Do you need to trigger something else?

Is there some kind of compensating action you need to perform? Some type of rollback, if you want to think of it that way?

Or do you mark the workflow as failed and say, “We are done. It is at this point. Somebody needs to look at it.”

Maybe that means an email. Maybe it means escalating through issue tracking. Maybe it means alerting someone.

How you decide which flow to go down depends on what resources are affected and whether this is localized to a particular queue worker or something systemwide.

The Patterns Are Still Useful

Retries are great for handling transient faults.

Circuit breakers stop repeated calls so you do not keep hammering the same service that is already failing. Timeouts bound waiting so you do not let something go on forever, like waiting 100 seconds for an HTTP call when that makes no sense in your workflow.

Fallbacks enable degraded behavior. It does not need to be perfect. It can just get you by so your system does not come to a crumbling halt.

But the question is not which resiliency library you should use. The question is not which patterns you should be applying. It has to start with the failure mode.

What is the failure type? And if you apply this pattern, what is the tradeoff? Because it is going to affect something. It is going to have an effect on other resources.

In the retry example, the retry policy affected the worker. It consumed more time. Because it was retrying, it could not process more messages.

There is always a tradeoff. There is always something that you are affecting.

You can still control the blast radius. Even if you apply a pattern that has a tradeoff, you can design around that tradeoff and localize the impact.

But you have to design around your context.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Stop Blaming Event-Driven Architecture
• Solving the ‘God Object’ Problem with Shared Identity
• Change Your Architecture During Deployment

The post Resilience Patterns Can Make Your System Less Resilient appeared first on CodeOpinion.

So, you adopted event-driven architecture because your system was a rat’s nest of coupling, and events were the answer to decouple it.

But now debugging is a nightmare.

You have events coming in out of order. You have retries causing duplicates and multiple different side effects. Local development is a pain. It’s frustrating, right?

But we use events for a reason.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Events can help us reduce temporal coupling within our system. We can have a publisher publish events for downstream services or other parts of our system that consume those events, process them how they need to, when they need to.

The producer doesn’t care who’s consuming them.

There’s real value there around workflows, integrations, and scaling, when you need it. There’s real value there when you have the right problems.

But here’s where things start going wrong. You think everything needs to be independently deployable. Or that everything has to be asynchronous. Or you start publishing events that aren’t really business facts.

And when you really start to feel the pain of this, that’s when you start blaming event driven architecture.

But events aren’t really the problem. Misunderstanding is.

The Pain Is Real

I ran across a post titled something along the lines of, “Our event driven architecture created more problems than it solved.”

As I was reading it, I thought, okay, this is somewhat relatable.

But then, as I continued, I realized that no, this was really poor design. Things related to debugging, local developer experience, retries, ordering, and a lot of the problems they were having, they really didn’t need to be having.

I’m not here to tell you event driven architecture is a silver bullet, because it is not.

It has complexities. There are trade offs.

But the pain they were experiencing came from the wrong conclusions.

This Is Not Eventual Consistency Hell

The first pain point they brought up was what they called eventual consistency.

Their example was sales and orders being placed in their system, and they had different parts of the system that needed to do different things.

Shipping needed to create a shipping label.

Inventory needed to reduce the quantity on hand.

Payments needed to charge the customer’s credit card.

The issue they mentioned was, what happens if there’s a failure?

Before, in a monolith, they said they could just roll back the transaction. But now they needed all these compensating actions. For example, they had to publish a different event so inventory could undo what it did, or shipping could undo the label creation.

And if it was in a monolith, they said, you would just roll back the transaction.

I’m not so sure it’s actually that easy.

They called this eventual consistency hell.

This isn’t eventual consistency hell. This is workflow modeling hell.

When they mention rolling back a transaction, in this exact use case, it doesn’t really work that way.

If you’re interacting with some shipping system to create a label, typically that’s going to a carrier. That is not something you just roll back in your database.

Same thing if you have some type of payment issue or timeout. Did it actually go through? You might need to reconcile that.

There aren’t just multiple systems where you can roll everything back together and magically have some distributed transaction. That’s not likely going to happen.

This is more of a modeling problem.

You should be modeling actual business workflows using events that the business actually cares about.

Events Should Represent Business Facts

Events represent business facts. Things the business actually cares about and refers to in a particular way.

Not CRUD.

One example from the post was that a customer places an order, so they publish an OrderCreated event.

But was that really what happened? Was the order submitted? Was inventory reserved? Was a payment authorized?

Those are actual business events. Those are things that are really happening within a boundary.

Naming matters.

Don’t get too caught up in the exact domain here, but I want to illustrate the point. There isn’t just some massive rollback of all of this because you probably wouldn’t want that anyway.

If a payment failed, you’d probably be contacting the customer.

You don’t have to roll back inventory because maybe you didn’t actually reduce it to begin with. That’s not when it happens. You might be reserving inventory instead.

There are completely different business cases related to inventory, payments, shipping, and ordering. The example was framed as, “We had all these compensating actions and we were better off with one giant rollback.”

But you probably wouldn’t have had one giant rollback to begin with.

Asynchronous Does Not Mean Everything Is Eventually Consistent

Another pain point they mentioned was performance. Their API response times went from 150 milliseconds to 2.5 seconds. Why? Network hops.

They mentioned a customer saying, “I updated my profile, but the change doesn’t show up for 30 seconds.”

And the response was basically, “That’s eventual consistency for you. Eventually.”

Hang on. What?

The conclusion they landed on was that asynchronous isn’t fast. It can be fast, but it doesn’t have to be. The better question is, why is a user updating their profile and having to wait for anything?

Not everything has to be asynchronous.

If a user is trying to change their profile information, that should be synchronous. It should be immediately consistent, and it should be something the user immediately sees.

You want to be able to read your own write.

Does that mean you can’t have events involved? No. You absolutely can.

You can have different boundaries that care about when a profile has its phone number updated. Maybe CRM cares because you have some external CRM that you keep updated. Maybe analytics cares. Maybe some other part of the system cares.

Whatever handles the profile update can publish an event.

It doesn’t really care what other parts of the system care about that event. Those parts can be asynchronous.

That’s a good use case for asynchronous processing.

But updating the information so the user sees it immediately? Why would that be asynchronous?

It should be synchronous, just like you’d expect it to be in any other type of system.

Kafka Being Down Is an Infrastructure Problem

Another issue they had, and I’ve heard this before, was the single point of failure.

They mentioned a Kafka cluster incident where everything depending on events stopped working.

Which was everything.

They couldn’t create users. They couldn’t process orders. They couldn’t update inventory.

In the monolith days, they said, if the database went down, everything stopped, but at least they had one thing to fix.

Here was the glaring thing they mentioned: With Kafka down, they couldn’t even fall back to synchronous processing because they had removed all that code.

Well, first, we already established that not everything should be asynchronous.

Second, if you have vital infrastructure, you deal with it appropriately. Like anything else.

It’s no different than their database example. Do you want your database to be down? No. If it is down, everything is down.

If it cannot be down because not everything can be down, then what do you have?

You have monitoring. You have high availability. You have failover. You have backup processes.

This is no different than any other piece of infrastructure.

Same thing with a cache. If you have a cache and it goes down, there’s an outcome to that. Often, that outcome is hammering other parts of your system, like your database, because your cache is gone.

So you end up making the cache highly available too.

It’s no different than any other part of your infrastructure that you require high availability for.

Having said that, with messaging, depending on the libraries or infrastructure you’re using, it’s common to have some kind of local mailbox where messages are kept in case they can’t be published. The outbox pattern is one example.

Yes, there is complexity.

But you’re often using this to make the system more resilient.

Event Driven Architecture Does Not Require Microservices

There were two other pain points that were very related.

They mentioned the hidden costs nobody talks about, especially infrastructure costs.

Before, with their monolith, they had two application servers, one PostgreSQL database, and a cache.

With event driven architecture, now they had six microservices, a Kafka cluster, six different databases, a service mesh, and all these different costs.

They also mentioned the developer experience.

With the monolith, you clone one repo, run Docker Compose, and everything is good.

With their event driven architecture, they had all these different repos. You had to install Kafka, Zookeeper, all these databases locally. It became a nightmare to deal with.

These are really the same issue.

The issue is assuming event driven architecture has something to do with physical deployments, different repos, and separate infrastructure.

It doesn’t.

You can use event driven architecture within a monolith.

You can have a monolith with different boundaries like customers, orders, billing, inventory, and notifications. That can all be deployed in a single instance. Or multiple instances if you are scaling out. It can use a single PostgreSQL database or whatever database you’re using.

But each boundary owns its particular schema.

Physical boundaries are not logical boundaries.

I will keep saying this every possible chance I get until people figure it out.

Event driven architecture does not require six different repos, six different database servers, six different deployments, and Kafka all over the place.

You do not need that.

You can be using event driven architecture within a monolith.

You do not need microservices.

You do not need Kafka.

It’s about publishers and consumers. It’s about decoupling in time.

That’s the point of event driven architecture.

Not all the infrastructure.

Messaging Can’t Fix Bad Boundaries

The pain they felt was real. That’s why they said event driven architecture caused more problems than it solved.

But the reality is that misunderstanding and misapplying event driven architecture is why they felt that pain.

The better lesson is this:

Messaging can’t fix bad boundaries.

I think this is why people experience all this pain and then blame event driven architecture.

My solution is to start with logical boundaries.

Start by defining what your boundaries are.

From there, choose how those boundaries communicate.

Is the communication synchronous?

Is it asynchronous?

Then choose how you’re going to deploy.

Are you deploying one particular boundary by itself?

Are you deploying everything together?

That’s the order.

1. Define logical boundaries.
2. Choose communication patterns.
3. Choose deployment model.

I think most people do this in reverse, and that’s how they get into trouble.

They start by thinking, “We need to deploy this independently for scale.”

Then because of that, they decide they have to always communicate through events or always communicate asynchronously somehow. Or they introduce HTTP APIs everywhere and now they have more network hops.

Then, after all that, they try to decide what should go where.

That’s backwards.

Start with logical boundaries.

Then choose how they communicate.

Then choose how they deploy.

In that order.

Events Are Not the Problem

Event-driven architecture is not the problem.

Events have complexity. Messaging has complexity. Asynchronous workflows have complexity.

But a lot of the pain people attribute to event-driven architecture comes from misunderstanding what it’s actually for.

Not everything needs to be asynchronous.

Not every event is a business event.

Not every boundary needs to be independently deployed.

Not every event-driven system needs Kafka, microservices, six databases, and a service mesh.

Use events when they solve the right problem.

Use synchronous communication when that is what the workflow or user experience requires.

Model the business process correctly.

Define your logical boundaries first.

Because events can help reduce coupling, but they cannot fix bad modeling.

And messaging can’t save you from bad boundaries.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Solving the ‘God Object’ Problem with Shared Identity
• Modular Monolith Boundaries Done Wrong
• Change Your Architecture During Deployment

The post Stop Blaming Event-Driven Architecture appeared first on CodeOpinion.

You start with a simple entity. Then, over time, you add more and more properties. The next thing you know, you have a god object sitting at the center of your system. Everything touches it. Everything depends on it. Every workflow flows through it.

And whenever you need to make a change, you hope it doesn’t break something else.

At some point, you start wondering: how did we end up here?

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

For a lot of systems, this approach makes perfect sense early on. A shared model/godobject is simple. You don’t have to worry about duplication. You don’t have to synchronize data across different parts of the system.

But as the system grows, the trade-offs start showing up.

The problem isn’t that different parts of the system care about the same thing. The problem is assuming they care about the same model.

Let’s walk through a practical example where multiple parts of a system can function without a god object, without shared data, and without shared behavior.

What they share is identity.

The Shipment That Became Everything

I’ll use a Shipment as the example, but don’t get hung up on the domain. This could be a Customer, Account, Order, Reservation, or whatever large entity you’re struggling with in your own system.

At first glance, that seems reasonable. It’s all related to a shipment, right? Not exactly.

This is what happens when we group things by data rather than by behavior. If you look at any large entity, it’s easy to say, “Well, it’s all related to a shipment.”

But that completely ignores behavior. And once you ignore behavior, things start going sideways.

Different Parts of the System Care About Different Things

Let’s start with dispatch.

Dispatch is responsible for executing shipments. It assigns drivers, trucks, trailers, and carriers. It cares about routes, stops, pickups, deliveries, and the workflow involved in moving freight.

Its capabilities define the data it owns.

It doesn’t care about everything.

Now look at compliance.

Then there’s billing.

These are all different concerns.

They all care about a shipment, but in different ways. They don’t share a model. They don’t share behavior. They don’t even necessarily share data.

What they share is identity.

Once you realize that, you stop trying to build one model that rules them all.

Workflow Makes Ownership Easier to See

Workflow is often a much easier way to understand ownership than looking at data.

Let’s look at a typical shipment lifecycle:

At first glance, that looks like one giant workflow. It isn’t.

It’s really a series of workflows connected by handoffs.

The sales side of the system books the order, enters details, rates it, and makes it available.

That’s a handoff to dispatch. Dispatch assigns resources, tracks execution, records arrival and delivery events, and completes the shipment.

That’s a handoff to billing. Billing audits paperwork, creates invoices, and posts to accounting.

Each workflow belongs to a different part of the system. Different responsibilities. Different rules. Different ownership.

So why would they all share the same model?

Sharing Identity

At some point, a shipment gets created.

For example, an Order Dispatched event might occur.

That’s enough. The event is notifying other parts of the system that a business fact occurred.

What it is not doing is distributing all shipment data.

One of the biggest mistakes people make when they start separating models is turning events into data distribution mechanisms.

Events shouldn’t be dumping grounds for data. Events should communicate business facts.

“But I Need the Data for My Screen”

This is usually where the conversation goes. People want to display information in a UI, so they assume they need to replicate data everywhere. You don’t.

Imagine shipment-related information spread across multiple boundaries:

Each owns different data. That doesn’t mean you need one giant model containing all of it. You can compose information for query purposes.

This is a read concern.

You can pull together information from multiple sources to display a screen without distributing all of that data throughout your system.

Just because you need to show information together doesn’t mean it needs to live together.

What About Business Decisions?

Eventually, you’ll hit a situation where you need data from somewhere else in order to make a decision.

That’s when you need to ask a very important question:

Can the decision tolerate stale information?

The answer drives the solution.

Example: Driver Availability

Suppose dispatch needs to know whether a driver is available.

A driver might be out of service because they’re on vacation, suspended, or unavailable for some other reason.

The Assets boundary publishes an event:

Dispatch subscribes to that event and keeps the information it needs locally. Now dispatch can make scheduling decisions independently. There are no synchronous calls to the Assets boundary. Dispatch has the information it needs.

Is it stale? Potentially.

Is it good enough for the decision being made? Probably.

That’s what matters.

Sometimes You Need an Authoritative Answer

Not every decision can tolerate stale information. Imagine billing is about to post an invoice to accounting.

Before posting, it needs to know whether the accounting period is still open.

Maybe you could maintain a local copy. Maybe not.

In this situation, it might make more sense to make a synchronous call to the accounting system and get the current answer.

Even then, there’s no guarantee the information won’t change immediately afterward.

That’s just reality in distributed systems.

The point isn’t to eliminate uncertainty. The point is understanding the trade-off. Can the decision tolerate stale information? If yes, local data and events might be enough. If no, you probably need a more authoritative source.

Maybe the Data Lives in the Wrong Place

Sometimes the real question isn’t how to get the data. It’s whether the data should live where it currently lives at all.

If another part of the system constantly needs information to make decisions, maybe that ownership boundary isn’t right.

Before reaching for replication, APIs, or events, ask who truly owns that information.

Sometimes the answer is that ownership should move closer to where the decisions are actually being made.

Shared Models Aren’t Always Wrong

Can you use a shared model (god object)? Absolutely.

If your system is small and the workflows are simple, it’s often the easiest solution. The pain shows up when systems grow. When workflows start ending and triggering other workflows. When different parts of the system develop different responsibilities.

That’s when separate models start paying off.

The most important takeaway is this: Just because you need to display data together doesn’t mean you need to distribute it everywhere.

Different parts of your system can care about the same thing without sharing the same model. What they often need is shared identity, not a shared model.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Modular Monolith Boundaries Done Wrong
• Stop Joining Tables In Your “Modular” Monolith
• Design more decoupled services with one weird trick

The post Solving the ‘God Object’ Problem with Shared Identity appeared first on CodeOpinion.

So you built a modular monolith. You have a clean structure. Different projects. Everything broken into modules. But somehow, when you make a change, it still ripples through the rest of your system. Why? Because its highly coupled.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

As an example, let’s say you have an HR module. You change an employee’s work status, and all of a sudden, that affects authentication.

Or you’re in a transportation system, and you change a vehicle’s compliance status, and somehow that affects shipment planning.

You defined the modules. So what happened?

You defined boundaries, but you were likely only thinking about boundaries in one way. More specifically, not everything revolves around data alone. And logical boundaries are not physical boundaries.

When you define your modules, you might think you’ve split up your system. But if you still have all this coupling and all these dependencies, did you really split up ownership?

Logical Boundaries Are About Ownership

Typically, when people think about modules in a modular monolith, they’re not that far off from how they think about services and microservices.

They’re usually thinking about the physical aspect.

A process. A database. A queue. A deployable unit. A container. Some network boundary.

But what we’re after, regardless of whether it’s a module in a modular monolith or a service in microservices, is a logical boundary.

We want to think about ownership over behavior.

Who owns the rules? Who owns the lifecycle? Who owns the meaning of something? Who has authority over it?

Physical boundaries do matter. They can enforce isolation and define how modules communicate with each other. For example, maybe you define a public API that determines how one module communicates with another.

But just because you define a physical boundary does not mean it’s the right logical boundary.

And that’s where coupling shows up.

You make a change in one module, and it ripples through the rest of the system because the ownership was never really separated.

Logical boundaries are all about ownership. Who owns the data? Who owns the business rules? Who owns the invariants you need to enforce? Who owns the lifecycle and workflows that happen in the system?

And maybe more importantly, who owns the concept of something?

A module in a modular monolith should be a logical boundary. A service in microservices should also be a logical boundary.

If that’s not making sense, it’s because a logical boundary is not a physical boundary. How you deploy is a different concern.

The Same Word Does Not Mean The Same Concept

A great way to figure out where logical boundaries are is through language.

The same word does not mean you’re talking about the same concept.

Here’s an example.

Let’s say we’re talking about a shipping system where we might have logical boundaries for compliance and dispatch.

A vehicle is not a vehicle.

What I mean by that is, in compliance, you have things like vehicle registration, vehicle insurance, and vehicle maintenance.

But in dispatch, you have the same vehicle, yet it’s a different concept for a different need.

In dispatch, you care about availability. Can you dispatch it to a shipment so it can go to the pickup? What drivers are seated in that vehicle? What’s the capacity of the trailer? Can you get another shipment on that vehicle?

They’re the same vehicle, but they’re not the same vehicle.

It’s the same word. It’s the same thing in the real world. But it’s a different concept. It has a different meaning. It has a different boundary.

If you had only one concept of a vehicle that everything needed to use, guess what happens?

You end up with coupling.

That’s how you make a change to vehicle and affect dispatch, compliance, and billing. Because vehicle is not a singular concept.

I often call this an entity service. You have a single entity that exists in the system, and you try to share that concept across different parts of your business. But those different parts of your system should own their own concept of a vehicle.

An Employee Is Not An Employee

Here’s another example that everybody can probably relate to.

Think about the concept of an employee.

You might have modules for HR and IT.

In IT, you might not even think of this person as an employee. It’s more like a user. That user has authentication, roles, and access to applications.

In HR, that employee means something very different. HR cares about compensation, benefits, onboarding, and employment status.

IT and HR might be talking about the same physical person, but they are entirely different concepts. Each module, each logical boundary, cares about that person in a completely different way.

Now you might be thinking, “But if I only had one model, one source of truth for employee or vehicle, isn’t that better? I don’t have duplication.”

But now you’re coupling two different concepts together into one.

That means if something needs to change in one module because there’s a reason to change there, it can affect another module that has absolutely no reason to change.

That’s the coupling.

One reason to change in one boundary is not the same reason to change in another.

For example, in a shipment system, one boundary might see a vehicle as a compliance asset. That’s the reason it changes there.

Another boundary related to dispatching sees the vehicle as shipment capacity. That’s the reason it changes there.

Not both.

Same thing with employee.

In HR, it’s an employment record.

In IT, it’s a user account.

In a medical system, a patient might be a clinical record in one boundary. But in billing, that same person is a billing account.

They’re not the same reason to change. And that’s another great way, beyond language, to think about where boundaries lie.

Ownership Is Not Integration

There’s a difference between ownership and integration.

Ownership is about having the internal model, the concept, how you define it, and what you expose to other boundaries.

Integration is how other boundaries learn about what they need.

When you lose that distinction and create some shared concept that all boundaries use, you lose the boundary you had to begin with.

You can still integrate. You can expose an API. You can publish events. You can create projections or read models.

But that’s very different from saying, “Here’s one shared model that every module uses.”

Because once every module depends on the same model, one change affects everything else. That’s when everything ripples through your system.

Logical Boundaries Are Not Physical Boundaries

I need to keep enforcing this idea because it matters.

Logical boundaries are not physical boundaries.

If you fundamentally understand this, you’ll look at the last 15 years of microservices, and now modular monoliths, and wonder why we talk about them as if they’re the same type of thing.

They’re different concerns entirely.

Let’s say you have a monolith with logical boundaries A, B, and C. They exist in the same source code repository, and you have a build process that turns everything into a deployable unit.

That’s your modular monolith.

The problem with microservices is there was a tendency to think this way:

“I have a logical boundary. It has its own source code repository. That turns into a deployment artifact. Then each boundary gets deployed separately and communicates over the network.”

Even if those source code repositories were in a monorepo, the mental model was still often:

Logical boundary equals source repository equals deployment unit.

But that’s not the case.

Your boundaries are whatever you define around behavior and ownership. They can be spread across frontend, backend, and infrastructure.

You’re really taking a slice out of what your entire system does.

That means you could have a logical boundary that has source code that turns into two deployable units. Maybe you have a web API and a background processing job that handles messages off a queue or event log.

You could also have a logical boundary that is spread across repositories. Maybe that’s frontend and backend, and they get deployed separately.

Or maybe you have a service that has two source repositories, but the build process puts them into the same deployable unit or container.

A great example is a backend service that also has a mobile app. The mobile app gets built into an APK or AAB for Android, or an iOS app, and maybe that mobile app includes functionality from multiple services.

Logical boundaries are not physical boundaries.

And that also applies to your database.

Database Ownership Still Matters

You might have a database instance with tables owned by one particular boundary. That does not mean another boundary cannot use the same physical database instance.

Maybe two logical boundaries use the same underlying database instance, but each owns its own schema.

Then another boundary might use a completely different database instance.

That’s fine.

The important part is ownership.

Who owns the schema? Who owns the tables? Who owns the data behind the behavior?

Physical separation can help enforce boundaries, but it does not define the boundary by itself.

You Still Have Coupling

You’re going to have coupling between modules.

That’s reality.

The question is how you want that coupling to exist and what you want to expose.

Is it through an API? Through a projection? Through a read model? Through events?

There are different ways to deal with coupling, but it becomes very different when you define ownership clearly.

That’s the difference between having boundaries that own concepts and having one shared model that every module is coupled to.

With one shared model, when one thing changes, everything else is affected. Everything ripples through your system.

Start With Ownership

If you’re defining a new system and trying to create boundaries, start with ownership.

If you’re working in an existing system and the boundaries don’t feel quite right, start with ownership.

Who owns the concept? Who owns the business rules? Who owns the behaviors? Who owns the data behind those behaviors and capabilities?

That’s the logical boundary.

A vehicle is not a vehicle. An employee is not an employee.

The same word does not mean the same concept. And if you miss that, you’ll end up with modules that look clean physically, but are still tightly coupled logically.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Stop Joining Tables In Your “Modular” Monolith
• Scaling Software Architecture Without Overengineering
• Designing UI for Microservices

The post Modular Monolith Boundaries Done Wrong appeared first on CodeOpinion.

Modular monoliths are all the rage.

You have well defined modules built around business capabilities. Maybe inside those modules you are using something like Clean Architecture. You have separation of concerns. You have direction of dependencies. Everything looks great and you think, “Wow, finally I have a really good structure.”

But do you?

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Modular Monlith

You are looking at an order management system, viewing a particular order, and somebody wants to see the quantity on hand in the warehouse for the items on that order.

That seems harmless. Straightforward. Since you are in a monolith, you have the same database. No big deal. When you get the order line items out, if you are using a relational database, you just join with the inventory table.

Problem solved.

Except what looked like a simple answer to show some inventory on a screen may have just welded two modules together.

Your codebase can be lying to you. Even though you have this well structured solution with different projects around modules, it might really just be a nice looking wrapper on top of a very coupled database.

Your code might say Sales and Warehouse. But what do your queries say?

The Query That Crosses the Boundary

You might think it is architectural nonsense to hear “don’t cross a boundary,” especially in a situation like this. You are in a monolith. Why would you create two different queries when you can just have one with a join?

What is really the harm?

That is the trap you might not realize you are putting yourself into.

Jumping back to the order detail page, the request seems simple enough. Add another column that shows quantity on hand from the warehouse.

To do that, it is going to be pretty simple, right? Sales is building out the page. Sales gets the order details and the line items. From there, if we are using a relational database, we can just join to the inventory table wherever that lives as part of Warehousing.

Sure, they are separate things, but just go to the data directly. That is the tempting version.

I say that because it is simple, easy, and probably really fast to get that data and render it with a simple join. But architecturally, something important just happened.

You decided to couple Sales and Warehouse at the database level.

Now it is not just a query. You have a dependency.

That is the hidden coupling. You do not see it in code. Everything can look well structured, but the coupling is hidden inside your queries and how you are dealing with data access.

I know a lot of people immediately think, “Yes, this is why my system is a disaster. It is a rat’s nest.”

Because there is a free for all of coupling and integration at the database level. You can have some level of ownership and organization within code, but if you are not doing that at your database schema level, all bets are off.

Just Put an API in Front of It?

A solution you might be thinking of is to create an API. Sales interacts with that API to Warehouse to get the inventory.

That way, Warehouse defines and uses its explicit schema, and so does Sales. Sales has to do the composition of getting the information from Warehouse through the API.

Sure, that solves a problem.

But you still have coupling. The form of coupling is just different. Before, you were coupled directly to the database. With an API, you are coupled to a contract.

That gives you more options. Maybe you have versioning. Maybe you can deprecate some APIs or mark something as obsolete. You have different ways of changing the behavior of how things work.

For example, say the requirements change and quantity on hand is stored differently. Now you have different warehouses, and you are showing which warehouse has what quantity on hand in what location.

If you were integrating directly into the database, you are handcuffed. That is especially true if you are not in the same codebase and cannot easily find all those queries.

Imagine you have some other codebase somewhere else that is also integrating into your database. Now you cannot change it. You are totally handcuffed.

With the API, at least you have some means to migrate. You have versioning. You have some course of action if you want to evolve and make change.

What Do You Actually Need?

There is more to it than that, because sometimes what you think you need and what you really need are two different things entirely.

I actually love this example because it highlights that.

If we are talking about needing the actual quantity on hand in the warehouse when looking at orders, maybe it is just for view purposes. Maybe it is also for placing an order, where you think you need the quantity on hand.

The thing is, that is not reality.

Sure, you may think this is a trivial example, but I can guarantee you that in your own business domain, this happens all the time.

With Sales, you might think about a particular SKU and the price for the product, or what you are selling it for. Purchasing might care about buying that product from a vendor and what the actual cost is. The Warehouse cares about that SKU, the quantity on hand, where it is in the warehouse, what bin it is in, and whatever else matters physically inside the warehouse.

But the idea of quantity on hand in Sales is not actually a thing.

It is not.

It is a totally different concept. It is a business function called Available to Promise.

Available to Promise is solving the actual issue, not necessarily from a technical perspective, but from a business perspective.

Sales does not actually care how many items are in the warehouse at any given time, or what the system thinks is there, because that is not reality.

Reality is going into the warehouse and seeing what parts or products are actually there and in what quantity. Something might be damaged. Something might be stolen. What is in y

our system is not reality. What is physically on the shelf is reality.

So something like ATP, Available to Promise, is a business function. It is the idea of, “What have we sold right now? What have we purchased from our vendors or manufacturers that we are going to receive? What number can I actually promise to sell because I know more are coming in?”

That is the number Sales cares about.

This goes back to the difference between what you think the requirement is and what you actually need.

When you model that properly, you may not need the coupling. You may not need to call some API to get another boundary’s data. You may not need to reach into its database or do a join or cross that boundary line at all.

You have all the data you need because you aligned everything with the business functions and the requirements within the boundary.

It Is All Tradeoffs

This is all about tradeoffs.

There really is no right answer, per se, because it comes down to what is right in your system.

What is the size of the system? How much is it evolving? If today it is just one extra query, no big deal. Maybe you have to do a little composition. Maybe it is a little bit slower. If that helps you tomorrow deal with coupling, then sure.

But if you have to deal with schema changes, and things are evolving, and you cannot change the schema because all these different things are coupled to the same underlying database, that is a problem.

There is your tradeoff.

How are you going to deal with regressions? If you change something, are you going to break other processes or application code? If you want to make a change, what different applications or parts of the same system do you have to coordinate with because you are changing the underlying schema?

There is no universal right answer here. It is a matter of the tradeoffs and the degree of coupling you are creating.

Which Camp Are You In?

I think there are probably two camps.

One camp is thinking this is insane. Sales and Warehouse? It is just a simple warehousing system. I can do the join. It is one database. It is a monolith. No problem.

And that may be working totally fine for you, depending on the size of your system and what else is accessing that database.

If it works, it works.

However, there are large enough systems where you absolutely need boundaries because you cannot evolve without them.

I can guarantee there are a lot of people living in those types of systems, where they are handcuffed because they cannot make any schema changes at all. They have no idea what application code or what queries are happening anywhere that are calling those tables.

You cannot make a change because you are going to break something else.

In those situations, you want to create an API because it is your contract. It is something you can evolve, at least a little bit easier than when clients are reaching directly into your database.

That is much more difficult to change.

Your Code Might Be Structured, But Your Data Might Not Be

If you just want to make that query and do that join, have at it. But realize the tradeoff you are making and the coupling you are creating.

Your application code might be well structured. It might look like it does not have much coupling. It might look like a modular monolith.

But if all the coupling is happening at the database level, what you still have is a big ball of mud.

It just has a nicer looking code structure wrapped around it.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Loosely Coupled Monolith – Software Architecture – 2025 Edition
• Why “Microservices” Debates Miss the Point
• Finding Service Boundaries, Illustrated in Healthcare

The post Stop Joining Tables In Your “Modular” Monolith appeared first on CodeOpinion.

Your system has 500 users and also has 20 microservices, including the gauntlet of Kubernetes, a message broker, distributed tracing, multiple different databases, and a pretty dashboard that probably nobody looks at.

Someone will ask the question: why is this so complicated for 500 users?

And usually the answer is, “Well, we’re building for scale.”

No. No, you’re not.

You’re building for scale you don’t have yet. That’s a lot of extra complexity.

Scaling software architecture is important to think about. But there’s a difference between actually implementing for scale and giving yourself the options to scale when you need it. You can build a path forward without paying all the upfront costs and complexity.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Scale Means Different Things

The word scale has a lot of different meanings, and that causes a lot of confusion.

You might hear things like:

“We need to split into microservices so we can scale independently.”

“We need to add a message broker so we can scale more background work concurrently.”

“We need to add a read model or a separate database because we have a read heavy system and a different shape of data.”

All of these are talking about scaling, but they’re not talking about the same thing. They all have different utility, and they all come with different tradeoffs.

That’s why the word scale by itself is so vague.

Are we talking about user scale, in terms of the number of requests, where you want to horizontally scale instances?

Are we talking about splitting into microservices for team or organizational reasons, where different teams have ownership over different boundaries?

Are we talking about database scale, read scale, write scale, or different shapes of data?

Are we talking about deployment scale, meaning how different parts of the system are deployed and how each part needs to scale differently?

Scale is just too vague on its own.

Every pattern has tradeoffs. I’m always trying to explain what the problems are, what the solutions are, and what the tradeoffs are, because sometimes the cost of complexity just is not worth it.

Patterns Are Great When You Actually Have the Problems

Microservices are useful for independent scale of code, teams, and deployments.

A message broker is useful for background work, asynchronous processing, buffering, reliability, and dealing with work that does not need to happen immediately.

Multiple databases can be useful for read and write scale, performance, or when the shape of data you mutate is different from the shape of data you need to query.

Kubernetes can be useful for infrastructure scale.

But all of it has costs. All of it has tradeoffs.

If we’re talking about microservices, most people know the list. You add a network boundary. You now have contracts, depending on how those services communicate with other boundaries. You have deployment, versioning, observability, failures, retries, timeouts, and the list goes on.

Messaging has a lot of the same concerns. Retries. Timeouts. Ordering, if you need it. Duplicates. Poison messages. Workflows. There is a lot of complexity in messaging.

Different databases have their own complexity too. What kind of replica is it? Are you using replication? Is there eventual consistency or lag? Are you creating a different shape of data where your application has to route between the read database and the write database? Do you have drift because of application code?

And Kubernetes has its own complexity. Everybody knows that.

There are complexities and tradeoffs. Don’t add them if you don’t need them yet.

So how do you build for scale so you can scale when you actually need to, without paying all the upfront costs?

Logical Boundaries Are Not Physical Boundaries

Let’s start with microservices.

A lot of people confuse logical boundaries with physical boundaries.

Say we have a monolith. We have modules A, B, C, and D. A is coupled to B. B is coupled to C. C is coupled to D. We have a rat’s nest of coupling, but it’s all inside one monolith.

Now take that same system and remove it from being a monolith. Each one of those modules is now a service or a microservice.

Are we less coupled?

No.

We’re still just as coupled. Actually, it might be worse now because we introduced network boundaries. We’re not magically decoupled because these things use HTTP to communicate with each other, or because they happen to use a message broker.

All we’ve really done is take a highly coupled monolith and turn it into a highly coupled distributed monolith.

That’s it.

If you take that source code and spread it apart into different repositories, that does not change anything. What we’re really after is defining logical boundaries.

How you deploy those boundaries physically, whether as a monolith or as different services, is a different concern.

What we’re after is logical boundaries and avoiding a big ball of mud.

A Service Does Not Have to Mean One Repository and One Deployment

When people think about microservices or services, they often think each service does something, has its own source repository, and gets built into its own deployable unit.

Service A has a repository and a deployment. Service B has a repository and a deployment. Service C has a repository and a deployment.

But that does not need to be the case.

In a monolith, you can still have different logical boundaries. Those boundaries can live in the same repository and be deployed as one deployable unit.

This is really the 4 + 1 architectural view model. You have logical views, development views, physical views, and process views. Not everything has to be one to one to one.

This is why I like the idea of a loosely coupled monolith.

The Loosely Coupled Monolith

Imagine a system with three different logical boundaries. Each boundary defines contracts. If this is inside a monolith, those contracts might just be DTOs, interfaces, delegates, or function definitions.

You might still have one database instance. But within that database, you can have specific schemas or tables owned by a particular boundary.

That means you already have segregation between what your code does, what feature sets it owns, and what specific data it references for read and write purposes.

There is not a tangled web where anything can reach into the database and talk to anything else. Boundaries talk to each other through APIs when they need to invoke actions related to data owned somewhere else.

If you build this way, there is nothing stopping you from changing the physical boundaries later.

Maybe one logical boundary suddenly needs to be deployed separately because it needs to scale differently in terms of requests or users. The other two boundaries can still be bundled together. You can still have a single database. The logical model does not have to change just because the physical deployment changed.

Again, logical is not physical.

Adding Asynchronous Processing Later

You can take this a step further.

Maybe you need to introduce asynchronous processing because you need to do more work in the background. You can introduce a broker for that.

Maybe messaging becomes a primary way you communicate between boundaries. If some interaction needs to occur, there could be an event that happened or a command that needs to be processed. You send that to the broker, and another part of your system consumes it.

That can still happen inside the same monolith. One boundary can produce a message, and another boundary can consume it, even though physically they are deployed together.

Then, if you split things apart later, you are still using the broker to message between those same logical boundaries. This could still all be in the same repository. It could be deployed together or deployed separately.

Logical is not physical.

A monolith does not have to be a rat’s nest of coupling. You can have defined logical boundaries and decide how you want those boundaries to communicate.

The same is true with microservices. Yes, you can have a distributed monolith because you have a rat’s nest of coupling. But it does not have to be that way if you define your boundaries well and are intentional about how they communicate.

Building a Path for Scale

This is how it relates to scale.

You’re not ignoring scale. You’re building a path forward depending on how you need to scale.

If you have well-defined boundaries and later need to separate something for independent deployability, you can do it.

If you need to separate something for team or organizational reasons, you can do it.

If a particular part of the system needs to scale differently for requests, users, or background work, you have a path.

That does not mean you need to build out the full scaled architecture before you actually have the problems that require it.

Messaging Does Not Fix Bad Boundaries

Another one that comes up a lot is asynchronous processing with a message broker or something like a distributed log.

Messaging has a ton of value and solves real problems.

It gives you temporal decoupling, where two different systems, or even two different boundaries inside the same monolith, do not need to be online or available at the exact same time.

It gives you buffering, where you can have messages going into another system and processed as load ebbs and flows.

It can improve reliability because you have durable messages sitting in a queue or topic, depending on how you are processing them.

One of the things I love about messaging is integrations, especially when you need to deal with external systems. Something happens in one part of your system, you need to react to it, and you need to talk to another system.

Messaging also helps with asynchronous workflows. In most business systems, that’s really what is happening. Something happens, then something else happens, then there is some handoff. There are a ton of workflows in business systems.

But there is also a lot of complexity that comes with messaging.

You have to deal with idempotency because duplicate messages are going to happen. You have timeouts, backoffs, retries, poison messages, ordering, and workflow coordination.

There is a lot to deal with. You need to have enough value to justify that complexity.

If you have a distributed monolith making HTTP calls from service to service and you decide, “Well, this is not reliable enough, so we’ll move to something asynchronous,” sure, you may improve reliability in some areas.

But you still have the same coupling.

Adding messaging, a message broker, an event log, or event driven architecture does not solve bad boundaries.

Coupling Is Still Coupling

Ultimately, it does not really matter whether you have a monolith with in-process communication, HTTP between services, a message broker, or events.

If you are temporally coupled, you have to coordinate everything between those parts.

If you are coupled by data, whether by events or by a database, you still have to coordinate everything because changes have to be understood by everybody who depends on that data.

If you are behaviorally coupled between services, where one service needs to know what to invoke on another service because that service performs some action, you still need to coordinate everything.

The mechanism does not remove the coupling.

For scale, it is about building a path.

That does not mean building out a full scaled architecture before you actually need to scale. It means defining good boundaries and understanding the distinction between a logical boundary, what something actually does, and the capabilities it provides, versus the physical boundary of how you deploy it.

That distinction matters.

Define the boundaries so you can build a path for scale. Don’t pay all the costs before you have the problems.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Loosely Coupled Monolith – Software Architecture – 2025 Edition
• Why “Microservices” Debates Miss the Point
• What Starbucks can teach us about software scalability

The post Scaling Software Architecture Without Overengineering appeared first on CodeOpinion.

So, you built an elaborate system with commands, queues, an event-driven architecture, retries, timeouts, and, most importantly, compensating actions all handled within sagas.

But do you really?

Because then you get a call from support. There’s an order where the payment is pending, and it’s been pending for 48 hours. You look into it and see that the payment provider did charge the customer, but your system shows that the payment didn’t go through.

So which is true?

Clearly, the payment provider.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

The Mess of Payment Logic

The first and obvious answer is, “Let’s just make it smarter.” We already have different code paths handling different use cases. Maybe we need to add another retry. Maybe, in this example, there was a timeout. The payment provider timed out, we didn’t get a result, but the payment actually did work.

So we think, “We just need to make it smarter and add more logic.”

But that’s exactly how this turns into a mess.

You start with something simple. Capture the payment.

But if there’s a timeout, maybe you need to check again to see if the payment actually went through. Or maybe there’s a payment provider exception. What was the reason for it? Was it a duplicate? Then you need to check the provider because you know you already sent this payment request, or at least you think you did for that exact order.

If it’s an unknown error, maybe you need to flag it for review.

Now you’re handling all these different cases, edge cases, and flows directly in your code. The logic starts spreading everywhere because you’re trying to understand every possible thing that could have happened across a network boundary.

And you might think a saga is the answer to this problem.

It’s not exactly.

What a Saga Is Actually Good At

A saga is great for coordinating workflow, but it is not the source of truth.

It can tell you:

• What step am I on?
• What message should I receive?
• What command should I send next?

Backing up to a checkout flow, a saga is a great fit. It demonstrates the workflow. That’s where you understand the process:

1. An order is placed.
2. Inventory is reserved.
3. Payment is captured.
4. The order is confirmed.

That’s coordination. That’s where sagas fit really well.

But there’s a giant gap here.

The saga is not the point of truth for an external system like your payment provider.

The Missing Piece Is Reconciliation

What you really want alongside a saga is reconciliation.

You want to know:

• What should be true?
• What is actually true?
• What corrective actions or compensating actions can you safely apply?

Anytime you’re dealing with a network boundary to a third party you do not control, you’re going to have uncertainty.

Did the call work?
Did it time out but still work behind the scenes?
Did the provider process the payment but your system never receive the response?

This is where reconciliation comes in.

A saga and reconciliation go hand in hand.

A Checkout Flow With a Saga

Here’s an example of a checkout flow in a saga, and it’s incredibly simple, as it should be.

First, we handle the OrderPlaced event. From there, we reserve inventory.

Once inventory is reserved, we handle that event and try to capture the payment. At the same time, we set a timeout for the payment capture.

If the PaymentCaptured event occurs, then we can mark the saga as complete and confirm the order.

But if the timeout happens because we never got the PaymentCaptured event back, then after 15 minutes we send a request for payment reconciliation.

A couple things to note.

The timeout does not mean anything failed.

It just means we stopped waiting.

We had an expectation that the payment captured event would occur within a certain amount of time, say 15 minutes, and it didn’t. So now we need to handle that situation.

Also notice that there are no compensating actions here about reversing inventory or cancelling the order because of a failure.

Because we don’t know that we have a failure.

A Timeout Is Not a Failure

To visualize the flow, the saga sends a capture payment command. That command is received, but now we’re waiting on the third party payment provider.

The payment provider never responds to us.

Eventually, we hit the timeout.

But because we have that timeout, our saga fires after 15 minutes and requests payment reconciliation.

That reconciliation process can then ask the payment provider, “Did you actually process this payment?”

And if the payment provider says yes, then we can update our order status because we now know the payment isn’t pending anymore. From the payment provider’s perspective, everything worked.

This isn’t a saga problem.

This is a data drift problem.

It’s about the point of truth.

And that’s where reconciliation comes in.

What Reconciliation Looks Like

Reconciliation should be simple.

We get the order and check its current status. If the order is already confirmed or cancelled, we can exit.

But if it’s still pending, we go to the point of truth. In this case, that’s the payment provider.

We ask the payment provider for the actual payment status.

If the payment was processed successfully by the payment processor or gateway, then we mark the payment as captured and confirm the order.

If the payment actually failed on the provider side, then we mark the payment as failed and cancel the order.

That’s it.

The pattern is:

1. Know there is drift.
2. Go to the source of truth.
3. Compare it against your data.
4. Apply a safe action.

In this example, the way we knew there was drift was with a timeout. But there are many different ways to trigger reconciliation.

Different Ways to Trigger Reconciliation

A saga timeout can work great, but it doesn’t have to be the only trigger.

It might be something as simple as adding a button that says “Check payment status” so an end user or support person can invoke it themselves.

Polling might not seem like the greatest option, but it might be a good option for your system.

For example, you could have a background service that runs every five minutes, looks in your database for pending payments older than 15 minutes, then iterates through them and requests payment reconciliation.

That could be the trigger.

It really can just be polling.

Reconciliation Is Not a Cleanup Job

Reconciliation is not a cleanup job.

It’s about consistency.

I get the sense that people sometimes feel like, “Well, I have to run this job because of a timeout and reconcile with some third party system, but everything should magically always be consistent.”

It won’t be.

There’s nothing wrong with doing reconciliation.

It’s not some dirty cleanup job that means your system is broken.

Sagas are great at coordinating workflows. Reconciliation is about verifying that what you think the state of the system is actually matches the real state of the system.

Because there are so many things that can fail.

Trying to shove all these use cases, logic, and edge cases into sagas, or other parts of your system, to handle every possible failure just isn’t going to work.

You don’t need to add all this branching logic every time something gets invoked, like capturing a payment.

You can reconcile.

You can do it safely.

Is something wrong? Yes? Then perform some type of safe action.

Sagas Coordinate. Reconciliation Verifies.

A saga helps coordinate the workflow. It knows what step you’re on, what message you’re waiting for, and what command should happen next.

But it does not know the truth of an external system.

The payment provider does.

So when your system and the provider disagree, don’t keep adding more and more logic into the saga to guess what happened.

Go ask the source of truth.

That’s reconciliation.

And if you’ve felt this pain before, where something in your system is trying to do too much and has all this logic for edge cases, you probably didn’t need more branching.

You probably needed reconciliation.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Debugging Event-Driven Systems: 5 Problems Teams Create
• Goodbye, long procedural code! Fix it with workflows
• Sagas Solve Stupid Timeout Problems

The post Why Sagas Feel Broken appeared first on CodeOpinion.

A post on Medium was shared with me by a member of my channel. A team went all in on event-driven architecture, and now they feel like they can’t debug anything. I get it, debugging event-driven systems can seem challenging.

But it’s not directly because of event-driven architecture. It’s because of their misunderstanding of it and how they were applying it.

So let’s break down the five pain points they had, why they had that pain, and how you can avoid it.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Where They Started

They started with what they called a REST based system, which really just means HTTP in most cases. They had services calling other services.

A client would make a request to Service A. Maybe Service A hits its database. Then Service A calls Service B to do something external. Then Service A calls Service C, which likely has its own database. Maybe Service C calls Service D, and eventually everything unwinds back to return a response to the client.

If that sounds like a disaster, well, it was.

They described it as a spaghetti mess of cascading HTTP calls, versioning nightmares between contracts across services, and a brittle orchestration layer.

That pain is real. But moving to events does not automatically fix it. In some cases, it just moves the confusion somewhere else.

Problem 1: Tracing Ended at Kafka

The first problem was tracing, specifically in an asynchronous workflow.

In their existing HTTP based system, they had something like this: the front end called the ordering service, the ordering service called the payment service, the payment service called the email service, and with OpenTelemetry and something like Zipkin or Jaeger, they could see that full flow.

With Kafka, however, they said that visibility completely broke. The order service produced an event, then the payment service, email service, inventory service, and other services listened to specific topics. But now they could not see what was actually happening as part of the workflow.

And yes, 100%, this is a pain.

You need good visibility and good tracing in any distributed application. Absolutely.

However, OpenTelemetry works just as well in an asynchronous system as it does in a synchronous system using HTTP between services. This should not be an unsolved problem.

To me, this sounds like a tooling problem.

They likely went from a more mature web framework where tracing just worked out of the box, to using Kafka or another messaging tool where they did not have the same level of instrumentation wired up.

I see this often when teams use the SDKs directly for whatever messaging system, event log, or message broker they’re using, rather than leveraging a messaging library that has tracing built in.

This is a solved problem. It just might not be solved in the tooling they chose, or in how they implemented it.

Problem 2: Dead Letter Queues Became Garbage Dumps

The second problem was that dead letter queues turned into garbage dumps.

They sound helpful. A way to isolate failed events. But in reality, they became a graveyard of mystery failures.

Dead letter queues are great, but they are just storage.

Imagine you have a queue and a consumer picking up messages. That consumer starts processing a message and needs to call some external service. That external service is failing. So you have some kind of backoff and retry. It keeps failing. Eventually, you put that message into a dead letter queue.

Later on, maybe manually, you pick that message back up and retry processing it.

That’s fine. But a dead letter queue without any way to inspect the message, replay the message, or see what errors and exceptions happened while processing the message is just a black hole.

It’s not useful by itself.

You need visibility into why the message is failing. You need to correlate the message with the actual error or exception that occurred.

Do you need to fix something in your code? Is it a poison message that is never going to work? Was there a transient failure when calling an external service, and if you call it now, it will succeed?

You need visibility into the failure and the message itself. Without that, a dead letter queue is just a place where problems go to be forgotten.

Problem 3: Fire and Forget Equals Debug and Regret

I liked this title from the post: “Fire and forget equals debug and regret.”

They made an analogy that REST is like ordering at a restaurant. You place an order and wait for confirmation.

Uh, you sure about that?

I often use a restaurant to explain asynchronous workflows, but in the opposite way.

You arrive at a restaurant and someone greets you and seats you at a table. They don’t stand there and wait for you to order. They go back and perform another action, like greeting somebody else and seating another table.

The wait staff takes your order and puts it into a point of sale system so the kitchen understands there is something they need to do. The kitchen picks that up asynchronously and starts cooking your meal.

When the order is ready, there is probably another message to let the wait staff, or whoever has that role, know that the order is up and needs to be brought back to the customer.

It’s full of asynchrony. You just need to embrace it properly.

The biggest red flag in their post was very telling, because I see this over and over again. They were forcing everything to be an event and everything to be asynchronous.

It does not all need to be events.

They described it like throwing a note into a room and hoping somebody reads it. But that’s not really the point of events. What they’re describing is publish and subscribe, and even then, they’re missing the important distinction.

Events are facts. Something happened.

If you publish an event, you as the producer do not care if there are any consumers. There might be zero consumers. There might be a thousand. You don’t care.

You also don’t care if a downstream service did its job. That’s not the point of an event. You’re not asking another service to do something. You’re saying that something already happened.

That’s where commands and events are different.

Commands and Events Are Not the Same Thing

Commands are about invoking behavior. They are owned by the consumer because there should be a single boundary responsible for handling that command.

There could be many different senders, depending on how you view your boundaries, but the command is handled by one consumer. Commands are named with verbs because they represent actions you want to invoke.

Events are totally different.

Events are facts. Something happened. You are notifying some other part of your system that this thing occurred.

Events are owned by whoever publishes them, because the event occurred inside that specific boundary. There should only be one publisher of that event. For consumers, there may be none, or there may be many.

Events are named in the past tense because they represent something that has already happened.

Commands and events have different utility. They are very different things.

Not everything needs to be asynchronous, just like not everything needs to be an event. Some things are naturally synchronous, like queries.

In their original HTTP system, they were calling different services to perform different actions. Of course those calls were synchronous. The mistake is assuming the fix is to turn all of that into events.

You have to understand what can be fire and forget and what is not best served that way because you immediately need a response.

There is also the option of doing asynchronous request reply when you need it.

The way that works is you have a separate reply queue. You make a request where you need to know the response. Did it succeed? What was the outcome of that processing?

Service B processes the message. Once it completes, it sends a reply message to a separate queue that the original service can consume. That original service can then correlate the reply back to the original message and understand the outcome.

The author of the post mentioned they did this, except they seemed to do it all the time, which they probably didn’t need to.

Again, it comes back to understanding the interaction you actually need.

Problem 4: Eventual Consistency Is Like Eventual Accountability

Another title I loved from their post: “Eventual consistency is like eventual accountability.”

Let’s say an order flows through four services.

The order is placed. Inventory is reduced. Payment is charged. A confirmation email is sent.

If any step fails, you need to compensate.

Yes. Absolutely.

If payment fails, maybe you need to release the inventory. If something else fails, maybe you need to refund the payment or prevent the confirmation email from being sent.

This is a real problem. Coordinating what happens next when part of a workflow fails is not simple.

In an asynchronous world, the saga pattern is often the solution to this. It helps coordinate a long running business process and track what has completed, what has failed, and what compensating action needs to happen next.

But here’s the thing: this is not automatically easier in a synchronous world.

Think about a long procedural codebase full of nested try catch blocks calling different services. If this call succeeds but the next one fails, then do this. If that fails, do something else. Now compensate for the previous thing. Now handle the failure of the compensation.

That can become a nested mess too.

The benefit of doing this asynchronously is that you can have dead letter queues, visibility into what happened, and state behind the saga that tells you whether the process is complete or not.

But again, we come back to visibility and tooling.

If you don’t have visibility, you’re going to feel lost no matter what communication style you picked.

Problem 5: Testing Became a Nightmare

The fifth problem was testing.

They said unit tests were fine, but integration tests became a nightmare. Now they needed to spin up embedded Kafka, mock consumers and producers, and wait for asynchronous side effects.

That pain usually comes from trying to test everything across boundaries.

Remember, events are contracts. They are not conversations. They do not go both ways.

That is the distinction again between events and commands. Are you trying to invoke something, or are you trying to tell some other part of your system that something happened?

From the producer’s point of view, the test is this: are you publishing the message you are supposed to publish?

From the consumer’s point of view, the test is this: are you consuming, handling, and processing that message correctly?

You do not need every test to spin up the entire world and prove every service talks to every other service through Kafka.

That does not mean you never have broader integration tests. But if every test requires the whole distributed system to be running, then testing is going to be painful regardless of whether you are using HTTP, Kafka, RabbitMQ, Azure Service Bus, or anything else.

Event Driven Architecture Is Not Your Whole Architecture

Event driven architecture is not your architecture. It is part of it.

It is not a full system replacement where you replace all your HTTP calls with asynchronous events. That’s not the case at all.

There is a combination of synchronous communication, asynchronous communication, events, commands, request reply, queries, and more. You need to pick the right style based on the outcome you are trying to accomplish.

It’s not about blindly following a pattern.

If you take a tangled synchronous system and convert everything to events without understanding commands, boundaries, ownership, tracing, failures, and compensation, you probably did not solve the problem. You just made a different version of it.

The issue is not event driven architecture by itself.

The issue is applying it everywhere, treating every message as an event, losing visibility, and not having the tooling or understanding to operate the system.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Is Event-Driven Architecture Overkill for Most Apps?
• Testing Needs a Seam, Not an Interface
• Using PostgreSQL as a Message Transport

The post Debugging Event-Driven Systems: 5 Problems Teams Create appeared first on CodeOpinion.

I’ve noticed a trend, and a lot of people are saying the same thing: just use Postgres as a queue. No Kafka, no Redis, no RabbitMQ, just one database for everything.

And I totally get it. I get the appeal. There are fewer moving parts. There is less infrastructure. There is only one thing to run.

But what feels simple at the very beginning can often lead to a lot of complexity later.

It’s like using Excel when you really need a database. Sure, it holds data. You understand Excel well. But are you really about to build a relational engine on top of it?

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Postgres as a Queue

So, could you use Postgres as a queue? Sure. That can work. But Postgres, Kafka, Redis, RabbitMQ, guess what? They are not interchangeable. They are not even close to the same thing.

Postgres obviously is a data store. Kafka is a distributed log. Redis is a data structure store. RabbitMQ is a message broker. They all store data, and they all store data differently. But that’s not the issue.

The issue is that they expose different semantics, and those semantics are what drive the trade offs.

So guess what that means? You have to add the semantics yourself. You have to add the behavior and the complexity that comes with it.

If You Want to Use Postgres as a Queue

So we’re talking about using Postgres as a queue. Well, it’s not a queue. It’s a data store. But can we leverage it like a queue? Sure.

But a queue is not the product. A queue is just a set of behaviors and things that you’re going to need to think about if you’re going to leverage it as a queue. And this is where the complexity comes in.

These are the kinds of things you need to start thinking about. What happens when a message is taking too long, or something occurs and it stalls out and never gets processed? Well, that’s your visibility timeout.

How about if there is a failure? Do you want to retry? Do you want to retry immediately? Do you want some kind of backoff so you can retry later if it’s more than just a transient issue?

What about dead lettering? There’s just a failure. There’s something completely wrong and it’s not going to work. You just want to put that message in a dead letter queue. How are you implementing that?

Ordering. Do you need ordering? Ordering is a tough scenario, especially when you’re talking about concurrency and using the competing consumers pattern. And of course, if there is some type of failure later on, maybe it is in your dead letter queue, how are you dealing with replay?

So if you want to use Postgres as a queue, you absolutely can. But guess what? That’s where the complexity lies. You have to add the semantics on top of it. You do not get them for free.

Just because you store a row in a database does not make it a queue. You have to add the behaviors.

The Implementation Tax

So here’s what I’ll call the implementation tax that you’re going to need to pay.

We’ll start with visibility timeout. What happens typically with a broker is when a consumer starts processing a message, there’s going to be some invisibility timeout. Let’s just say it’s 60 seconds.

That means the consumer has 60 seconds to acknowledge back to the broker that, yes, it fully processed the message. If it does acknowledge that, then that message is removed from the broker so another consumer doesn’t get it.

However, if it doesn’t, if that consumer does not acknowledge back to the broker in that 60 seconds, then another consumer can pick up that message.

Now for implementation, you might just think, okay, we’ll just put a timestamp on when the message actually got delivered to the consumer and when it may be acknowledged. But is that really going to be that simple? You might have some contention there.

Hang on though. Let’s keep going.

Competing Consumers

The competing consumers pattern is ultimately what allows you to scale out and process more messages concurrently.

So we have a message come into our broker, or into our actual database. We may have one consumer pick up that message because it has no work to do. Then we might have another consumer, another instance, another thread, whatever the case may be, that’s available. It can pick up another message and process it concurrently at exactly the same time.

This just allows you to scale out.

But how do we implement that?

How we can implement it in Postgres is we start a transaction, then we select from our queue table. Let’s say we’re just doing one row at a time. The magic here is the FOR UPDATE SKIP LOCKED.

Really what we’re doing is we’re locking our records, and then we’re skipping any other rows that are already locked. We process our message, everything’s good, then we can delete it from our queue, then we commit our transaction.

That can work.

The Trade Off With Polling

Now, for the invisibility timeout though, you might have noticed I didn’t do anything, because another solution is just to do it in application code. Have your thread, or whatever it is, be terminated after a particular time, which ultimately kills your connection or your transaction after your 60 seconds.

But the issue here is your consumers need to poll the database.

So you’re really in this trade off that you have to deal with. If I poll too often, is that going to hurt my database in terms of performance? Or if I don’t poll that often, does my throughput and my latency, in terms of how quickly I process messages in that queue, go way down?

So there’s this balancing act that you need to find.

Now, you can use LISTEN/NOTIFY, but that’s really just like saying, “Hey, go check out the table. You got to do something.” And if you’re using competing consumers, then you’re notifying all of them.

The Complexity Doesn’t Go Away

Now, I can keep going on with the complexities of retries, dead letter queues, backoffs, and so on. I gave you the list, and there’s a lot more.

Those are things you’re going to have to deal with.

But there are benefits.

One Big Benefit: No Outbox Pattern

One of the biggest benefits that’s going to be noticeable is that you don’t need to have the outbox pattern.

The way the outbox pattern works is because you have a dual write problem. Typically, you have a broker and a database separately, and you want to reliably publish messages along with your state. But you don’t have that problem because your queue is in the same database.

So typically what would happen here is we have our state, we’re saving it to our database, something happened, and then we want to publish a message to our broker, to our queue, for something to get processed asynchronously.

This is the dual write problem.

Guess what? You don’t have this problem, because when you’re persisting your state, you don’t have another connection or some other infrastructure like your message broker or your queue. You’re persisting your state and your message in your queue within the same transaction.

So reliable publishing of messages, you got it out of the box.

The Questions I’d Be Asking

So here are the questions.

If you’re going to use Postgres, or really any database, as a queue alongside your business data, these are the questions I’d be thinking about.

What’s the workload? Can it handle it? Do you know it can handle it?

What kind of retention do you need?

What kind of throughput do you need?

Is polling going to work? Is it not going to work? Can you have some other means?

How well do you understand the database, like Postgres, that you’re using? Can you really leverage those skills?

Do you know all those semantics that you’re going to need to apply on top of it?

Know the Trade Offs

So my answer isn’t, “Well, you should never use Postgres as your queue.” That’s not the case at all.

Really, my answer is: know your trade offs, and know when it’s a viable option and when it’s not.

Postgres, Kafka, Redis, RabbitMQ. Very different tools.

Can you use them as a queue? Sure.

Are you adding a lot of complexity along the way? Possibly, depending on what semantics and what behaviors you need.

One Big Note

There is one big giant note to all this, which is if you’re going to use some type of library, a messaging library specifically built on top of this, that kind of provides all the semantics and all the patterns for you so that you don’t have to go down the road of implementing it yourself, that’s the way to go.

There are some, especially in the .NET space. There are a lot of them that can be used as a transport, as a message transport, for persisting your messages in a database.

So I’m totally with it.

Again, I’m just saying realize the trade offs that you’re making when you’re using a tool that wasn’t specifically designed for being a queue.

Postgres as a Queue

So yes, you can use Postgres as a queue.

The point isn’t that you should never do it. The point is that a queue is more than just storing rows in a table. A queue comes with semantics and behaviors, and if you use Postgres for that job, you’re the one who has to provide them unless you’re using a library that already does.

That’s the trade off.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Is Event-Driven Architecture Overkill for Most Apps?
• Testing Needs a Seam, Not an Interface
• Using PostgreSQL as a Message Transport

The post Just Use Postgres as a Queue? appeared first on CodeOpinion.

In my last video, I said that clean architecture was killing your velocity. And man, you guys had thoughts. The number one pushback, the hill a lot of people were willing to die on, was testing.

And I get it. On the surface, it sounds reasonable. Create some interface or abstraction for testing purposes. But that assumption starts to fall apart once you look at what testing actually needs.

Testing needs a seam. It does not automatically need an interface.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

The typical example

I want to walk through a real example. Not something contrived. Something I just ran into. And honestly, it’s so stereotypical that if you had a coding agent generate it, it would probably come up with code very close to this.

Let’s say I have a VIN decoder. You pass in a VIN, a vehicle identification number, and I want back the year, make, and model of the vehicle.

So what’s the first thing people do?

Obviously, we need an interface, right?

So now we have an IVinDecoderService. And yes, I’m joking a bit here, because every time I use an LLM it wants to create everything as a service. But I digress.

Then we have the actual VinDecoderService, which takes an HttpClient. It’s simple. There’s one method that makes an HTTP call to some API, then takes the result and deserializes it into a VIN decoder result that contains the year, make, and model.

Somewhere in the application, I need to use this. So let’s say I have a vehicle registration flow with a command to register a vehicle. From there, of course, I inject the interface we created, and dependency injection gives me the concrete implementation.

In the application code, I call the decoder, do the lookup, and then use the result. In my example, I’m just returning the year, make, and model to keep it simple and make the testing angle obvious.

And that’s where the justification usually comes in: tests.

That’s why we created the interface to begin with, right?

The interface tax

Once we get to the test, the next step is predictable. We create a mock. We tell it what the lookup should return. Maybe it returns a 2026 Ford F-150. Then we inject that mock into our application code, call the handler, and assert the result.

This is the classic example people use to justify interfaces for testing.

And sure, it looks clean. It feels testable.

But this is what I call the interface tax.

What’s really happening here?

You created an interface with one implementation and one usage. The only reason that interface exists is for testing. That’s it.

But the real need was never the interface. The real need was a seam.

That’s the part I think gets missed. A seam is what lets you vary behavior in a test. An interface is only one way to get that seam, and in a lot of cases, it’s not even the best way.

What the seam actually is

If you look at the VIN decoder and think about what it really does, the seam becomes obvious.

It uses HTTP.

That’s the seam.

So if I want to test the VIN decoder itself, I can focus on the actual dependency: HttpClient. More specifically, I can pass in a custom message handler that returns a specific JSON payload. That lets me test exactly what I care about. Did it make the request? Did it deserialize the response? Did it map the data correctly?

That is a real seam, and it lines up with the behavior I’m actually testing.

The important thing here is that I didn’t need an interface to create that seam. The seam was already there.

What about the application code?

Now when people hear this, the next objection is usually about application-level tests.

And I agree with the concern. If I’m testing application code, I should not have to know about HttpClient. I should not have to build a custom message handler in every test. That setup is too low-level for the thing I’m trying to test.

But that still doesn’t mean the seam has to be an interface.

It can just be a fixture.

In this case, I can create a VinDecoderFixture that builds a VinDecoder with a custom message handler based on whatever scenario I want. Maybe I have a CreateSuccessful helper where I pass in the year, make, and model I want returned. Maybe I also have a failure helper that lets me return specific status codes and error bodies.

Now in my application test, I use the fixture, configure the behavior I want, and pass the concrete VIN decoder into the registration handler.

Same end result.

The difference is I didn’t create an interface just to satisfy a testing pattern. I used the concrete thing and hid the setup behind a fixture.

And honestly, if you’re already using mocks heavily, there’s a good chance you’ve done this anyway. A lot of teams end up with utilities that configure mocks the same way over and over again. At that point, you’ve basically created fixtures already. You just did it with a mocking framework and an extra abstraction layer you may not have needed.

“But that feels like an integration test”

This is where some people get uncomfortable.

They’ll say, “Well, I’m still using the concrete implementation, and that feels like an integration test.”

My answer is: I don’t care.

In this example, the behavior is deterministic. There are no side effects. The dependency is controlled. I’m totally fine with the application code depending on the real VIN decoder in that kind of test.

We get way too hung up on labels sometimes. Unit test. Integration test. Whatever. The real question is whether the test is fast, reliable, and useful.

If it is, then I’m not worried about whether it crossed some imaginary purity line.

Sometimes you don’t even need a class

There’s one more part of this example that matters.

I often say that if you have a class with one method, you probably wanted a function.

That’s really what was going on here. The thing I needed was one operation: decode a VIN.

So instead of wrapping that in a class and then creating an interface for the class, there’s nothing stopping me from representing that dependency as a delegate. I can wire it up to the actual method in production, and in tests I can just pass in a delegate that returns whatever result I need.

That makes the test setup even simpler.

If all your application code needs is one operation, a delegate may be the better abstraction. Not because interfaces are bad, but because a delegate might more accurately reflect what the dependency actually is.

The real takeaway

The point here is not that interfaces are bad.

That’s not what I’m saying.

The point is that testing needs a seam, and that seam should line up with the thing you’re actually testing.

When I was testing the VIN decoder, HTTP was the seam.

When I was testing the application code, I used the real VIN decoder and hid the setup behind a fixture.

And when all I needed was a single operation, a delegate was another good option.

That’s the point.

Stop treating interfaces like they’re the only way to test and the default answer to every dependency. They’re one option. Sometimes they’re the right option. Sometimes they’re not.

Use a seam that matches the problem.

Testing Needs a Seam, Not an Interface

A lot of codebases are littered with interfaces that exist for no reason other than testing. One implementation. One usage. Extra indirection. More files. More ceremony. And people defend it because that’s just how they were taught to make code testable.

But the seam was the thing that mattered all along.

So the next time you reach for an interface, ask yourself why. Is it really modeling a meaningful abstraction in your system? Or are you just paying the interface tax because you assume that’s what testing requires?

Because it doesn’t.

Testing needs a seam, not an interface.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Is Event-Driven Architecture Overkill for Most Apps?
• Coding Isn’t the Hard Part
• Keynote: CTRL+SHIFT+(BUILD) PAUSE

The post Testing Needs a Seam, Not an Interface appeared first on CodeOpinion.

We’ve been told that clean code and deadlines are opposites.

That if you want to ship fast, you have to write garbage code full of hacks. But if you want to get it right, you need to add boilerplate.

That’s a lie.

Here’s the thing: a lot of the so-called best practices people tell you to follow are a scam. Adding layers, interfaces, and abstractions does not automatically help you ship faster. That is not architecture. A lot of the time, it’s just liability.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

The interface tax

Here’s a simple example I call the interface tax.

Say we need to process a payment using Stripe. Of course, because it’s a third party dependency, best practices say we need to abstract it. So we create an IPaymentService with a ChargeAsync method.

Then we create StripePaymentService, which implements that interface and uses the Stripe SDK directly.

Then in our actual usage, we have a ProcessPayment command handler that injects IPaymentService, and that is all wired up through dependency injection so we can call ChargeAsync.

Now back up for a second.

We have this interface. How many implementations do we have? One.

We have one implementation of StripePaymentService.

How many usages of IPaymentService do we have? One.

We have one actual usage, in this command handler. So what is the value exactly?

Do not confuse abstraction with optionality.

If the thought is, “Well, we might switch off Stripe one day,” when is one day? Is it tomorrow? Next week? Never? You do not know.

If your intent was to create a boundary or seam, you already had one. It was the handler.

That means if we ever did need to change something, the rewrite is simple. No interface. No extra implementation. Just use the Stripe SDK directly in the handler.

If we do end up changing payment providers, we change it in that singular place.

If we have to update the SDK and there is a breaking change, we update it in that singular place.

You do not need to add interfaces to feel architecturally responsible.

But what about testing?

Maybe you’re thinking: “But tests. I have other implementations because of tests.”

Testing is not a reason to immediately jump to interfaces like that is the only way you can possibly test.

Most third party libraries I use already give you some way to test. They provide fakes, alternate types, or static helpers. And even if they do not, the point is still just to create isolation.

You can do that in simpler ways, even with something as basic as a virtual method you override in a test.

The bigger issue here is not interfaces. It is that people lose sight of what they were actually trying to get when they applied Clean Architecture, Onion Architecture, Ports and Adapters, or Hexagonal Architecture.

What they were trying to get was isolation. They were trying to control blast radius by managing the direction of dependencies.

Unfortunately, that often turns into adding layers and indirection everywhere, and that just creates pain.

Five layers to do one simple thing

This is such a common complaint, and you’ve probably felt it too:

We have an API trying to pursue Clean Architecture, and I have to go through five layers of code before I get to the actual code to see what it’s doing.

Nine times out of ten, it is doing a simple database query or making a request to a third party API.

Take an example like fulfilling an order.

You inject an IOrderRepository and maybe some IMapper. Often people drift into some generic repository pattern too. You use the repository to get the entity out. Then you map that into some domain model. Sometimes that mapping is hidden in the repository, but it is still there. Then you call Fulfill.

And what is all of this actually doing?

It is just changing the status of a property on the underlying data model.

Then you use the repository to map it back and update it.

Repository. Mapper. Domain model. More mapping.

What are we trying to do here?

We are trying to get data out of the database so we can update it.

That’s it.

So if that is really all the implementation is doing, why not just inject the DbContext directly, find the record, update the status, and save the changes?

If there are no invariants, no rich business rules, no real complexity, what are we gaining from the repository?

What are we gaining from the domain model? Nothing.

The real issue is coupling

It is kind of wild to me that when people talk about Clean Architecture and similar approaches, they rarely talk about coupling directly.

That is really what you are trying to manage. But a lot of people act like coupling itself is bad. It is not.

Without coupling, you have nothing.

Coupling is not bad. Uncontrolled coupling is bad.

In the Stripe example, or in the database example, if you have a single usage of something, you do not have uncontrolled coupling.

If you have a thousand usages of something, now you have uncontrolled coupling. Maybe that is where an interface or some abstraction actually makes sense.

This is not about eliminating coupling.

It is not about decoupling everything.

It is about managing coupling.

Slices create isolation

This is where slices can help, because they create isolation.

In the Stripe example, that payment processing command handler can be a slice. It can define everything around that behavior. Maybe that includes an HTTP API, some application code, and maybe some domain code underneath.

That isolation inside the slice lets you decide how you want to manage coupling and what that slice is coupled to.

One slice might just be an HTTP API with some application code and a simple underlying data model.

Another slice might be more elaborate. Maybe it uses messaging, infrastructure, queues, a bus, application code, and a richer domain model because there is workflow and real business complexity.

Another slice might just be CRUD, and that is fine too.

Not everything needs the exact same layers.

Not everything needs the exact same structure.

This is really about being pragmatic and adding what you need where it actually gives you value.

To me, that is pragmatic architecture.

Control the blast radius

If something changes, it should be localized.

That should sound familiar.

Earlier I talked about the idea that “what if something changes?” and how people use that to justify an interface. But that is just one way to deal with change, and often not a very good one.

This is not about denying dependencies.

It is about controlling where they live.

One of the most senior moves you can make is to stop following patterns blindly and start understanding the value they are supposed to provide. Then ask whether you actually need that value.

Sometimes the right move is to ship code that is less than ideal.

Sometimes it is a bit of a dirty hack.

That is fine, as long as it is isolated. As long as it is contained.

Sometimes you need to do things for today so you can have a tomorrow.

Just do not do it in a way that handcuffs you tomorrow.

There are different ways to control blast radius and isolation. It does not always come down to turning everything into abstractions and layers.

Stop paying the tax

A lot of teams are paying an interface tax, a repository tax, and a layering tax without getting any real value in return.

They are solving problems they do not actually have.

The goal is not to look architecturally responsible.

The goal is to build software that is easy to change in the places where change is actually likely.

That means understanding where coupling matters, where isolation matters, and where simplicity is the better tradeoff.

That is architecture.

Not blindly adding layers. Not wrapping every dependency in an interface. Not turning simple code into ceremony.

Just thoughtful tradeoffs, made in the right place.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Is Event-Driven Architecture Overkill for Most Apps?
• Coding Isn’t the Hard Part
• Keynote: CTRL+SHIFT+(BUILD) PAUSE

The post Why “Clean Code” is Killing Your Velocity appeared first on CodeOpinion.

I get it. Most apps really are that simple. Typically, just CRUD. A user submits a form, you validate some data, save it in a database somewhere, and return a response. That is it. So is Event-Driven Architecture Overkill then?

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Event-Driven Architecture Overkill?

This topic stemmed froma Reddit post that I thought was pretty interesting. Primarily because it’s confusing a lot of concepts and seems to miss the point of Event-Driven architecture. So that’s what this post is trying to clarify.

Pretty much my blog/channel.

I bring up these topics because I think they have a lot of utility and value within a given context. The problem is when there is confusion about what these ideas actually are, and you do not really understand the problem they are trying to solve.

Here’s what confuses me: a friend’s startup processes maybe 10k orders monthly, yet they’re implementing event sourcing. Another friend at a larger company says their monolith handles 100x that traffic just fine.

So what’s the reality? Are we overengineering because it’s trendy? Or am I missing something fundamental about when events actually matter? Real thresholds where event-driven becomes necessary

Neither example has anything to do with the point.

Event sourcing has nothing to do with scaling. A monolith versus services has nothing to do with whether events make sense. So what is the reality? Are we overengineering because it is trendy, or are people missing something fundamental about what events actually are and where they matter?

The reality is, there is just a lot of confusion around event-driven architecture and the different ways you can use events. That is the confusing part.

Scaling is a benefit, not the reason

One of the first things to clear up is scaling.

Yes, scaling can be a benefit of using asynchronous messaging, but that is not the reason to use event-driven architecture.

Imagine a client interacting with an HTTP API. That API publishes messages to a topic on a message broker. Then a separate worker consumes those messages and interacts with the same database. This could even all be happening inside a monolith.

When people think about scaling an HTTP API, they usually think of adding more instances behind a load balancer so more requests can be processed concurrently. That same idea applies to workers consuming messages. You can scale out more workers and process more messages concurrently.

That is the competing consumers pattern.

So yes, there is a scaling benefit there. But again, that is not the primary reason you would apply event driven architecture.

Events as notifications between responsibilities

One of the main reasons to use event-driven architecture is that something happened in your system, and some other independent concern might need to react to it.

That means you have a publisher producing an event because something happened, but it has no idea whether there are any consumers, whether there is one consumer, two consumers, or ten. It does not know if any other part of the system even cares.

What you are doing is decoupling responsibilities in time.

A simple example is shipment delivery.

You order something online. You get tracking information. Maybe you use an app on your phone. Then all of a sudden you get a notification that your package has been delivered.

That notification is one responsibility.

The actual act of somebody dropping off the package, taking the picture, and persisting that to the system is a completely different responsibility.

So when the package gets delivered, that fact can be published to a message broker. From there, different parts of the system can react.

One consumer might send a webhook to some third party system. Another might send an SMS through Twilio. Another might send a push notification through Firebase. These are all different responsibilities, and they are not directly part of the delivery action itself.

That is the point.

Those downstream actions were not part of the decision making process of whether the delivery should happen. The delivery already happened. The decision was already made. Now other parts of the system are reacting to that fact.

That is a good way to think about events. They are facts. Something occurred, and now other responsibilities can respond to it independently.

That is one of the big benefits of event driven architecture in the form of notifications.

Commands and events are not the same thing

This is another place where people get confused.

Just because you are using asynchronous messaging, a message broker, or queues, that does not automatically make your system event driven.

Commands and events are different things.

A command is trying to invoke behavior. It is something like DeliverShipment. It is an instruction. Typically, there is one consumer responsible for handling that command.

There may be multiple parts of the system that can send that command, but the command itself is about telling something to do something.

An event is different.

An event is a statement of fact. Something happened. ShipmentDelivered. PackageDelivered. OrderPlaced.

Typically, an event has one publisher, and there can be zero, one, or many consumers. The publisher does not know who those consumers are, and it does not need to.

That distinction matters.

When someone signs for a package on a device and that triggers a request to your HTTP API, that request is handling a command. It is doing the work of recording the delivery. It should not also be directly responsible for sending every webhook, every SMS, and every push notification as part of the same workflow.

Those are separate concerns. That is where events fit.

Event sourcing is about state, not notifications

There is even more confusion because events can serve different purposes.

Earlier I was talking about events as notifications. But event sourcing is something else entirely.

Event sourcing is about state.

Instead of recording current state, you record the series of events that state can be derived from.

That is a completely different utility than using events for notifications.

Take a shipment as an example. You might have events like:

• Dispatched
• ArrivedAtShipper
• Loaded
• Departed
• ArrivedAtConsignee
• Delivered

Each shipment has its own stream, its own series of events. Another shipment has a different stream.

Those events become your source of truth.

From that stream, you can derive state. You can project it into a relational table. You can put it into a document store. You can shape it however you want. At any point in time, you can rebuild the current state or derive what the state looked like earlier.

That is what event sourcing is.

So yes, you can be using events for collaboration and notifications between parts of your system. And yes, you can be using events as the source of truth with event sourcing. Those things can overlap.

But they are not the same thing.

Do not use events just because everyone else is

Do not use events because you want to use a broker. Do not use them because everybody is using Kafka. Do not use them because you heard it scales better.

That is not a good reason.

Use events for notifications because a business fact occurred and other consumers want to respond to that fact on their own timeline.

Use events when you want to remove temporal coupling.

Use them when you have integration boundaries between parts of your system, whether that is inside a monolith, between services, or with external systems.

Events can act as a contract at those boundaries.

And do not use event sourcing just because you want to be event driven.

That is where people get themselves into trouble.

CRUD sourcing is not event sourcing

One of the worst examples of this confusion is when people use event sourcing even though all they are really doing is recording current state changes.

I often call this CRUD sourcing.

The event stream becomes something like:

• ShipmentCreated
• ShipmentUpdated
• ShipmentUpdated
• ShipmentUpdated

That is not event sourcing in any meaningful sense.

On the flip side, event sourcing makes sense when history matters.

When the sequence of events matters.
When understanding how something got to its current state matters.
When those events are real domain language.

In the shipment example, Dispatched, ArrivedAtShipper, Departed, ArrivedAtConsignee, Delivered. That is domain language. That history has meaning. That history has value.

That is the exact opposite of CRUD sourcing.

If you care about that history, and you want to use it as your source of truth and derive different views or models from it, then event sourcing can be a really good fit.

So, is event-driven architecture overkill?

It can be.

If your app really is just request-response, and you do not have different workflows, independent responsibilities, or integration boundaries, then yes, it could absolutely be overkill.

Do not invent problems for solutions you do not fully understand.

Scaling is a benefit in some cases, but it is not the reason to use events as notifications, and it is not the reason to use event sourcing.

Use events for notification purposes when you want to decouple responsibilities in time.

Use event sourcing when history matters, when the domain language matters, and when recording that sequence of facts as your source of truth gives you real value.

Do not make it more complicated than that.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Coding Isn’t the Hard Part
• Vertical Slices doesn’t mean “Share Nothing”
• Keynote: CTRL+SHIFT+(BUILD) PAUSE

The post Is Event-Driven Architecture Overkill for Most Apps? appeared first on CodeOpinion.

I keep seeing posts pushing back on the idea that coding isn’t the hard part. And I get why. A lot of the disagreement comes down to what people mean by coding.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

But in the world I work in, coding usually is not the hard part.

I’m talking about line of business and enterprise apps. Order management, healthcare, insurance, logistics, and similar systems. In those kinds of systems, the real difficulty is usually not writing code. That is not to say building software is easy, because it is not. But if you understand your tools, your language, your libraries, your frameworks, and you have a solid foundation, the coding itself is often the more mechanical part.

That is because these systems usually are not algorithmically complex. Some domains absolutely are, but most of the systems I’m talking about are more workflow complex than algorithmically complex. Once you have a good foundation and you know the tooling you are working with, building the system often becomes a matter of assembling pieces. You keep adding on to what is already there. In that sense, the implementation can start to feel routine.

The hard part is figuring out what to build

What is not routine is figuring out what needs to be built in the first place.

That is where the real complexity shows up. What events occur in the system? What triggers them? What business rules apply? What data has to remain consistent? What edge cases exist? How does the process actually work from end to end?

Those are the questions that make this hard.

For line of business systems, the best developers I know understand business well. They know how to break down workflows, decompose a problem, and understand how things move through a system. You are not just writing code. You are trying to understand how a business operates and then model that in software.

Why boundaries are difficult

That is why I keep saying that defining boundaries is one of the most important things you can do, and also one of the most difficult.

There are techniques that can help, like event storming, but it is still hard to take a large system, break it into smaller parts, and decide where responsibilities belong. That is where most of the real design work is.

Part of the disagreement around this topic is probably just different definitions of coding. One reply I saw said that applying a good design to an existing system is coding. And if that is your definition, then we are not that far apart. Because I am talking about system design and implementation together. That work is difficult. Building line of business systems absolutely has complexity.

Workflow complexity is different from technical complexity

I use messaging and event driven architecture examples a lot because they make this easy to see.

If you are building a workflow based system, you may have to deal with idempotency, retries, backoffs, dead letter queues, concurrency, claim check patterns, and all kinds of technical concerns. Those things are difficult. In many cases they are more difficult than implementing a specific business step in a workflow.

But even there, the hardest part usually is not the code for a single step. It is understanding the workflow, modeling it correctly, and knowing how it can evolve when the business process changes or when your original assumptions turn out to be incomplete.

A simple shipment example

Take a shipment workflow as a simple example.

At a high level, you might say a shipment is dispatched, a driver arrives for pickup, the shipment is loaded, the driver departs, arrives at the destination, and completes delivery.

That sounds straightforward when you say it like that. But real systems are rarely that simple.

Maybe one truck is handling multiple shipments.

Maybe a pickup becomes unavailable. Maybe the shipment is delayed and there is no point in sending the driver. This is called a dry-run.

Maybe parts of the process branch depending on the customer, the carrier, or the type of delivery.

So now what are you modeling? Is it one workflow or several? Where do those boundaries exist? What belongs together and what should be separate?

That is the hard part.

Writing the code for each step is usually the easier part once you actually understand the model.

Once the model is clear, implementation becomes mechanical

That is why methods like event storming are useful. They help you focus on the events, actions, side effects, users, and different perspectives before you jump into code.

You want to understand the workflow first. You want to understand how smaller workflows fit into larger ones and how they cross boundaries. That work can be done with business people long before you start writing implementation code.

Once you understand it well enough, the implementation often starts to feel templated.

You have probably felt this if you work in .NET and use a messaging framework. Good frameworks handle a lot of the technical heavy lifting for you. They deal with plumbing like the outbox, inbox, logging, database concerns, and idempotency so you can focus on the specific behavior you need to implement.

That is a good thing. But it also makes the point pretty clear.

At that stage, a lot of the work becomes filling in the blanks. The workflow has already been defined. The messages already exist. The handlers are just implementing the behavior you already modeled.

That does not mean there are no hard technical problems

There absolutely are hard technical challenges.

Scaling problems, data consistency issues, infrastructure concerns, deployment issues — those are all real. I like those problems. But those are often architectural issues around the shape and growth of the system, not questions about the business workflow itself.

They are different kinds of difficulty.

A lot of the pushback on “coding isn’t the hard part” comes from people saying that translating ideas into precise, working systems requires deep knowledge and experience. I agree with that completely. But I still separate understanding the business and designing the system from the actual implementation work.

Those are closely related skills, but they are not the same skill.

What the best developers actually do well

The best developers I know are not just technically capable. They understand the business domain. They can decompose a system. They know how to reason about workflows and boundaries.

Yes, they are also good with their tools and can handle concurrency, messaging, and technical complexity. But that is not what makes them stand out most.

What makes them stand out is that they can answer the hard design questions.

Where do responsibilities belong? Who owns the data? What capabilities belong in which part of the system? How are boundaries crossed? How coupled are different parts of the system? What kind of data coupling, timing coupling, or deployment coupling exists?

Those are technical questions, but they are also design questions. They sit above the level of just writing code.

So is coding the hard part?

So when I say coding isn’t the hard part, I am not saying building software is easy.

I am saying the hardest part of building business systems is usually understanding the business, modeling workflows, defining boundaries, and designing a system that can actually support the way the business works.

Once you have done that well, the coding often becomes the easier part.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Vertical Slices doesn’t mean “Share Nothing”
• Read Replicas Are NOT CQRS (Stop Confusing This)
• Keynote: CTRL+SHIFT+(BUILD) PAUSE

The post Coding Isn’t the Hard Part appeared first on CodeOpinion.

How do you share code between vertical slices? Vertical slices are supposed to be share nothing, right? Wrong. It is not about share nothing. It is about sharing the right things and avoiding sharing the wrong things. That is really the point.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Boundaries

If you have watched my videos before, you probably know I talk a lot about boundaries. A vertical slice is not that different from a logical boundary. What matters here is that a vertical slice defines a boundary around a use case.

That is the lens I want you to look through, because once you do that, the question of sharing becomes a lot clearer.

A Shipment Workflow Is a Good Example

A good example is a shipment. It is a workflow.

You have different actions that happen along the way that make up a life cycle from beginning to end. Think about ordering something online. It gets dispatched, which means the order is ready to be picked up at the warehouse. The carrier arrives at the shipper. Then the freight is loaded onto the truck. Then it departs. Then it arrives at the destination. Then it is delivered.

That is the workflow.

Now the natural question is this: is the vertical slice the whole workflow, or is each step its own slice?

Because if each step is a slice, how do you share between them?

A Vertical Slice Can Be One Step In the Workflow

Each step in that workflow can be a vertical slice.

You could model the entire workflow as one slice. Sometimes that might be fine. But often, each step can be its own slice because the workflow can change. It can deviate based on the actions that occur.

Take the same shipment example. The order gets dispatched, the vehicle is on the way to the warehouse, it arrives there, and then finds out the order was cancelled. There is nothing to pick up anymore.

That is a different use case.

In shipping, that might be called a dry run.

How do you implement that? It is just another vertical slice. It is part of the workflow, but it is also a deviation from that workflow.

That gets us back to the original question. What can you share between those vertical slices that are part of the same workflow?

There Are Two Different Kinds of Sharing

The first kind of sharing is technical infrastructure and plumbing.

Things like error and result types, logging, tracing, authorization helpers, messaging support, outbox primitives, event bus abstractions, and small utility code. That kind of stuff is normal to share. Some slices will use it. Some will not.

A slice gets to decide what dependencies it takes on and what tactical approach it uses. That is part of the slice owning its implementation.

But that leads to the more important question: what does the slice actually own?

A Vertical Slice Owns Its Data and Behavior

A vertical slice owns its data. That use case owns the data it needs and how that data is persisted.

It also owns the dependencies it takes on. It chooses the tactical patterns it wants to use for that specific use case.

That is important because people hear that and then assume everything must be completely isolated. But that is not really true, especially when several slices are part of the same workflow.

Shared State Is Not the Problem

In the shipment example, what you really have is a state machine. You have state transitions across the life cycle, from dispatched all the way to delivered.

So yes, there is shared state.

That does not mean there is shared ownership of everything.

Imagine a shipment with state like status, dispatched at, arrived at, pallets loaded, and emptied at. If that was mapped to a table, each piece of that state is owned by the slice responsible for that action. The dispatch slice changes the dispatch related state. The arrive slice changes the arrival related state. The loaded slice changes the loaded related state.

Each slice owns the behavior around its part of that workflow.

This Still Applies With Event Sourcing

You can think about the exact same idea with event sourcing.

Instead of changing columns in a table, you are appending events to a stream. Dispatched. Arrived. Loaded. Emptied.

Same concept.

Each use case owns the behavior that produces those events. It owns the data tied to that behavior. It owns where that data lives, whether that is in a table or in an event stream.

That can still all live together. You are still sharing an aggregate. You are still sharing a concern because there are invariants around that workflow.

That is not bad sharing.

Sharing an Aggregate Is Not Bad Sharing

An aggregate is a consistency boundary. You need that consistency boundary around the state.

A slice is a use case.

So if you have several use cases related to the same underlying model, that can be shared. If two slices share validation because both operate on the same domain model, that can be shared too.

At the same time, you can have other slices that are not part of that workflow at all and use a completely different model. That is fine too.

The point is not that every slice must have its own isolated universe. The point is understanding what actually belongs together.

Different Slices Can Use the Same Domain Model

Another way to visualize this is by looking at what each slice does from top to bottom.

One slice might be invoked by an HTTP API. It has application code and a data model under it. Another slice might be invoked by a message or event. It has different infrastructure, different application logic, but still uses the same underlying domain model as the HTTP slice.

The entry point is different. The infrastructure is different. The application code is different.

That does not mean the domain model cannot be shared.

And then you might have another use case that is not related at all, even if it lives in the same broader boundary. That one may have a completely separate model.

Again, the point is that slices define their own dependencies and their own behavior. But that does not mean they cannot share anything.

Where Sharing Becomes a Problem

The problem starts when you begin sharing things that have no business being shared.

In the shipment example, I am talking specifically about the workflow and the shipment life cycle from beginning to end. Nothing in that example has anything to do with compliance, customer support, customer tracking, or what was actually ordered.

Those are separate concerns.

The trap people fall into is that they start sharing and coupling things they should not. The model becomes unfocused. That is how you end up with one giant Shipment object for your whole system.

That is where you get into trouble.

Do not share one god object.

A Vertical Slice Is a Logical Boundary, Not a Physical One

This part is really important.

People often talk about vertical slices in the context of code organization, and that is useful. But a vertical slice is not a physical boundary. It is a logical boundary.

That means it does not have to live in one C#, Java, or TypeScript file. It does not have to live in one project.

If you have a mobile app deployed separately to iOS or Android, and it is dealing with specific actions as part of a use case, that can still be part of the slice. If the same use case is invoked through an HTTP API, that is also part of the slice.

The slice is the logical boundary around the use case. It is not just a folder structure.

Good Sharing Versus Bad Sharing

Good sharing is when vertical slices are operating on the same underlying thing as part of a workflow, a life cycle, or a set of common invariants.

Bad sharing is when a change unrelated to one vertical slice affects another slice unexpectedly.

That is when you are sharing things that have unrelated reasons to change.

That is the distinction.

Do Not Share Domain Meaning

Put another way, do not share domain meaning.

In the shipment workflow, dispatched, arrived, and loaded are use case specific. Dispatch is its own thing. No other feature should be doing something related to dispatch unless it actually owns that behavior.

If dispatch publishes events or changes dispatch related state, that should happen there. If there is state related to dispatching, that slice should own it.

You are not sharing that ownership.

Could you still share an underlying domain model that handles the workflow transitions? Absolutely.

But ownership of behavior still matters.

Focus on Actions, Not Just Data

Hopefully one thing stands out in this example. When I describe vertical slices and use cases, I am describing actions. I am not starting with data.

That matters.

And that is the real issue underneath all of this.

This Is Really About Coupling

When we talk about sharing, what are we really talking about?

Coupling.

That is what this usually comes down to.

If you understand what you are coupling to between vertical slices and use cases, you can manage it. If several slices depend on the same underlying domain model because they are part of the same workflow, that can be perfectly fine.

If every vertical slice can touch any piece of data and change state anywhere, then yes, you are going to have a problem.

At the end of the day, this is about managing coupling.

Share the Right Things

Vertical slices are not about sharing nothing.

They are about sharing the right things.

Technical concerns and plumbing? Sure.

Shared invariants as part of the same workflow? Sure.

A shared aggregate when several use cases are part of the same consistency boundary? Sure.

What you want to avoid is coupling things together that do not belong together.

That is the difference between good sharing and bad sharing.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Read Replicas Are NOT CQRS (Stop Confusing This)
• Your Idempotent Code Is Lying To You
• How Many Microservices Do I Need?

The post Vertical Slices doesn’t mean “Share Nothing” appeared first on CodeOpinion.

What’s overengineering? Is the outbox pattern, CQRS, and event sourcing overengineering? Some would say yes. The issue is: what’s your definition? Because if you have that wrong, then you’re making the wrong trade offs.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

“The outbox pattern is only used in finance applications where consistency is a must. Otherwise, it’s just overengineering.”

Not exactly.

“CQRS is overengineering and rarely used even at very high scale companies. One master DB for writes and a bunch of replica DBs for reads are sufficient.”

No. And it has nothing to do with scaling.

“Event sourcing, another overengineering term, but in reality, most production systems do not implement strict event sourcing as described in books and system design articles. In the practical world, only current state is stored in the primary DB and events and business metrics are stored in an analytics DB.”

The giveaway that this is wrong is the discussion of business metrics related to event sourcing.

In the “practical world”, I’ll give some examples where event sourcing is natural.

Let’s go through them one by one, explain what they are, and when you should be using them.

The Outbox Pattern

Is it about finance? It has nothing to do with finance. Is it about consistency? Yes, that part is correct. It’s really a solution to a dual write problem.

Here’s the dual write problem.

You have your application. Some action gets invoked. You persist a state change in your system.

That’s the first write. The second write is you need to publish an event and write a message to a message broker so other parts of your system know it occurred. That’s the second write.

Here’s the issue. It fails in between. So you do the state change. Everything passes. Everything is saved. Transaction is good. But then you fail to publish the message to your message broker. Now you’re inconsistent. Your state change happened, but the event never got published.

Is it a big deal if you fail to publish that event? It depends what you’re using the event for, and what downstream services care about. If it’s best effort metrics or analytics, it might not be a big deal.

If it’s part of a workflow, it can be a much bigger deal. You want that consistency, and that’s where the outbox comes in.

So how do you solve the dual write problem? Like most problems, don’t have it in the first place.

To solve the dual write problem, we’re going to have a single write. That means you persist your state to your database and within the same transaction you persist the message to an outbox table in that same database.

Separately, you have a publisher that queries the outbox table, pulls messages that need to be published, and pushes them to your message broker. If it succeeds, it reaches back to the database and marks the message as completed or deletes it from the outbox table.

If there’s a failure, you retry. You haven’t lost any messages you wanted to publish.

So is the outbox pattern overengineering? It totally depends on your use case.

If you’re using events as a statement of fact that something occurred within your system and other parts of your system need to know it happened, then it’s probably not overengineering.

If you’re using events as best effort analytics and it’s totally fine if some events aren’t published because nobody depends on them, and lost messages are fine, then yes, it’s overengineering.

One side note: if you’re using a messaging library, it probably already supports the outbox pattern.

CQRS

“CQRS is overengineering and rarely used even at very high scale companies. One master DB for writes and a bunch of replica DBs for reads are sufficient.”

This is confusing two things entirely. It’s talking about scaling at the read write database level, when in reality this is about your application design.

CQRS literally stands for Command Query Responsibility Segregation. Commands change state. Queries read state.

That has nothing to do with databases. One database, two databases, whatever the case may be.

This is about having two different code paths for different responsibilities.

But since scaling was brought up, especially on the query side, that’s the angle I want to tackle. In a lot of query heavy systems, you often have to do a lot of composition.

That composition could be to a single database, multiple databases, a cache, whatever. But you’re making multiple calls to different places to compose data together to return to a client.

Because a lot of systems experience this, people create views or materialized views so you’re not doing all of that composition at runtime.

Instead, you have a separate table, a view, a different collection, a different object, something that represents what’s optimized for a specific query.

Example: an order and line items.

Maybe instead of joining tables and calculating totals on every request, you have a view that does it.

Or you have a materialized view that’s persisted and updated every time there’s a state change to an order.

So when you make a state change, your command updates your write side. Maybe that’s a relational database with normalized tables. And because you have a materialized view, you update that too. That could be in the same transaction. Then when a query comes in, you read directly from the materialized view.

This is all about optimizing reads or writes.

In my example, it’s optimizing reads, using a materialized view.

It doesn’t need to be that at all. It could be a relational database, a document store, a single table, a collection, some object that already contains what you need.

The point is you have different code paths, so you have options.

You could still have your query side do composition and your command side use the exact same database, the exact same schema, and update what it needs to update.

You just have the option to do different solutions if you have different code paths.

So is CQRS overengineering? Not really. You’re likely already doing it in some capacity because you already have different paths for reads and writes.

Where this gets conflated is when you start thinking about it purely from a scaling perspective. If you’re doing a lot of composition and you add read replicas, that’s fine.

But here’s the question.

Are your read replicas eventually consistent?

Because that plays a part in the complexity you’re adding by just adding read replicas. If you want pre computation because you want materialized views to optimize the query side, that’s a strategy if you need to optimize.

Event Sourcing

“In the practical world, only current state is stored in the primary DB and events and business metrics are stored in an analytics DB.”

We’re talking about different things here. Events are facts. What event sourcing is doing is taking those facts and making them the point of truth.

Then you take that point of truth, that series of events, and you can derive current state or any shape of data from any point in time.

Let’s use a practical example because there are a lot of domains that naturally have events. You can just see them. A stream of things that occur.

Here’s a shipment.

You persist these as a stream of events for the unique thing you’re tracking.

Shipment 123 has its own series of events. Another shipment has a different series of events. Those event streams are the point of truth.

You can derive current state from them.

It has nothing to do with analytics, but you can use them for analytics because just like current state, you can turn them into any shape you want.

So if you have an event stream, you can transform it any way you want.

Maybe you transform it into a relational table so analysts can write SQL like “select all shipments dispatched on a particular day”. Or maybe you transform it into a document shape that’s optimized for an application query.

That’s the point.

Your source of truth becomes an append only log of business facts, events. Your state is derived from those events.

A lot of the issues I read about people having with event sourcing are twofold. First, they’re not actually doing event sourcing. They have an event log, but it isn’t the point of truth. Their real database is still current state, and the event log is just “extra”. Or they’re using events as a communication mechanism with other services like a broker, which is a different thing.

Second, there’s a huge difference between facts and CRUD. “Shipment created” is not an event. That’s CRUD. “Order dispatched” is an event. Something happened.

“Shipment modified” is not an event. “Shipment loaded”, “shipment arrived”, “shipment delivered”, those are events.

Is event sourcing overengineering? It can be if all you view your system as is CRUD, and that’s how you build systems.

But there are a lot of domains where, once you start seeing it, you naturally see a series of events and it becomes obvious that’s where event sourcing fits.

The Real Point About Overengineering

Everything has trade-offs.

If you do not understand a concept, you won’t be able to understand what those trade-offs are, because you don’t even know what they are.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Your Idempotent Code Is Lying To You
• You Can’t Future-Proof Software Architecture
• How Many Microservices Do I Need?

The post Read Replicas Are NOT CQRS (Stop Confusing This) appeared first on CodeOpinion.

You have some code that handles placing an order. This could be an HTTP API or a message handler. You made it idempotent. You added a unique constraint on some kind of message ID.

And somehow… you still end up double charging the customer’s credit card.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

Idempotent

You did everything right. You have idempotency. You have an inbox table and a unique constraint on that message ID. Your handler should be exactly once, right? Wrong.

And it’s because of this call to our payment gateway outside of our database transaction.

Our database can tell us whether we processed the message, but it doesn’t stop us from double charging our customer.

So let’s talk about why this happens, how concurrency can make it worse, and some solutions.

The happy path: idempotency with internal state

Let’s say you have an HTTP API where you might get multiple requests from the same user. Or this could be a message handler from a message broker where a message can be delivered more than once.

What that looks like is:

• A request comes in
• We take a message ID and persist it to our inbox table
• Because we have a unique constraint, if it wasn’t there, everything is fine

Then that exact same message (or HTTP request) comes back in. Guess what? It’s already there. So our request fails.

This is the happy path. It’s idempotency inside the database.

As long as you’re only updating internal state, this works. But it gets a lot more complicated once you start calling something external… like a payment gateway.

Why the payment gateway breaks “exactly once”

In the happy path, all we do is:

• start a transaction
• add our message ID to the inbox
• persist state
• if it was a duplicate, we throw when we save or commit

Simple.

But we also need to make the call to the payment gateway. And this is where the issues start.

It might seem like a good idea to call the payment gateway immediately. Maybe you get a transaction ID back, some receipt, something you can persist in your database to mark the order as paid.

But here’s the problem: That payment gateway call is outside of our transaction. Outside of that unique constraint.

So now this can happen:

1. First request comes in. We charge the customer. We save our inbox record. We persist our state. All good.
2. Second request comes in (same message, same request) We charge the customer again. Then we hit the database and fail because of the unique constraint when we save and commit.

So our database protected our internal state. But it didn’t protect the external side effect.

Concurrency makes this easier to see

If you’re looking at this from an HTTP API perspective, you can have two identical requests come in at the same time.

The load balancer sends them to two different instances.

They both hit the payment gateway at roughly the same time.

One of them wins the unique constraint. The other fails on commit.

But both potentially charged the customer. At the start I said concurrency can make it worse. A different way to think about it is: concurrency makes it easier to reproduce and prove you have the issue.

There’s no magic distributed transaction coming

One “solution” is a distributed transaction. The reality is you’re not going to get one.

That inbox table protects your internal state, but the moment you cross that network boundary to the payment gateway, all bets are off. You kind of went from exactly once to at least once.

But we can design for it. There are a few approaches here. It’s not one magic fix. It’s usually a combination depending on your situation.

1) The third-party service supports idempotency

Your third-party service might support idempotent requests.

A good example is Stripe. It supports an idempotency key that you pass in the header to make idempotent requests.

So you decide what the key is for that specific business operation. If the same request happens more than once, you send the same key again.

Now it becomes idempotent on the payment provider side too.

Side note: if you’re creating an HTTP API, support idempotent requests. Your clients will love you. If you don’t, they’re the ones who have to deal with the rest of this stuff.

2) Lookup by a reference ID before charging

If the provider doesn’t support idempotency keys, another option is a lookup against some kind of reference.

In my example, I have a payment gateway and I want to know: is this invoice paid?

My reference at this point is the order ID. So the flow becomes:

• Ask the gateway: “Is order 123 already paid?”
• If not, charge it

You can still have race conditions here. This can still be an issue. But it may be good enough, especially if the third party has a unique constraint on something like your order ID.

3) Serialize the operation per business key (locking)

If lookup isn’t enough, another solution is serializing the operation by a granular business key.

What I’m talking about here is locking.

You’re basically creating a distributed lock. That might be:

• using your database and row locking
• using Redis for locking
• something else entirely

In my example, “granular business key” might be per order. That means only one payment attempt for that order runs at a time. If I can’t acquire the lock, I retry, or return something that tells the caller to retry. Now at any point, we only execute one at a time for that order, charge the customer once, and release the lock.

Caveats

The trade-off is throughput. That’s why the business key has to be granular. If you lock too broadly, you slow everything down.

Also, timeouts matter. If the payment gateway times out, that does not mean it failed. It might have actually succeeded. So even with locking, you still need to think about what “timeout” really means.

4) Separate internal state from external calls (inbox/outbox)

Another option is inbox/outbox and splitting internal state from the external call.

What does that mean?

• We get the order
• We mark it as payment pending
• We add a message to our outbox saying “charge payment”
• We save all of that together in the same transaction

Then separately, a processor reads the outbox and performs the external call to the payment gateway.

If the provider supports idempotency keys, that outbox processor uses them. And then if it succeeds, we mark the order as paid. This doesn’t magically fix double charging. You could still double charge.

What it does give you is better internal consistency and better failure handling, because message handlers let you handle retries, backoff, timeouts, and errors differently than your main request path.

That’s one of the reasons I love messaging.

5) Reconcile and compensate

This generally always happens: you need reconciliation and compensating actions.

There’s nothing wrong with this. If something realizes “uh-oh, we double charged”, you void or refund.

This can be a workflow step, or a periodic reconciliation process that compares your system with the third-party system. You’re not failing as an engineer because you need compensation. This is just reality when you’re dealing with external systems.

Putting it all together

You have a lot of options, and it’s often a mix:

• Use an inbox table to dedupe incoming messages
• Use a unique message ID to prevent reprocessing internal state
• Use an outbox pattern so your intent to call an external system is persisted with your internal state
• If the provider supports idempotency keys, use them
• If not, can you lookup by a reference ID to see if the action already happened?
• If acceptable, can you serialize with a distributed lock by a granular business key like order ID?
• And when those don’t cover everything, reconcile and compensate

Ultimately, I don’t think the goal is “exactly once” as in “the operation only ever happens exactly once.”

A better goal is designing a system that’s effectively once — it behaves correctly even when you deal with race conditions, concurrency, and timeouts that are outside of your control.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• You Can’t Future-Proof Software Architecture
• Context Is the Bottleneck in Software Development
• Watch Out for Superficial Invariants

The post Your Idempotent Code Is Lying To You appeared first on CodeOpinion.

“Future proof your architecture” sounds good. But the reality is you can’t future-proof Software Architecture. When you really think about it, the future is just what’s breaking assumptions. You can’t really future-proof that.

What you can do is contain changes so they don’t ripple through your system.

Where people go wrong is trying to future-proof with abstractions everywhere. What you really want to be doing is controlling the blast radius, meaning controlling where change goes.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

I’m going to explain this using a thread by Aaron and elaborate on some of the things he’s pointing out.

He posted:

I posted a lot of bangers about SDK bin’s terrible software choices and how it generally made life unpleasant for us. So I wanted to detail how we’re addressing the dumbest and worst design choices in the codebase.

So the first one: what did our dev do when we needed to renew an annual subscription? Modify the subscription created date and reset the renewal reminder hard coded as N months from the creation date.

Now you might be thinking, “That’s ridiculous. I would never do that.”

But it underlines why things change.

The Unknown Is Usually Boring Stuff That Stops Being Boring

In the context of future proofing, the unknown usually lies in things like:

• pricing rules changing
• renewals and billing schedules changing
• tax changes
• refunds and how those show up
• partial payments
• a new payment provider, or a second payment provider/gateway
• workflow that used to be simple, becoming not so simple

That’s the real problem. Early on, when you’re building a system, you can have a lot of assumptions about the unknown.

What kills you is early decisions that leak everywhere in your codebase and cause coupling.

You have assumptions. You make decisions. Those decisions leak everywhere. Now you’re coupled. And that coupling is going to cause a lot of pain later when you try to make change.

You know you have this problem because when a change comes in, you have to touch all these things:

• the UI
• the “domain”
• persistence
• some random shared helpers and utils
• reporting
• background jobs
• and my favorite: three or five or a dozen other places you didn’t even know existed

You’ll often hear, “Well, we have a very large system and it’s very complex.”

In the context of what I’m talking about, that’s not complexity. That’s coupling.

Stripe Isn’t the Problem. Leaking Stripe Everywhere Is.

In Aaron’s case, he’s feeling the pain of everything being coupled so tightly to Stripe that it’s taken a mini Manhattan project to move off it.

Related, sure, but fundamentally separate concerns. So the assumptions and unknowns causing pain here are exactly what I said at the beginning:

• the idea of a renewal wasn’t a first class concept
• moving off Stripe becomes a disaster
• conflating invoices and payments creates more pain

That “Created Date Renewal Hack” Is a Symptom

Here’s the type of thing that happens when the Stripe assumption leaks into your system.

You end up with leaked information in your subscription, and who knows where else, like:

• Stripe customer ID
• Stripe invoice ID

And then the only real concept you had was a created date time.

But because you needed renewals after the fact, you didn’t model it.

So that created date turns into a hack. You “renew” by pretending it started again. You overwrite the created date and reset the reminder hard coded off that date.

This is also one of those situations where if the business actually knew you were overwriting this data, you’re potentially losing a lot of valuable info. Audit info. What actually happened. When did it renew. What was the history.

If you talk to someone non technical in the business, whether they care about that, they’ll probably say yes.

Your Data Model Isn’t Your Domain Model

This is where I’ll say something you’ve heard me say before:

Your data model isn’t your domain model.

How you persist data, what your schema looks like, that’s not your domain model.

If you look at your model and think, “This doesn’t really express the domain,” then yeah, you probably don’t have a domain model. You have a data model. A bucket of data. Getters and setters.

Bonus tip: it’s also not your resource model.

You have an HTTP API. Those are different things.

What you return to clients isn’t your underlying schema and isn’t your domain model. It’s a representation of what you want to show to clients.

So What’s the Fix? Not Interfaces Everywhere.

So what’s the fix?

In the Stripe example, it was coupled everywhere.

Is the fix immediately to jump to interfaces and put them everywhere? Use the adapter pattern everywhere all the time?

No.

It’s what I said at the beginning: controlling the blast radius. When you make a change, it should be localized to one particular place.

With Stripe specifically, it depends how coupled you are:

• Is the Stripe customer ID leaked into your database?
• Are other clients using it because you exposed it via your HTTP API?
• Do other libraries use it?
• Are you using the Stripe SDK in 10 places, or 200?

If you have 200 usages, and it’s all through the UI, the domain layer, persistence, reporting, background jobs, and everywhere else… you don’t have a blast radius.

You have a disaster.

Separate Concepts: Invoice vs Payment

Another simple example is the invoice vs payment problem.

They were treated as one concept and it was a disaster.

They don’t need to be. They should be separate things. You can have an invoice that has nothing to do with Stripe. It’s just an invoice. No Stripe internals. It’s your concept. Then payments are a separate concept.

Now you can apply payments to an invoice. Partial payments? Fine. Refunds? That has nothing to do with invoices. That has everything to do with payments.

Separate concern. Easier to support. Easier to change.

Watch Your Nouns: Third Party Vocabulary Leaks

Pro tip: when you’re using third party services heavily, especially if they matter a lot to you, the nomenclature from that third party starts leaking into the core of your system.

You’ve got to be careful there.

You want your product’s nouns and verbiage to be yours, not the third party’s.

A Concrete Blast Radius Example

Here’s what “controlling the blast radius” can look like when paying an invoice.

You fetch the invoice from the database.

You create a payment. Separate concept. You call Stripe to charge the account. If it succeeds, you mark the payment as succeeded. If it fails, you mark it as failed.

Then you save your database changes.

Yes, you’re going to have reconciliation, because if something fails on your side but the charge actually went through, you deal with that after the fact.

But the point is this: You’re controlling the blast radius of where you deal with Stripe. It’s concrete. It’s real. But it’s contained.

If you need to change payment providers, you change it where you have that capability exposed. It’s not coupled everywhere.

How People Make It Worse

This is where people take the right problem and make it worse.

Before you say, “Let’s make an IPaymentProvider used across the whole system” — congratulations, you just created shared coupling.

Or “Let’s build a generic billing framework” — no, you created a framework specific to one implementation that isn’t generic at all.

“Let’s reuse this shared library across services or slices” — no, what you created is a distributed monolith starter kit.

Designing for the Unknown

So how do you design and architect for the unknown?

It’s not about trying to future-proof Software Architecture. It’s about containing the blast radius.

For the things that often change — rules, workflows, integrations — if you segregate them, you can change them without it affecting your entire system.

Where things go wrong, like the Stripe example, is leaking internal information throughout the system. Then if it changes… now what?

Because you didn’t localize it. It permeated everywhere, and the blast radius is huge.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Context Is the Bottleneck in Software Development
• Why “Microservices” Debates Miss the Point
• Watch Out for Superficial Invariants

The post You Can’t Future-Proof Software Architecture appeared first on CodeOpinion.

Software development context is the real bottleneck, not writing code. AI can generate code fast, but without context, boundaries, and language, you get coupling and brittle systems.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

With AI, I think people are taking a leap that is fundamentally wrong. It is not about producing cheap code. I do not think that has ever been the bottleneck. The bottleneck has been context. If you have watched enough of my videos, you probably know my slogan is context is king. And context is probably more important now than ever.

It is not about what syntax or folder structure your source code looks like. It is the context of why it does what it does. Why did you write the code, or the AI write the code, given your instructions? What are we optimizing for? What constraints did we have? What are the invariants and the things that can never happen? Probably most importantly, what tradeoffs did we intentionally accept? What decisions were made, and why?

AI can provide the implementation. It can write all the code. But it needs context. It needs to understand how to make the tradeoffs. Giving the instructions I see online like “write clean code” or “DRY code” is the most useless instruction for actually developing a good design.

“Design does not matter anymore” is the trap

I understand the rebuttal people have. “Well, it does not even matter anymore about design. Because if AI can read the code and can write the code, and therefore change the code, it does not matter. It does not matter what type of folder structure you have, organization, it does not matter at all about the design because AI can just handle it.”

But that is a trap.

The pain has never been writing code. It is about making behavioral changes safely.

Coupling is still the thing you have to manage

I stumbled upon a post that basically said the way the code looks should be irrelevant. What matters is the end result. On the surface, I get it. But I think it is naive.

When people talk about “how the code looks” they might be thinking structure, syntax, whatever. But to go beyond that, yes, it matters how it looks, because coupling still needs to be managed. Coupling is arguably the thing that when we are writing software we need to handle the most. If you want a long lived system that can evolve and change, you need to manage coupling.

If AI makes producing code cheap, guess what is also going to be cheap. Creating coupling. Creating a rat’s nest turd pile of coupling. What people call a big ball of mud. Something that is hard to change.

Everybody can relate to this. You make a change to one part of your system and it affects another part of the system unintentionally. Why does that happen? Coupling.

And you might be thinking, “But AI knows everything.” It does not.

“But it is going to know all the coupling.” Sure. But you already know all the coupling right now as well. And you still have this problem. So what is going to be different with AI?

If you are using a statically typed language, say in a monolith, you can find usages. You can run tools to know what your coupling is between different boundaries, or how your system runs. You can already know this, and you still end up with a turd pile that is brittle and hard to change.

So I do not think the answer in the era of AI is to ignore design. It is likely the opposite. It is providing a design that is built on constraints and context.

And that gets us to the real question. Where does that context live?

Where context actually lives

Context lives in the structure. It lives in the dependencies. It lives in the boundaries. And boundaries are still more important than ever because of coupling. All these foundational things you are doing now, even with AI in the mix, are still relevant.

So how do you capture the design and context in your system? First, you have to be explicit in the domain and the language you are using. I preach this so much. It is the opposite of CRUD, and the language now is more important than ever.

Use the context to understand what the domain is. Let me give you an example in logistics and shipping. What are the use cases? What are the things you do as part of the workflow?

You can dispatch an order. When a vehicle arrives at the shipper to pick up the freight, you arrive. When you pick up the freight, that is the load. When you leave and you are on route to delivery, that is depart. When you unload the freight or deliver it, that is empty.

These are explicit. It is the exact opposite of CRUD. CRUD provides no context. All the context is living in your end user’s head, because that is the workflow of how they interact with your system. If you have create shipment, update shipment, update stop… what is this? What does the system even do?

You would not be able to tell me what the workflow is, because the workflow is in somebody’s head. You are just recording current state. And if we are talking about current state and how you are recording state, it tells you how the system is now. But it gives you no context about how it got that way.

Events give you the story, not just the state

This is where events fit so naturally. There is a big difference between “shipment created” or “shipment updated.” What does that even mean? Versus being explicit about actions and commands.

An order was dispatched. It arrived. Loaded. Departed. Delivered.

These are explicit behaviors of your system. That language is the story of the domain. One of the most underrated places context lives is in language in the code. It tells you what the system does, and how it does it.

Not all language is created equal. You can be using terms, especially in a large system, that mean different things depending on the boundary. That is why boundaries matter. Boundaries preserve context and control coupling between them. The language inside that boundary encodes the intent of what it does. Events preserve intent by capturing what happened, and why.

Boundaries keep your concepts from being smashed together

When I talk about boundaries, I mean the different parts of the system that have their own context. In the logistics example, you might have sales, rating, orders, ordering. Visibility to customers about the status of their order. Execution, dispatch, tracking the vehicle and the path through delivery. Auditing. Communications with the customer. Billing and approvals, making sure documents are in place so you can invoice, pay carriers, pay drivers, whoever is executing the shipment.

Each one has its own context. And here is a really good example of what happens when you do not respect that. A system coupled everything so tightly to Stripe that it took a miniature Manhattan project to move off of it.

Using Stripe as a credit card processor is fine. But conflating Stripe payments with invoices is a mistake because they are not the same thing. Payments are money sent. Invoices are amounts owed for services and taxes and everything else. They are fundamentally separate concerns. Different concepts.

That is design. That is context.

AI can write code. It guesses your intent.

Design has always been important. But I am hoping now people see the value in capturing context within your design.

Create boundaries with specific context. Use the domain language explicitly. Make the concepts, the actions, the events, and the reasons why things happen, obvious in your code. Because CRUD only gets you so far. With CRUD, workflow and concepts are in the end user’s head. They are not in your system.

And if you are using AI to generate code, it does not have that workflow context unless you put it there. And of course, manage coupling between boundaries. Because AI producing code faster just means you can build a big ball of mud quicker if you are not managing coupling.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Why “Microservices” Debates Miss the Point
• Aggregates in DDD: Model Rules, Not Relationships
• Watch Out for Superficial Invariants

The post Context Is the Bottleneck in Software Development appeared first on CodeOpinion.