Category: software

The “Passive-Aggressive” Event

Many developers believe that events inherently offer advantages over other message types, such as commands, making them “better” by default. In other words, they think that messaging between services should heavily rely on events while minimising the use of commands.

In this article, I aim to dispel that myth. As is often the case, the correct answer is: “It depends.” Neither events nor commands are inherently superior; each is best suited to specific scenarios.

  1. Commands are from Mars, Events are from Venus
  2. Coming from the left or the right side
  3. How does a “good” event look?
  4. What is the “passive-aggressive” event?
    1. Modelling with an event
    2. Using a command instead
    3. Why does it matter?
  5. Events enable decoupling, commands don’t?

Commands are from Mars, Events are from Venus

Clemens Vasters, Principal Architect of Azure Messaging Systems (Azure Service Bus, Azure Event Hubs, etc.), proposed a classification of distributed messaging types that I find particularly insightful. He divides them into two categories:

  1. Intents (left column): Reflect a transfer of control from one service to another, such as commands and queries.
  2. Facts (right column): Represent something that has happened in the past and, as such, cannot be retracted. Domain events (such as in Event Sourcing) fall into this category.

Coming from the left or the right side

Most developers start their distributed messaging journey, favouring one side over the other: either systems dominated by “intents” (especially commands) or fundamentally “event-driven” choreographies and orchestrations.

Until about five years ago, your first experience with distributed messaging was likely using brokers like SQS or RabbitMQ to queue commands and load-balance work across worker pools. Applications at the time were less data-intensive and more focused on behaviour and interaction.

Then came the Big Data movement, “Designing Data-Intensive Applications” (DDIA), and Kafka. Suddenly, we modelled systems as data flows represented by events. Data was king, and systems became a series of computations built on top of data movements.

Regardless of how your journey began, it likely left you with a “blind spot” toward the other side. If you started with events, everything looked like an event. If you started with commands, everything looked like a command.

I know this firsthand; I spent ten years primarily building event-driven systems, where events were foundational and Kafka was the default transport layer. Over time, I noticed the emergence of a problematic pattern I call the “passive-aggressive” event.

How does a “good” event look?

Before we understand the “passive-aggressive” event anti-pattern, we need to define what a good event is. Jonas Bonér offers a helpful perspective in his legendary Voxxed Days talk on how events are reshaping modern systems.

In short:

  • Events represent facts from the past and, therefore, cannot be changed. However, they can be corrected and accumulated.
  • Events are broadcast without any expectations from the producer about what consumers will do. In fact, the producer neither knows nor cares who the consumers are (Bonér describes this as “anonymous”).

This model supports greater system autonomy and decoupling. However, “can” is the operative word—abusing events can lead to an “insidious” form of coupling.

What is the “passive-aggressive” event?

In simple terms, a “passive-aggressive” event is a command improperly modelled as an event. The giveaway is when the producer cares about the consumer’s reaction or expects feedback.

In proper event modelling, the producer should have zero expectations about how or whether the event is handled. In a passive-aggressive event, a hidden “backchannel” exists where the producer expects actions or responses from the consumer.

Let’s explore this with an example.

Imagine an application that manages payments. When money is received, it is flagged as “Received.” Upon reception, we want to notify the sender by email. Once the email is sent, the payment is moved to “Confirmed.”

There are two services:

  • A Payment Service that manages the payment lifecycle.
  • An Email Service responsible solely for sending emails.

Modelling with an event

Let’s model this interaction with an event first.

Flow:

  1. A user sends a payment.
  2. The Payment Service receives and stores it as “Received.”
  3. It broadcasts an event: “PaymentReceived.”
  4. The Email Service consumes the event, sends an email, and then emits a “PaymentConfirmed” event or calls the Payment Service directly to update the payment state.

Using a command instead

Let’s model it with a command now.

Flow:

  1. A user sends a payment.
  2. The Payment Service receives and stores it as “Received.”
  3. The Payment Service sends a command to the Email Service: “SendConfirmationEmail.”
  4. The Email Service processes the request and responds with a success or failure message.
  5. The Payment Service updates the payment state to “Confirmed” based on the response.

Why does it matter?

The second model maintains better encapsulation (a principle emphasised by David Parnas since the 1970s). In the event model, business logic leaks into the Email Service, breaking the Payment Service’s encapsulation and increasing the cost of change.

Furthermore, the Email Service becomes less reusable because it must understand business logic unrelated to email delivery.

If this approach is repeated across integrations—e.g., linking Email Service with User Account Service, Bank Account Service, and so on—it becomes unsustainable, leading to a “Distributed Monolith”: the worst aspects of monoliths and microservices combined.

Clear boundaries and separation of concerns (echoing SOLID’s Single Responsibility Principle and DDD’s Bounded Contexts) improve organisational flow and reduce cognitive load. Clean, explicit contracts facilitate easier understanding and modification.

Events enable decoupling, commands don’t?

Many like to repeat this mantra, but it is misleading.

Encapsulation is what enables decoupling. Encapsulation allows systems to integrate via low-touch interfaces, exposing only what is necessary.

In our example, using a command allows the Payment Service to know only the minimal information needed to send an email. The Email Service is concerned solely with sending emails, not with the reason behind the email being sent.

Does this mean we should always prefer commands to events? Absolutely not. When it’s safe to simply publish an event and let downstream consumers react (or not), events are preferable. However, we must be cautious not to inadvertently increase coupling through hidden dependencies and back channels.

Data modelling recommendations for Apache Avro

I am a big fan of Apache Avro, said nobody ever…

Actually, in the context of Kafka/Streaming/Messaging, I like Apache Avro quite a bit, especially when paired with a Schema Registry that “protects” your topics/streams from “pollution”. Other alternatives are either too complex (JSON Schema) or designed for ephemeral data (Protobuf).

That said, Avro tends to confuse people with a data modelling background in SQL/Relational databases. In this post, I want to share some general modelling recommendations that will make your schemas more robust, semantically meaningful and easier to maintain.

Avoid nullable fields

This is a classic recommendation that I extend to EVERYTHING:

  • Table fields
  • Objects
  • API schemas

After all, null is the root of all evil, right?

Reducing the number of nullable fields in your schemas is also not complicated. Let’s take the following example:

record MyRecord {
  // NO!!
  union { null, string } country;

  // BETTER!
  string country;

  // PERFECT!
  string country = "United Kingdom";
}


In many cases, fields can be made “not null” when using a reasonable default (e.g. country can default to UK, currency can default to GBP).

While Java represents these fields as nullable by default, when using the Builder pattern implementation in code-generated POJOs, calling build() fails if a non-null value doesn’t have a value (and there is no default), guaranteeing that incorrect data isn’t instantiated (and published).

Assign defaults whenever possible

This recommendation complements the previous recommendation regarding nulls.

While not all fields will have natural candidates for a default (e.g., name), you should strive to define one whenever possible.

This is particularly important if new fields are added to an existing schema. The only way to maintain backward compatibility (i.e., reading old data with the new schema) is to give the new field a default. Otherwise, the new schema won’t be able to read old data because it will be missing the new field.

Use Logical Types

nulls are poor data modelling; they don’t define a type. Instead, they represent an absence of type altogether.

Programs are safer when we can constrain what is representable via restrictive types. If we apply this logic, embracing Avro’s logical types is the next sensible step.

record MyRecord {
  // NO!!
  float amount;

  // YES!
  decimal(18,8) amount;
}

Logical Types make schemas more semantically meaningful and reduce the possibility of error in downstream consumers. Using Logical Types, we guarantee that producers and consumers can restrict the data range to the most specific cases.

This recommendation includes using UTC-referencing Logical Types like date or timestamp-millis, which captures more semantical information than using a simple number type.

Be aware of union quirks

Avro unions are an implementation of tagged/discriminated unions: they represent a data structure that can “hold a value that could take on several different, but fixed, types”. They are used to represent nullability for Avro fields:

union { null, actualType } yourField);

They can also represent a field that might contain different data types.

record MyRecord {
  union {
    PaymentRequested,
    PaymentAccepted,
    PaymentReceived,
    PaymentReconciled
} event;

Unions are excellent as part of rich schemas that capture all data types a field might contain, aiming to constrain the space of representable data/state.

However, unions have two counterpoints that should be considered:

  • Evolving the list of allowed types follows particular rules to maintain backward compatibility (see below).
  • Java doesn’t support them naturally. Instead, union fields are represented as Objects in the Java POJOs and require the use of instanceof to route logic based on the actual data type.

While using them in Java is not very ergonomic, unions are still the correct data type to represent multi-datatype fields and Java limitations shouldn’t stop us from leveraging them when appropriate.

Be careful when evolving Enums and Unions

Enums are poor man’s unions: they represent multiple data types stripped of all their content except a “name” (the enum values). In other words, they are just tags, while Unions can define specific fields for each type.

This approach is common in languages where creating types (i.e., classes) is costly. Developers will optimise for speed and use enums to create a class hierarchy quickly, while sacrifiding type safety in the process

In both cases, they require some care when evolving the list of allowed values to maintain backward compatibility:

  • Reader schemas (and consumers) must be changed first to incorporate new allowed values. If we didn’t and we changed producers (and their schemas) first, we risk producing values that downstream consumers won’t understand and break them.
  • Only add new values at the end of the list, never in the middle or at the beginning. While this sounds like weird advice, some languages treat enums differently from Java. For example, C# is “order aware” based on how enum values are defined in the corresponding C# file, and a number is assigned to them that is used during (de)serialization. Changing the order of values will break this order and make consumers fail. The solution is adding new elements at the end.
  • Never remove allowed elements from the list for the exact same reason explained above, but also because doing so would prevent consumers using the new schema from reading old data (that was using the removed elements).

Software engineering trends that are reverting (I)

When I entered the software industry a long time ago, people who had been part of it warned me that software trends came and went and eventually returned. “This thing that you call ‘new’, I have seen it before”. I refused to believe it. Like a wannabe Barney Stinson, I thought ‘new’ was always the way.

I have been around long enough to see this phenomenon with my own eyes. In this series of posts, I want to call out a few examples of “trends” (i.e., new things) that a) aren’t new anymore and b) people are walking away from. The series starts with one of the most “controversial” trends in the last 10-15 years: microservices!

Microservices are so 2010s

Microservices are dead! I’m joking. They are not dead, but they are not the default option anymore. We are back to… monoliths.

While there have always been people who thought microservices weren’t a good idea, the inflexion point was the (in)famous blog post from Amazon Prime video about replacing their serverless architecture with a good, old monolith (FaaS is just an “extreme” version of microservices).

Why was this more significant than the thousands of posts claiming microservices were unnecessary complexity, talking about distributed monoliths and criticising an architectural approach that came from FAANG and only suited FAANG? Well, because… it came from FAANG. The haters could claim that even a FAANG company had realised microservices weren’t a good idea (“We won!”).

Realistically, this would have been anecdotal if it weren’t for something more important than a bunch of guys finding a way to save money when they serve millions of daily viewers (do YOU have THAT problem?).

It’s the economy, stupid

US FED interest rate since 1990

The image above shows the US FED official interest rates. Historically, interest rates have been pretty high (about 5%, according to a recent interview with Nassim Taleb on Bloomberg). From 2008 to post-COVID 2022, we experienced an anomaly: close to 0% rates for almost 15 years. Investors desperate to find good returns for their money poured billions on tech companies, hoping to land the next Google or Facebook/Meta.

Source: https://goingdigital.oecd.org/en/indicator/35

Lots of startups with huge rounding funds started to cosplay as future members of the FAANG club: copy their HR policies, copy their lovely offices, and, of course, copy their architectural solutions because, you know, we are going to be so great that we need to be ready, or we might die of success.

We all built Cloud Native systems with Share-Nothing architectures that followed every principle in the Reactive Manifesto and were prepared to scale… to the moon! 🚀 Microservices were the standard choice unless you were more adventurous and wanted to go full AWS Lambda (or a similar FaaS offering) and embrace FinOps to its purest form.

The only drawback is that it was expensive (let’s ignore complexity for now, shall we?). That didn’t matter when the money was flowing, but now the music has stopped, and everybody is intensely staring at their cloud provider bill and wondering what they can do to pay a fraction of it.

What is next?

Downsizing all things.

BeforeAfterComment
Microservices/FaaSMonolith(s)“Collapse” multiple codebases into one and deploy as a single unit.

The hope is that teams have become more disciplined at modularising (unlikely) and “build systems” have become more efficient in managing large codebases (possibly).
Messaging (Kafka et al)Avoid middleware as much as possibleMiddleware is expensive technology. With monoliths, there will be fewer network calls that require it.

Direct communication (e.g., HTTP, gRPC) will be the standard (again) when necessary. Chuckier monoliths will reduce network traffic compared to microservices
NoSQLRelationalMany NoSQL databases optimise for high throughput / low latency / high durability, which will happily be sacrificed for cost savings. Relational databases are easier to operate and run yourself (i.e., self-host), which is the cheapest option (some NoSQL, like CosmosDB or DynamoDB, can’t be self-hosted).

On the complexity side, relational databases are seen as easier for developers to understand (until you see things like this).
Stream ProcessingGone except for truly big dataStream Processing is expensive and complex. Most businesses won’t care enough about latency to pay for it, nor will have volumes that require it.
KubernetesCloud-specific container solutionsWe should see a transition towards more “Heroklu-like” execution platforms. It will be a tradeoff between flexibility (with K8S offers bucketloads) and cost/simplicity.

Sometimes, containers will be ditched too and replaced by language-specific solutions (like Azure Spring Apps) to raise the abstraction bar even higher.
Multi-region / Multi-AZ deploymentsNo multi-region unless compliance requirement.
Fewer multi-AZ deployments		Elon has proved that a semi-broken Twitter is still good enough, so why wouldn’t companies building less critical software aim for 3-5 9s?
Event-Driven ArchitectureHere to stayThis approach isn’t more or less expensive than Batch Processing (if anything, it’s cheaper) and still models business flows more accurately.

What are we gaining and losing?

Microservices are neither the silver bullet nor the worst idea ever. As with most things, they have PROs and CONs. If we ditch them (or push back harder against their adoption), we will win things and lose things.

What do we win?

  • It is easier to develop against a single codebase.
  • Local testing is simpler because running a single service in your machine is more straightforward than running ten. Remote testing is also more accessible, as hitting one API is less complicated than hitting many across the network.
  • It is also easier to deploy a single service than many.
  • Easier maintainability/evolvability. When a business process has been incorrectly modelled, it is easier to fix on a monolith (with, ideally, single data storage) than across many services with public APIs and different data storages.

What do we lose?

  • Once a codebase is large enough, it is tough to work against it. Software is fractal, which is also valid for “build systems”: you want to divide and conquer.
  • Deploying a single service can be more challenging if multiple people (or, even worse, teams) need to release changes simultaneously. More frequent deployments can alleviate the problem, but most companies don’t go from a dev branch to PROD in hours but days/weeks.
  • The blast radius for incorrect changes will be higher. Systems are more resilient when they are appropriately compartmentalized.
  • Organisations growing (are there any left?) will struggle to increase their team’s productivity linearly with the headcount when the monolith becomes the bottleneck for all software engineering activities.
  • FinOps and general cost observability against business value will massively suffer. A single monolith will lump everything together. With multiple teams involved, it will be harder to understand who is making good implementation decisions and who isn’t, as the cost will be amalgamated into a single data point.

Summary

Microservices are not dead. However, they are suspicious because they are expensive in terms of infrastructure cost and, indirectly, engineering hours due to their increased complexity. However, they are also crucial to unlocking organisational productivity as the engineering team grows beyond a bunch of guys sitting together.

As the industry turns its back to FAANG practices and we sacrifice various “-ilities” on the altar of cost savings, the future of microservices will be decided based on how often we identify when they are the absolute right solution and how well we articulate its case. When in doubt, the answer will be (and perhaps it should have always been) ‘NO’.

As a parting thought, I have been involved in 3 large-scale monolith refactors/rewrites to microservices. All these projects were incredibly complex, significantly delayed and more of a failure than a success (some never entirely completed). Starting with a monolith is, most of the time, the correct answer. However, delaying a transition to smaller, independent services is almost always as bad (if not worse) than starting with microservices would have been in the first place. We are entering a new era where short-time thinking will be even more prevalent than before.

Software is fractal

Fractal image generated by ChatGPT

I have been building software for nearly 20 years and keep stumbling upon the idea that software is fractal. It is like a nagging feeling that I have been unable to concretise. This post is a (probably poor) attempt at it and why it matters.

What does “fractal” mean?

Let’s ask ChatGPT, the source of all modern knowledge.

A fractal is a complex shape that looks similar at any scale. If you zoom in on a part of a fractal, the shape you see is similar to the whole. Fractals are often self-similar and infinitely detailed. They can be found in nature in patterns like snowflakes, coastlines, and leaf arrangements.

ChatGTP, 2024

In other words, fractal structures exhibit similar shapes (and, potentially, behaviour) at different scale levels (i.e., zooming in and out). Thus, one can use a similar set of concepts and rules to understand them regardless of where in the scale you focus.

Examples of software “fractalism“

What better way to prove that software is fractal than with examples. Pretty much all software systems are a combination of the following components:

  • Code, which captures desired behaviour.
  • Messages, which represent communication between code components.
  • Load Distribution, which guarantees the best possible performance.

Example 1 – Code

First, let’s look at how we capture “behaviour”, i.e., code that executes what we want the software to do. While not the lowest level, a reasonable low level would be classes and functions capturing code written by developers according to specifications that capture the desired behaviour.

Every class or function offers a series of “promises” (à la Promise Theory) about what it can do. Classes and functions use other classes and functions to compose higher-order behaviours. Modules and components aggregate classes and functions, producing new promises for more complex behaviours.

Eventually, modules and components are built up to (micro/macro)services, resulting in fully-fledged systems.

At every level, we forfeit details. The newly formed aggregate hides information about how it implements its promises, which isn’t relevant for those consuming them. It is crucial to design the appropriate APIs (aka, boundaries) and restrict premature abstractions with incomplete information (see Rule of Three).

Example 2 – Messages

It is all good to package your code with the appropriate boundaries and promises, but it will only be helpful if someone/something uses it. Users and machines decide to consume your promises, but they require a way to “exercise” them. Messaging happens again at all possible layers.

Clients and servers can be of any nature: at a low level, they can be two classes or functions. For example, one function A invokes another function B. B expects a given set of parameters, which A prepares and “sends” in its invocation. This is akin to a message from A to B, which waits for the result. The interaction can be async/sync, blocking or non-blocking; it doesn’t matter. The mechanics are the same.

Object orientation done right was all about message passing. Smalltalk, the OG object-oriented language, represented every interaction as a message sent to an object.

Components and modules rely on similar interactions. They can be as low level as between classes and functions (i.e., direct invocation through memory address) or more abstracted (like in-memory event buses like Spring Application Event).

Once we move further up and we need to cross the network to communicate, messages become more explicit. Events, commands, notifications, etc., represent messages that different code components/services exchange in a (not always) beautiful choreography to implement complex behaviours. In this layer, you find REST APIs with JSON payloads, gRPC (or, if you are unlucky enough, previous interactions like CORBA, DCOM, Java RMI, .NET Remoting, etc.), Kafka/Avro, etc.

At the top, we have systems interacting with other systems. It is the same metaphor (and, quite often, the same protocols and formats) but with reduced trust and a more straightforward boundary/interface.

Some components repeat themselves across all levels:

  • Sender / Recipient information
  • Body (parameters and values)
  • Metadata (time reference, size, integrity checksums)

Example 3 – Load Distribution

So far, we have behaviour represented as code and “communication intent” captured as messages. We now need to make sure those intents travel between code components. Enter “Load Distribution”.

At the lowest level, we have memory addresses (i.e., find my message at the beginning of this pointer and read consecutively) and the kernel scheduler assigning CPU time “equally”. This is both a way to distribute the message representing the intent and implement a form of load balancing since multiple code components “compete” for scarce resources. These components “wait” until resources are assigned; in other words, they queue. We scale by throwing more CPU cores to every problem (if we are lucky enough to have a parallelisable problem).

Moving up the stack, we find thread pools and managers that aim to multiplex the underlying physical (or virtual) CPUs as exposed by the OS. It is the same premise but applied a a higher level of abstraction with added support from the tooling. We start dealing with various types of explicit (in memory) queues and other synchronisation objects.

Things become more interesting when we get to the service level and the network is involved. Addresses move from memory to explicit (i.e., IP protocol), Load Balancers become network appliances, and where we once threw CPU cores at problems, we now throw instances/containers (if we are lucky enough to have a share-nothing architecture).

At the top level, we start thinking of distributing load across (cloud provider) Regions. While uncommon from a load perspective (most companies don’t need this), it can be required for availability/redundancy purposes. At this point, we have zoomed out so much that the whole system deployed to a set of Availability Zones in a single Region is just a node in the graph.

Why does this all matter?

I believe all of this matters because, when problems, patterns and structures repeat at a certain level, solutions become reusable.

For example, our mental model should not be different when we test a class/function or a whole service. What differs is “how” we apply that mental model.

AspectClass/FunctionService
BoundaryMethod/Function signaturePublic API
ArrangePrepare input data
Set dependency expectations (e.g., mocks, stubs)
Inject dependencies (other classes)		Prepare input message
Roll out infrastructure (database, message bus)
Prepare required state (e.g., preload database)
ActInvoke method/functionSend “message” to public API
AssertAwait results
Validate returned data
Verify dependency expectations		Await results
Validate response message
Validate state mutations

This is just one of many examples where, while the specific change, the mental model remains the same. Other examples include:

  • (Some) SOLID principles apply at multiple levels
    • Both SRP and IS are about granularity, which leads to Microservices architecture.
    • Extending is always safer than modifying, whatever your boundary.
    • The Inversion of Control that DI seeks can be achieved at service-level injecting client libraries that decouple actual service implementation(s).
  • Data structures showing up at multiple levels
    • Hashmaps (along with lists) are the most common data structure in software engineering (every JavaScript object is basically one).
    • Key Value databases are wildly popular and pretty much the same thing, at a different scale.
  • Contracts as a way to enforce promises
    • At the language level, we have method signatures with types (if you are using statically typed languages).
    • At the system level, we have attempts like Protobuf, Avro, JSON Schema and the whole “Data Contracts” movement to replicate.

If this rings true, then we should focus more on the rules that handle these commonalities and less on the specifics. If we see these patterns emerging, solutions that we know well in one scale can apply to other scales, just like your knowledge about SQL helps you with multiple databases or your mastery of Java translates (to a degree) to Kotlin or C#.

Summary

This metaphor needs to be more consistent (i.e., there will be many cases where it doesn’t apply) and complete (i.e., I’d need to dedicate the rest of my career to finding all the missing examples).

However, I believe once you start “seeing it”, it cannot be “unseen”. You will find more and more cases where your knowledge reapplies and there is a feeling of familiarity.

Some might call it “common sense,” but I prefer to think of it as more elegant than that.