Event-driven plumbing. When plain SQS + Lambda beats EventBridge Pipes

As a proponent of reactive, event-driven systems, I often use Amazon EventBridge to wire together parts of a distributed architecture. It is a great service that takes a big part of the heavy lifting when you design and operate such systems:

In addition to all of the above, EventBridge Pipes provide a highly available pipeline to filter and enrich event messages. They easily convince you to be the "right" managed glue between a source and a target.

On the economics side, you pay $0.40 per million requests after filtering (how generous, isn't it?), where each 64 KB chunk of a payload is billed as 1 request. Let's not forget that the usual charges for other services also apply.

And that's the whole point: a pipe with a Lambda-based enrichment doesn't really land for me. Because why would you pay, in money and in lost failure granularity, for something your existing serverless primitives already do better?

One remark about the stage here, though. When you work on an event stream and use some HTTP API for enrichment, EventBridge Pipe is a no-brainer for a low-code integration. But, for the "SQS - Lambda - SQS" type of pipe, the game isn't worth the candle. I wanted to like Pipes and use them for this, I did...

What Pipes promise, and the one job we're testing

An EventBridge Pipe, in a nutshell, is a point-to-point integration building block:

In addition, they provide built-in error handling — retries with exponential backoff. And in the event of nearly continuous failures with enrichment or the target, the pipe will automatically disable itself. And, of course, you can configure the Dead-Letter Queue for every stage of the pipe.

As a bonus, AWS provides you with lots of built-in metrics grouped into handy dashboards for every stage. Sounds like a dream, almost a pipedream.

So, with all this in mind, I leaned on the EventBridge Pipes as a natural fit for my job:

What could go wrong?

Where enrichment batching betrays you

EventBridge Pipes support message batching: take N messages and send them to the enrichment or target. Which is great, just like with SQS batching, right? Well, no.

If you're accustomed to SQS event sources for your Lambda functions, you know the partial-batch failure reporting — a special shape of Lambda function response, that allows you to report back which messages failed. Effectively, this sends only failed messages for retries, rather than the whole batch.

So, the problem with Pipes Enrichment is that it doesn't support a partial-batch failure mode: if your enrichment function fails, the whole batch retries. The target stage got a full-fledged batch support, though with a quirk: failed messages are retried through the entire pipe, re-invoking your enrichment:

The batch-of-1 escape hatch, and what it costs

The "obvious" fix is to drop the enrichment batch size to 1, so any failure maps cleanly to a single message:

Mind you, this is the DLQ on the source queue, not the pipe.

Again, just like with regular SQS.

The batch-of-1 completely eliminates the cost lever of the Pipe — with 64 KB per request, you could fit multiple messages into 1 batch, billed as 1 enrichment request. With batch=1, the same five messages cost five requests, and in exchange, you get nothing extra. You have just got the same failure semantics that plain SQS + Lambda would give you for free...

Yes, in absolute numbers, the difference is tiny. However, this money isn't going to anything you use. Instead, you pay to neutralise something the service does that you can't leave enabled.

Biting the bullet and letting the whole batch of N to be retried opens another can of worms — you pay for retrying messages that succeeded the first time. Plus, your enrichment Lambda must be idempotent — the system state or the payload must remain the same if the message is received more than once.

The clever hack that wasn't

Sometimes, I am pretty stubborn — I won't let the idea go unless I've tried everything and feel it's not worth it. Well, that's also the reason why this write-up exists to begin with — to save you time and effort on what I've tried. You're welcome 😉.

So, the attack plan was:

Aha, how smart! Fasten your seatbelts, we're taking off to meet ... the wall — CouldNotDetermineMessageSource error.

If you send the message directly to the DLQ from your workload, it does not contain the necessary origin metadata for a redrive-to-source to succeed. You can make that work with a "redrive to a custom destination" and passing the source queue ARN, effectively re-adding the manual plumbing Pipes were supposed to remove.

What you actually want is already in the box

Let's step back to the serverless classics — a plain SQS → Lambda event source mapping supports ReportBatchItemFailures natively:

With this, you are getting exactly what Pipes enrichment cannot give you — you either:

A boring, familiar primitive working better than a new shiny managed service...

When a Pipe still earns its place

I mentioned this at the beginning already, but let's give it a bit more light.

EventBridge Pipes shine for:

Without Pipes, you'd have to hand-roll the integration, which, in turn, makes your team responsible for the consumer workload, retries and all that jazz.

My take here is against Pipes for local, deterministic, Lambda-based enrichment on an SQS source; it's not against Pipes as a whole.

The wrap

If you plan to use a Pipe for Lambda enrichment, check whether classic SQS + Lambda already does it more cheaply and with better failure control.

Also, think about this — if enrichment fails, do you have to throw an error and retry? Will it succeed the second time? Maybe, it's just pointless...

Or, is it safe from a business logic point of view to pass messages to the target even if enrichment failed?

The thing is that high error rates make your throughput back off:

The pipe throttles its execution due to a high error rate from the enrichment Lambda, dropping it to just ~240 calls per minute.

So, I changed my perspective on the enrichment failure — instead of throwing an error on messages that I couldn't process in any way, I just added a custom skipped: boolean field to its payload.

The target consumer down the line used the same filter — do not pick up anything marked as skipped: true. As a result, 10 times fewer errors, way better throughput, and errors that mean something — I could refine my enrichment logic, improve processing of edge cases, etc.

Reframing the failure fixed the throughput within Pipes; the cost and per-message-retry arguments against enrichment still stand. Whether plain SQS + Lambda beats even this tuned setup is the next question.

Stay tuned, to be continued.

Photo by Benjamin White.