The Future of Analytics Engineering

For Analytics Engineers, SQL is the most visible part of their work. It's what gets committed, reviewed, and blamed when pipelines break. But the real craft lies underneath this surface.

Last year I opened a Knowledge Exchange session at Xebia with a question: "How many times have you struggled explaining to others what you do for work?" The room laughed because everyone had felt it. If you work in Analytics Engineering, you know the answer people reach for: "I'm the bridge between data engineering and the business." Which is true. And also tells you almost nothing about what the job actually involves.

What it actually involves is system design. Naming conventions. Release cadences. Decisions about who is responsible for what. Data models that are modular, reusable, reliable, and built to change without breaking. The architecture and design principles required to produce data that the business can rely on and act on. That has always been the sharpest edge of the Analytics Engineering role. SQL was how you expressed those decisions in code. It was never the decision itself.

The Tool That Shaped the Role

In 2019, a post on Locally Optimistic gave the role a name: Analytics Engineer. The dbt community adopted the term and made it the vocabulary of the modern data stack, putting SQL front and center as a first-class citizen of transformation pipelines. That was a good thing. SQL became a shared language across data teams. Transformation logic moved out of scattered scripts and into version-controlled repositories with tests, documentation, and review gates.

But it also stretched the role. AEs inherited the full value chain: ingestion, transformation, serving, dashboards, data storytelling. Broad skills, high execution load. The deeper architectural work was still there, but it had to compete for time and attention with the daily execution demands.

As a consultant, I had an advantage: I landed on teams with critical eyes. Most of the time, the changes I proposed were not about SQL at all. They were about process. "What will be our naming convention?" "How should we release to production, on what cadence, who is responsible?" These processes are the backbone of repeatable outcome. I have seen them move teams from firefighting to proactive execution, repeatedly, across different clients and stacks.

That instinct, encoding the standard and enforcing it at scale, is exactly what made dbt successful. And it is exactly what the next shift in Analytics Engineering is built on.

The First Time I Used Claude Code

I had been using agentic coding in my IDE for a while. Started with Cline on VSCode, had my set of rules, but it was a very involved and guided process. I was still the one writing the code. The agent was helping with the chores and reviewing. Very much like an open conversation, flowing naturally in the context of one file being modified.

When you move to agentic coding with a team of agents, things shift quickly. You are modifying the full repository and touching many files at a time. There is no time to have the IDE open and check every individual change. You need to be very attentive to what the team is doing, catching wrong assumptions, giving context, guiding direction. The dynamic is quite different.

If you try agentic coding the same way you try a tool like ChatGPT, you will be overwhelmed very quickly. Generic prompts, kitchen-sink context windows, and rookie mistakes like not documenting your intent before a compaction event removes the memory your agents have, compounded by the number of agents running in parallel. I made all of these mistakes.

What Working With a Real Team Taught Me

The shift that changed everything was simple: I stopped treating it like a better autocomplete and started treating it like a team.

That meant applying the same tools I already knew from working with human teams. Planning, refinement, and documentation. Architecture Decision Records written before a single line of code. Business requirements gathered and translated into atomic, self-contained, well-scoped tickets. Agile techniques that exist precisely because building software with other people is hard.

Before starting to code, a lot of things need to happen. That sentence sounds obvious. In practice, it is the hardest discipline to maintain when the agent is sitting there, ready to go, and the temptation is to just start.

What surprised me most was the volume of output. The agentic team was fast. The code was piling up. Reviewing it all was becoming impossible. So I did what any team lead does when output quality drifts: I introduced retrospectives.

"What went wrong and how can we do it better?" Run that question every cycle, fix the identified issues, move forward. The improvement in output quality was significant. Combined with TDD (writing tests before the code existed), CI/CD cycles, and adversarial code reviews that were explicitly instructed to be brutally honest, the team was catching real bugs I would never have found myself.

If you use an agentic team as a mind dump, you will get a nice proof of concept. If you want production-grade code, you need production-grade frameworks applied to your work. The same frameworks software engineering teams have been refining for thirty to forty years.

Encoding the Standard

I caught myself repeating the same instructions at the start of every session. "Remember to use TDD. Follow DevOps best practices. Write the ADR before implementing." The same briefing, session after session, different models, different contexts.

That is not sustainable. And it is also a pattern I recognised. It is exactly what happens when a team grows and the senior engineer who carries all the standards in their head goes on holiday. Things drift.

The solution is the same one dbt solved for SQL: encode the standard, not the reminder. I built a Way of Working skill that gets triggered at the start of a task. A set of hard gates gets enforced. TDD, adversarial review, architectural documentation, phased delivery, retrospectives. The agents follow the agreed patterns consistently across sessions and across models. I do not repeat myself. The work can be picked up in a completely new session with no loss of quality or direction.

That skill is the natural extension of the AE instinct. The conventions you encoded in your dbt project structure did not care which analyst wrote the model. The skill does not care which model runs the session.

The Barrier That No Longer Exists

Here is a detail worth naming directly. A while back I needed to build a documentation platform for dbt: a single-page application with navigation, lineage exploration, and health dashboards, built in React and TypeScript. The problem was that I had close to zero TypeScript experience. The task required a skill set I did not have.

My agentic team did. They wrote the TypeScript, applied guardrails and industry best practices on every turn, and kept the code in check. I did not have to validate every type or chase every semicolon. I am now working on a separate tool built in Rust. Same situation. Close to zero experience in the language.

What I do have is an understanding of how programming works: the patterns and frameworks that produce reliable, testable, maintainable code. The architectural instincts I built working with Python and SQL transfer. But I will not pretend that knowledge alone is enough. The other half is knowing what you do not know.

When I encounter a decision I cannot make from first principles, I ask the agents to do the research: propose the industry standard, evaluate what fits the use case, and write an ADR that documents the choice and the reasoning behind it. That is not a workaround. That is exactly what a good engineering lead does with a team that has deeper specialist knowledge in a given area.

The language barrier was the last execution dependency the AE role carried. Agentic coding removed it. What remains is judgment: knowing which decisions require yours and which can be delegated.

What the Role Becomes

With the execution layer delegated, the Analytics Engineer can operate at full depth. I can iterate on data model design ideas with a client in real time, mock the data to show the business impact, and have automated diagrams and documentation keeping pace with the changes. Work that used to take months can be done in a few weeks, to a solution that actually solves the problem rather than the problem as it was understood six months ago.

The human interface, gathering requirements, iterating with the business, translating between what the data can say and what the organisation needs to hear, becomes the primary work. The hardest problems, finally without the queue.

The Shift Is Already Happening

Are you experimenting with agentic coding but feel that you are not coding that much anymore? If you are enjoying talking with your agentic team about goals and projects, it means you have already made the shift to agentic analytics engineering.

If you have not tried it yet, one word of caution. It can be addictive, how fast and how well you can deliver something that answers a question that was just an idea inside your head a few minutes before.

The future of Analytics Engineering is not a prediction. For some of us, it is already Tuesday.

(Of course, handing over execution to autonomous agents requires serious guardrails to be safe and effective. In my next post, I'll share how I built a security baseline and challenge protocols to harden agentic workflows for daily, continuous use.)