How Small Can the Server Room Get? I Went to GTC Berlin for an Answer

My father used to tell me about computing at university. He punched his program onto cards, handed the stack in, and waited overnight for it to run. If there was a compilation error, he fixed the card and queued again, so one typo could cost him a day.

I thought about that story at GTC Berlin in October. Today, that scene plays out in research labs: PhD students queueing for time on the one machine that can run the models they need. What I saw in Berlin is that the queue is getting shorter, because the machine is shrinking to a size that fits in the same room.

From rack to backpack

Jensen Huang's keynote covered the whole range. At one end are the AI factories: Vera Rubin, with racks where dozens of GPUs work as one. He put the progress in one sentence: "In the course of the last 10 years we've improved AI performance by 1 million times."

Jensen Huang holding up the Vera Rubin liquid-cooled compute tray on stage at GTC Berlin. Photo by Julian de Ruiter.

At the other end of the same keynote he held up a computer you can carry, which is the end of the range that matters for a small team or a lab.

At the back of the exhibition hall, a demo by an NVIDIA engineer made the point better than any slide. Two Dell towers sat under a wooden desk running DGX Station hardware. Next to them, a sign: thirty developers working off the network, at the same time. On the screen, a team of thirty AI agents was building and shipping a website with no internet connection, all running locally across those two boxes. What kept it safe deserves its own post, but the immediate takeaway was clear: the hardware under that desk was enough for an entire team.

The two DGX Stations running thirty AI agents off the network.

What I said before

In my newsletter, I have explored this idea for several months.

"Owning your inference" means running AI models on machines you control, instead of renting every answer from a cloud provider per token. Your data stays in your building, and after the purchase your cost is mostly electricity.

In September I asked how small the server room could get. My answer was that a cluster of a few capable machines can serve a whole team at once, and that a focused investment in such a cluster every few years can make more sense than building a full server room. You keep availability, sovereignty and privacy, as long as you manage it well.

I also wrote down the catches. The week before, I found that hardware prices were going up, not down, and that on-premises hardware only pays off if you keep it busy. Back in June I was more sceptical still: "I do not think building your own GPU cluster is the answer for most teams." Walking through Hall 2 in Berlin, that sentence was challenged faster than I expected: a cluster of capable machines is now within reach of a small team. The catches are still real: most of these machines cannot be ordered yet, shipping will take time, and rising memory prices keep pushing the cost up, so how much you need to invest remains an open question.

Two shapes of the same idea

The hardware on the GTC floor came in two shapes.

In your backpack. The RTX Spark is a new line of PCs NVIDIA built with Microsoft: one petaflop and 128 GB of memory. On stage, Jensen held it in one hand and compared it with the DGX-1 he delivered to OpenAI in 2016, which had the same specs and "took two people to carry upstairs".

Give one to each person, or stack a few into a cluster. The conference floor had the newsletter's "cluster of small machines" on display as a product: a rack of DGX Sparks, wired together with their own switches and storage.

A small cluster in a rack: DGX Spark units and ASUS Ascent GX10 systems, wired together with dedicated switches and storage.

In the building. The DGX Station takes the other route and puts everything in one deskside box: 20 petaflops of FP4 AI performance, and two can be linked together, as in the demo. I was told at the booth that each costs around 100,000 dollars and draws about 1,800 watts. For a small office, a research lab or a university department, that buys one machine everyone shares, with no data leaving the room.

DGX Station for Windows from three vendors. NVIDIA announced availability arriving in December.

In a lab with a single DGX Station, my father's punched-card queue returns: researchers scheduling token budgets and booking slot times on the quiet box down the hall.

Who should care

Pay-per-token stops making sense when the work never stops or the data cannot leave the building. These are the use cases I would look at first, and each one was on the floor in Berlin:

The hybrid architecture

Local compute complements cloud infrastructure rather than replacing it. The architecture that makes practical sense is a clear division of labor: fast local models handle high-volume throughput, while cloud-hosted frontier models manage high-level orchestration and validation. Jensen described the same split in the robot demo: run everything locally "except when I need to burst larger workloads to cloud".

Tokens were the theme of the pre-keynote sessions, where one line summed up the mood: "In the age of AI, intelligence tokens are the new currency." A hybrid setup keeps that spending predictable and within budget, while frontier models stay available when the work needs them. Routing the right work to the right model on the right infrastructure is the new budgeting act, and it reminds me of the years when data and analytics teams had to decide what ran on-premises and what ran in the cloud. Another speaker in the same sessions described the shift I care most about: "rather than being only a token consumer, you can also be a token producer in this token economy... It gives you some agency. It gives you choice."

What changed my mind, and what didn't

What changed: two deskside boxes running a team of thirty showed me that, for a small team or a lab, owning its inference is now a real option without a server room.

What did not change: expensive hardware only pays off if it is busy, and local models still struggle with long, multi-step reasoning. Those are the questions to answer before you buy.

A petaflop that took two people to carry upstairs in 2016 now fits in a backpack. My bet is that this hardware keeps getting smaller and more affordable, sold in bundles that make sense for a ten-person team, though that is a prediction and time will tell.

How to keep thirty agents on those two boxes safe and coordinated is the subject of the next post.