The Hidden Cost of Databricks' ai_parse_document in Production

ai_parse_document + ai_query makes PDF extraction look easy. Here's an honest evaluation of the trade-offs, a free deterministic alternative, and what to weigh before you build.

A quick note: this is a reupload. I first wrote about this back in February 2026, which in AI time is basically a lifetime ago. The core trade-offs below still hold.

A colleague and I were looking at a healthcare project that had hit a familiar wall. The structured data was usable, dates, diagnoses, billing records, but the information people actually needed lived elsewhere: discharge summaries, referral letters, intake forms, nursing notes. All unstructured and inconsistent.

We were weighing OCR tools, custom parsers, and third-party APIs, the usual cost/accuracy/maintenance trade-off, when Databricks released ai_parse_document, which seemed to cut straight through all of that.

A few lines of SQL, and messy PDFs came out the other side as structured JSON. The first time you see it work, it genuinely feels like magic, a proof of concept in an afternoon.

That simplicity is a bit deceptive, honestly. Once you move past the demo, the problem changes shape. It's no longer about extracting fields from one document. It's about running that extraction as a system: reliably, repeatedly, at scale.

That shift, from query to system design, is where the real problems surface. That's what this post is about.

The examples use Databricks, specifically ai_parse_document and ai_query, in SQL or notebooks. You'll need Databricks Runtime 17.1+ and basic familiarity with Delta tables and the medallion architecture.

The Starting Point: ai_parse_document + ai_query

Before the SQL, one distinction matters: not all PDFs are the same.

Digitally-born PDFs have real, selectable text baked in, think of saving a Word document as a PDF. Scanned PDFs are photos of pages: readable to a human, but just an image to a computer until OCR reconstructs the text. Extraction quality differs sharply between the two.

Figure 1. Digitally-born PDF (left) vs scanned PDF (right).

Here's what ai_parse_document and ai_query look like in practice:

Figure 2. The proof-of-concept query: parse PDFs with ai_parse_document, extract fields with ai_query.

Two steps, really:

Parse. The text_extracted CTE calls ai_parse_document on each binary file from read_files. It handles both document types, reading embedded text directly for digital PDFs, running OCR first for scans. The result is a structured document.elements array, text blocks, tables, section headers, which transform and concat_ws flatten into one string, full_text.

Extract. The outer SELECT sends that text to ai_query with a prompt: extract five fields as JSON. patient_ref, document_date, document_type, primary_diagnosis, follow_up_required, enough to build a queryable patient cohort and flag who needs a follow-up. It gets the fields right most of the time. When something's ambiguous or unstated, it infers, and doesn't always tell you it did.

Here's a single row from the result, before any cleanup:

Figure 3. A single ai_query response, before any cleanup — prose, fenced JSON, and an inference note all in one string.

A few things to notice. extracted isn't a JSON object, it's a plain text string: a prose sentence, JSON wrapped in markdown fences, then a "Note:" explaining the model's reasoning. All one continuous string, and it needs cleanup before the JSON inside can be parsed.

This is inference, not extraction. The model made a judgement call on follow_up_required and explained itself here, but it won't always do that, and it can be confidently wrong either way.

Where the Cracks Show

The query works. The first run isn't the problem, the second one is, and it surfaces four issues a notebook never shows you:

Cost. Both functions charge on every run. No checkpoint means every run reprocesses every document, and every prompt change re-runs everything. The first bill isn't the problem. It's that every correction, every prompt revision, every reprocessing cycle reopens the meter.

Healthcare documents arrive in irregular batches, end-of-month exports, legacy migrations, corrected records re-sent in bulk. At medium-complexity rates (~$4.20–$4.55 per 1,000 pages), a 30,000-page corpus costs $126–$137 to parse, manageable once. But prompt iteration isn't a one-time cost. Every refinement re-runs ai_query across every document, and onboarding a new hospital's format can trigger a full re-parse.

Duplicates. Hospitals resend documents, a corrected discharge summary, a referral letter re-sent after a system migration. Nothing in the query above catches that. Both versions land in Silver: the same patient shows up twice, and if the correction changed the diagnosis, both versions of it do too.

Non-determinism. The uncomfortable part: your pipeline can be "correct" and still produce different answers on different days. A rule-based parser always returns the same output for the same input. An LLM doesn't. ai_query defaults to temperature 0, which helps, but even temperature 0 doesn't guarantee identical outputs — floating-point rounding and GPU parallelism introduce variation regardless.

In practice, follow_up_required can flip between true and false for the same document on different runs, no error, no warning, just a quietly different result. In a regulated environment, that breaks auditability: the report changes without the underlying data changing.

You can mitigate this, run extraction multiple times and take the most common result, or build a labelled test set to check your prompt against known-good outputs. That second option earns its keep once you're supporting a new document type or hospital format, without it you can't tell if a prompt change helped the new format without quietly breaking an old one. Just know that multiple passes multiply your ai_query cost directly. Non-determinism and cost aren't separate problems here.

Input noise. Every page carries something like "Confidential – Amsterdam UMC – Page 4 of 11", stamped on it, consistent per hospital, meaningless for extraction, and it goes to the LLM too. Because ai_parse_document handles the binary internally, noise arrives pre-baked into the text, so you clean it up in Silver instead of before the LLM ever sees it. The usual symptom: a primary_diagnosis that starts with the hospital letterhead instead of the actual diagnosis.

None of this is fatal. All of it needs system design the demo hides from you.

Bronze: Where the LLM Stops

Once you stop treating the query as the product and start treating it as ingestion, streaming with checkpointing is the natural design.

One thing worth flagging first: the real PII concern is the full document text. Discharge summaries and clinical notes carry names, dates of birth, diagnoses, and all of it goes to ai_query. Before running this on real patient data, either strip identifying information before it reaches the LLM, or confirm your data processing agreements cover routing clinical content. Out of scope here, as mentioned, but worth being aware of before you build.

Figure 4. From PDFs to queryable clinical data. cloudFiles streams new files automatically; ai_parse_document parses each binary into a structured elements array; the assembled text goes to ai_query for field extraction; Bronze stores the result with a streaming checkpoint; Silver unnests the struct into a flat, queryable table.

The instinct is to stay in SQL: hash each file, track what's processed in a Delta table, version by prompt. That works, but it's boilerplate you write and maintain yourself.

Structured Streaming is the cleaner path. Spark manages state through checkpoints, new files flow through, already-processed ones are skipped, and nothing re-runs the LLM unnecessarily. Less boilerplate, less to get wrong.

One practical note: patient_ref is one hospital's local ID. The same patient gets a completely different ID at the next institution. Store the source hospital identifier alongside every patient_ref from the start, one line now, months of pain if you skip it.

The pipeline splits into three streaming tasks, each with its own Delta table and checkpoint:

Figure 5. The three streaming tasks — parse, assemble text, extract fields — each writing to its own Delta table and checkpoint.

On duplicates: cloudFiles tracks files by path, so a re-sent file under the same name gets skipped by the checkpoint. A corrected document under a new filename is a different path, though, and lands as a new row alongside the original. Dedup belongs in Silver, where ROW_NUMBER() OVER (PARTITION BY patient_ref, document_date, document_type ORDER BY _commit_timestamp DESC) keeps only the most recent extraction per logical document.

The three-task split also helps with cost. Parsing and extraction are separate, so you can update the prompt without reparsing anything. ai_parse_document charges per page, rerunning it on historical files every time you refine extraction is wasted spend. If a hospital uses "patiëntnummer" instead of "patient_ref", or you want to extract medications too, reset only the Task 3 checkpoint and rerun ai_query against the already-parsed text. Parse cost for existing files stays at zero.

Stamp each run with a new prompt_version, and old and new generations coexist in the table, audit trail intact.

Databricks publishes a Databricks Asset Bundle with this same three-task pattern as a deployable reference. In production, the stages typically run as a Databricks Workflow: parse, assemble, extract, rebuild Silver, run a quality check. If Bronze fails, nothing downstream runs.

Silver: Mostly Just SQL

Bronze does the hard work. Silver is mostly SQL: flatten the extracted struct into a queryable table, and strip the noise that arrived pre-baked in the text. A SQL UDF keeps that cleanup reusable:

Figure 6. The strip_noise SQL UDF, reused across every free-text field.

Apply it to any free-text field where letterhead might bleed in, strip_noise(extracted.primary_diagnosis) below. Each hospital gets its own header pattern; build up a library as you onboard new institutions.

Figure 7. The Silver query: flatten the extracted struct, clean the text, add derived date fields.

follow_up_due_date here is a placeholder on a fixed 30-day window. Real follow-up intervals vary by diagnosis and should ideally come from the document itself.

Here's a typical row in bronze_documents_structured, and the same record after Silver flattens and cleans it:

Table 1. The same record, before and after strip_noise and the Silver query.

Same principle as Bronze: do the expensive work once, don't repeat it. Materialise Silver as a table, not a view. You pay a small compute cost once per run, and everything downstream reads from a clean, pre-computed dataset. Analyst queries are faster, Bronze isn't hit repeatedly, and the LLM only ever runs at ingestion. New prompt versions land as new rows, and you just update the filter.

At this point you have a clean, flat Silver layer, structured fields, noise stripped, prompt version tracked. What you build on top in Gold depends entirely on your use case, so that's out of scope here too.

But before you build any of this, there's a question worth asking first.

Not Every Pipeline Needs an LLM

A large share of healthcare documents follow predictable templates. Discharge summaries from the same institution look roughly the same. Referral letters follow a standard structure. Where that's true, an LLM is the wrong tool, you're paying for flexibility you don't need and accepting non-determinism you can't afford.

OpenDataLoader PDF is worth knowing about here. Open-source (Apache 2.0), and its local mode is fully deterministic, using the XY-Cut++ geometric algorithm for reading order and table detection, no model involved. Parsing cost drops to zero, and you can still add ai_query on top for field extraction if you need it. The limitation is template variance: if hospitals label the same field differently ("Patiëntnummer" vs "Patient ID"), you'll still need an LLM to normalise them. But where templates are consistent, deterministic parsing plus regex is cheaper, faster, and fully reproducible.

Other tools sit at different points on the same spectrum, layout-aware ML parsers like Docling or MinerU in the middle, fully LLM-native tools like LlamaParse at the other end. The trade-off holds across all of them: more convenience and flexibility, less determinism and cost control.

Which Approach Is Right?

The honest answer depends on your documents:

Table 2. Match the approach to the documents.

When ai_parse_document Is a Good Fit

ai_parse_document isn't a bad tool. It favours fast setup, broad document support, and flexible extraction over strict determinism and cost efficiency at scale. That makes it a good fit for:

In those scenarios, the convenience genuinely outweighs the trade-offs.

The Databricks-native route is the most convenient starting point, and sometimes the right end state, but not always where you land after a full production evaluation. If you can't tolerate output drift, don't start with an LLM. Most teams don't need smarter parsing, they need more predictable systems.

I had a great time working on this. If you're building something similar, or want to compare notes, find me on LinkedIn.