Chai With JaiGPU Study Hall
Menu

Document AI in Snowflake

The Long Tail of
PDF Extraction
and LLMs

Finding the right PDF is no longer the whole problem. The hard part is turning one plausible model answer into a fact a company can actually trust—across the layouts, periods, units, scale notes, and reference gaps hiding in the long tail.

Take the 60-second trust test

One annual report. Two Snowflake paths.
One independent reference.

Start with a decision: would you let one plausible row into production?

Then inspect the Snowflake architecture and the evidence contract behind it.

60-SECOND PDF TRUST TEST

Would you ship this row?

Make a decision before opening the source detail.
APPLE INC.2023 FORM 10-K

CONSOLIDATED STATEMENTS OF OPERATIONS

In millions, except number of shares, which are reflected in thousands, and per-share amounts

Net sales
2023
2022
Total net sales
383,285
394,328
Source excerpt · statement page 31

MODEL OUTPUT

metric
revenue
value
383,285
unit
USD
period
FY 2023
VALID JSON · PLAUSIBLE VALUE

Can this row enter a production table?

01The demo can pass while the data fails

The dangerous result is the one that looks right.

Apple’s 2023 income statement prints net sales as 383,285. An extractor can return that exact number, label it USD, produce valid JSON, and still be wrong by six orders of magnitude. The scale note—“in millions”—sits above the table. The trustworthy value is 383,285,000,000 USD.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
Store the source, method, raw output, normalized fields, reference, and status together. Run the lab →

This is the part of document AI that a clean demo tends to hide. Retrieval can find the correct annual report while extraction chooses the comparative-year column, loses the scale, reads a subtotal, or cites text that does not support the number. The file is right. The row is not.

That distinction matters because companies do not want PDFs merely to become searchable. They want the facts inside financial statements, claims, filings, manuals, and operational records to enter systems that make decisions. “The model returned an answer” is the beginning of that job, not the acceptance test.

ContinueThe PDF long tail

02Document variation

A PDF is a delivery format, not a data contract.

PDFs share a file extension, not a data model. One has a clean text layer; another is a scan. A table may use borders, whitespace, rotated headers, nested columns, or an image. The current and comparative periods may sit inches apart from a scale note that governs both.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
Store the source, method, raw output, normalized fields, reference, and status together. Run the lab →

The long tail is the accumulation of these ordinary differences: shifted page indexes, negatives in parentheses, fiscal years that do not match calendar years, and reference systems with two facts for what looks like one metric. None is rare enough to ignore once a workflow touches thousands of documents.

The seasoned response is not a heroic prompt. It is a data contract around the prompt: preserve source identity, select pages explicitly, record the method and run, separate value from scale and period, save evidence, and report when no independent reference exists.

FILING AND REFERENCE ALIGNMENT

Select the reference independently of the model output.

DocumentAAPL/2023.pdf↔SEC accession
Selectionpages 30–35↔fiscal period
Questionrevenue↔US-GAAP concept
Experimentrun_id + method↔saved output
The filing accession, metric, period, and unit determine the SEC fact used for evaluation.
ContinueTwo Snowflake methods

03Method comparison

Snowflake gives you two useful failure surfaces.

The shortest path sends selected pages directly to AI_EXTRACT. The more inspectable path runs AI_PARSE_DOCUMENT in layout mode, stores page-level Markdown, and then sends that text to AI_EXTRACT. Both can target the same response schema; only one leaves behind an intermediate representation you can examine.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
Store the source, method, raw output, normalized fields, reference, and status together. Run the lab →

Direct extraction avoids an extra call and may preserve information that a text representation loses. Parse-then-extract creates more surface area, but when a table or reading order fails, the Markdown can show whether the error entered during parsing or extraction.

A useful comparison holds the file, pages, questions, schema, and reference facts still and changes only the path. Use a fresh run ID for every experiment. The result will not crown a universal winner. It will tell you whether inspectability earns its cost for this document family.

CONTROLLED COMPARISON

Hold the document and response schema constant.

PATH APDF pages → AI_EXTRACT

Fewer calls. Less intermediate state.

PATH BPDF pages → AI_PARSE_DOCUMENT → Markdown → AI_EXTRACT

More inspectable. More surface area.

Compare accuracy, inspectability, latency, and cost for the same document family.
ContinueResponse schema

04Model the answer

Make the answer carry its own audit trail.

A financial metric is not one value field. In the worked example, the answer includes the printed number, scale multiplier, unit, fiscal start, fiscal end, and supporting evidence. The evaluator normalizes the number, checks the semantic fields, and only then applies numeric tolerance.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
ANSWER CONTRACTSix fields travel together.
Printed value383285Signed numeric text
Scale1000000Multiplier declared by the statement
UnitUSDExact catalog unit
Fiscal start2022-09-25Required for duration metrics
Fiscal end2023-09-30Matches the approved filing
EvidenceNet sales … 383,285Statement line, header, and scale note

383285 × 1000000 → 383,285,000,000 USD · FY 2023 · evidence present

Store the source, method, raw output, normalized fields, reference, and status together. Run the lab →

That contract turns “the model was wrong” into diagnoses an engineer can act on. A period mismatch calls for a different fix than a missing quote. A parsing failure calls for a different fix than an ambiguous SEC reference. Agreement between two model paths is not a substitute for independent evidence.

This looks less glamorous than model selection because it is data modeling. It is also the layer that lets a reviewer trace a production row back to the filing and lets a team improve one failure class without destabilizing the rest of the pipeline.

RESPONSE FORMAT

Fields required to evaluate a financial metric.

Printed value
383285Signed numeric text
Scale
1000000Multiplier declared by the statement
Unit
USDExact catalog unit
Fiscal start
2022-09-25Required for duration metrics
Fiscal end
2023-09-30Matches the approved filing
Evidence
Net sales … 383,285Statement line, header, and scale note
A value is scorable only when its filing, period, unit, scale, and reference are defined.
ContinueEvaluation

05Ground truth and failure classes

Evaluate the system you actually operate.

Lowering temperature may reduce sampling variation. It cannot restore a missing scale note, select the right filing, choose the current period, or manufacture ground truth. A deterministic pipeline can repeat the same mistake with admirable consistency.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
FAILURE CHECKChoose an extraction failure.
RUN 01383,285 USD
RUN 02 · TEMP 0383,285 USD
EVALUATORVALUE_MISMATCH

Repeated result: The run repeated the same value without the ×1,000,000 scale.

Engineering response: Extract scale separately and normalize before comparison.

A repeated result can still fail on scale, period, evidence, layout, or reference coverage. Run the lab →

Evaluation therefore needs information the model did not generate. For public-company metrics, SEC Company Facts can provide an independent source, but the match still depends on filing accession, concept, unit, and fiscal period. If no usable reference exists, the result is unscorable—not a convenient pass and not a model failure.

Save the method, run ID, raw response, normalized fields, reference value, tolerance, evidence, and failure class. That record separates three things teams often collapse into one accuracy number: extraction quality, document coverage, and reference-data quality.

ContinueThe senior decision

06What leadership actually requires

Your architecture recommendation should be conditional.

A senior recommendation sounds like this: for this document family, under this evaluation set, choose the method with the stronger measured result; pay for the parse step when its inspectability lowers review cost or exposes failures the direct path hides. If the methods tie, report the tie. If cost was not measured, say so.

FIGURE 01 · EXTRACTION TEST BENCH

Compare two methods on the same filing

PDF EXTRACTION Annual report PDF
FL_FILTER_PAGES↓AI_EXTRACT(file)

One inference path; less intermediate state.

REFERENCE DATA Independent SEC fact
Accession
0000320193-23-000106
Period
FY 2023
Unit
USD

Matched by filing identity and period—not by resemblance to the model answer.

MATCH BY filing · metric · period · unit · run SQL EVALUATION
01Select
02Extract
03Normalize
04Compare
05Explain
06Rerun
Change one variable. Start a new run. Keep the reference fixed. Run the lab →

Pass rate belongs only over scorable metrics. Report reference coverage separately, then track extraction coverage, failure mix, latency, inference cost, manual-review rate, and repeated-run stability. A high score on the easiest referenced documents can conceal the cases the business actually fears.

Snowflake is useful because the file reference, parsed pages, raw extraction, structured facts, and SQL evaluation can remain queryable together. But the larger opportunity is platform-agnostic: turn document families into measured data products with visible uncertainty and a review path. That is the engineering work in the long tail.

Production measurement

What to measure
before production.

  1. 01Source coverage

    Which document families, formats, languages, and page layouts are represented?

  2. 02Reference coverage

    How many requested fields have an independent, unambiguous comparison value?

  3. 03Extraction coverage

    How often does each method return a usable response rather than an error or missing field?

  4. 04Pass rate and failure mix

    Among scorable fields, which methods pass and which obligations fail?

  5. 05Cost, latency, and review rate

    Does an additional parsing step reduce enough manual review to justify its operating cost?

Companion lab · 01

Run the worked comparison.

The guided flipbook takes you from the Snowflake workspace through both extraction paths, SQL evaluation, failure diagnosis, and a six-part architecture decision memo. Progress is earned with evidence, not page views.

Open the lab companion →
Sources and claim boundaries

This essay is based on a controlled AAPL 2023 annual-report lab from Zach Wilson’s DataExpert Snowflake course and the inspected course repository at commit c03992d. Its 40 local tests passed. The example is not a benchmark of all PDF extraction workloads.

Snowflake documents AI_EXTRACT as structured extraction from text and files, and AI_PARSE_DOCUMENT as OCR, layout, and agentic document parsing with page filtering and page-level output. Feature availability, limits, and billing can change; use the current documentation for production decisions.

SEC Company Facts supplies an independent structured reference for the worked financial metrics. A matching accession and period still require review; the evaluator reports missing or conflicting references instead of choosing whichever value flatters the extraction.

Return to the beginning ↑