CONSOLIDATED STATEMENTS OF OPERATIONS
In millions, except number of shares, which are reflected in thousands, and per-share amounts
- Net sales
- 2023
- 2022
- Total net sales
- 383,285
- 394,328
Document AI in Snowflake
Finding the right PDF is no longer the whole problem. The hard part is turning one plausible model answer into a fact a company can actually trust—across the layouts, periods, units, scale notes, and reference gaps hiding in the long tail.
One annual report. Two Snowflake paths.
One independent reference.
Start with a decision: would you let one plausible row into production?
Then inspect the Snowflake architecture and the evidence contract behind it.
60-SECOND PDF TRUST TEST
In millions, except number of shares, which are reflected in thousands, and per-share amounts
MODEL OUTPUT
Can this row enter a production table?
01The demo can pass while the data fails
Apple’s 2023 income statement prints net sales as 383,285. An extractor can return that exact number, label it USD, produce valid JSON, and still be wrong by six orders of magnitude. The scale note—“in millions”—sits above the table. The trustworthy value is 383,285,000,000 USD.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
This is the part of document AI that a clean demo tends to hide. Retrieval can find the correct annual report while extraction chooses the comparative-year column, loses the scale, reads a subtotal, or cites text that does not support the number. The file is right. The row is not.
That distinction matters because companies do not want PDFs merely to become searchable. They want the facts inside financial statements, claims, filings, manuals, and operational records to enter systems that make decisions. “The model returned an answer” is the beginning of that job, not the acceptance test.
ContinueThe PDF long tail↘02Document variation
PDFs share a file extension, not a data model. One has a clean text layer; another is a scan. A table may use borders, whitespace, rotated headers, nested columns, or an image. The current and comparative periods may sit inches apart from a scale note that governs both.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
The long tail is the accumulation of these ordinary differences: shifted page indexes, negatives in parentheses, fiscal years that do not match calendar years, and reference systems with two facts for what looks like one metric. None is rare enough to ignore once a workflow touches thousands of documents.
The seasoned response is not a heroic prompt. It is a data contract around the prompt: preserve source identity, select pages explicitly, record the method and run, separate value from scale and period, save evidence, and report when no independent reference exists.
FILING AND REFERENCE ALIGNMENT
03Method comparison
The shortest path sends selected pages directly to AI_EXTRACT. The more inspectable path runs AI_PARSE_DOCUMENT in layout mode, stores page-level Markdown, and then sends that text to AI_EXTRACT. Both can target the same response schema; only one leaves behind an intermediate representation you can examine.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
Direct extraction avoids an extra call and may preserve information that a text representation loses. Parse-then-extract creates more surface area, but when a table or reading order fails, the Markdown can show whether the error entered during parsing or extraction.
A useful comparison holds the file, pages, questions, schema, and reference facts still and changes only the path. Use a fresh run ID for every experiment. The result will not crown a universal winner. It will tell you whether inspectability earns its cost for this document family.
CONTROLLED COMPARISON
PDF pages → AI_EXTRACTFewer calls. Less intermediate state.
PDF pages → AI_PARSE_DOCUMENT → Markdown → AI_EXTRACTMore inspectable. More surface area.
04Model the answer
A financial metric is not one value field. In the worked example, the answer includes the printed number, scale multiplier, unit, fiscal start, fiscal end, and supporting evidence. The evaluator normalizes the number, checks the semantic fields, and only then applies numeric tolerance.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
383285Signed numeric text1000000Multiplier declared by the statementUSDExact catalog unit2022-09-25Required for duration metrics2023-09-30Matches the approved filingNet sales … 383,285Statement line, header, and scale note383285 × 1000000 → 383,285,000,000 USD · FY 2023 · evidence present
That contract turns “the model was wrong” into diagnoses an engineer can act on. A period mismatch calls for a different fix than a missing quote. A parsing failure calls for a different fix than an ambiguous SEC reference. Agreement between two model paths is not a substitute for independent evidence.
This looks less glamorous than model selection because it is data modeling. It is also the layer that lets a reviewer trace a production row back to the filing and lets a team improve one failure class without destabilizing the rest of the pipeline.
RESPONSE FORMAT
383285Signed numeric text1000000Multiplier declared by the statementUSDExact catalog unit2022-09-25Required for duration metrics2023-09-30Matches the approved filingNet sales … 383,285Statement line, header, and scale note05Ground truth and failure classes
Lowering temperature may reduce sampling variation. It cannot restore a missing scale note, select the right filing, choose the current period, or manufacture ground truth. A deterministic pipeline can repeat the same mistake with admirable consistency.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
Repeated result: The run repeated the same value without the ×1,000,000 scale.
Engineering response: Extract scale separately and normalize before comparison.
Evaluation therefore needs information the model did not generate. For public-company metrics, SEC Company Facts can provide an independent source, but the match still depends on filing accession, concept, unit, and fiscal period. If no usable reference exists, the result is unscorable—not a convenient pass and not a model failure.
Save the method, run ID, raw response, normalized fields, reference value, tolerance, evidence, and failure class. That record separates three things teams often collapse into one accuracy number: extraction quality, document coverage, and reference-data quality.
ContinueThe senior decision↘06What leadership actually requires
A senior recommendation sounds like this: for this document family, under this evaluation set, choose the method with the stronger measured result; pay for the parse step when its inspectability lowers review cost or exposes failures the direct path hides. If the methods tie, report the tie. If cost was not measured, say so.
FL_FILTER_PAGES↓AI_EXTRACT(file)One inference path; less intermediate state.
Matched by filing identity and period—not by resemblance to the model answer.
Pass rate belongs only over scorable metrics. Report reference coverage separately, then track extraction coverage, failure mix, latency, inference cost, manual-review rate, and repeated-run stability. A high score on the easiest referenced documents can conceal the cases the business actually fears.
Snowflake is useful because the file reference, parsed pages, raw extraction, structured facts, and SQL evaluation can remain queryable together. But the larger opportunity is platform-agnostic: turn document families into measured data products with visible uncertainty and a review path. That is the engineering work in the long tail.
Companion lab · 01
The guided flipbook takes you from the Snowflake workspace through both extraction paths, SQL evaluation, failure diagnosis, and a six-part architecture decision memo. Progress is earned with evidence, not page views.
Open the lab companion →This essay is based on a controlled AAPL 2023 annual-report lab from Zach Wilson’s DataExpert Snowflake course and the inspected course repository at commit c03992d. Its 40 local tests passed. The example is not a benchmark of all PDF extraction workloads.
Snowflake documents AI_EXTRACT as structured extraction from text and files, and AI_PARSE_DOCUMENT as OCR, layout, and agentic document parsing with page filtering and page-level output. Feature availability, limits, and billing can change; use the current documentation for production decisions.
SEC Company Facts supplies an independent structured reference for the worked financial metrics. A matching accession and period still require review; the evaluator reports missing or conflicting references instead of choosing whichever value flatters the extraction.