AI can draft a technical memo in minutes. But how do you know if it got the analysis right? A framework for reviewing AI output.
Q1 close is a good time to talk about this.
A lot of teams are under pressure right now — deadlines stacking up, complex transactions that need to be documented, auditors asking for support. AI tools that can draft a technical memo in minutes are genuinely appealing in that environment.
And they should be. A good AI-generated draft can save real time. But "saves time" and "can be relied on without review" are not the same thing.
The most useful thing you can know about AI-generated memos is not how to use them — it's how to evaluate them. How to look at a draft and quickly determine what's solid, what needs work, and what's wrong.
This is a skill that's worth developing now, because AI is only going to become more embedded in this workflow. The accountants who learn to review AI output rigorously will use it effectively. The ones who don't will eventually sign off on something they shouldn't have.
Here's the framework I use.
Step 1: Check the question before you check the answer
This sounds obvious, but it's the most commonly skipped step.
Before you evaluate whether an AI memo got the analysis right, make sure it answered the right question. AI will give you a confident, well-organized answer to whatever question it thinks you asked — and that isn't always the question you actually had.
Look at the memo's statement of the issue. Is it accurate? Is it specific to your fact pattern? Does it capture the complexity, or has it simplified something that shouldn't be simplified?
If the question is wrong, the analysis doesn't matter. Start over with a more precise prompt before investing time in reviewing the rest.
Step 2: Verify every codification citation
This is non-negotiable.
AI tools — especially general-purpose ones — will sometimes hallucinate codification references. They'll cite a paragraph number that doesn't exist, or cite a real paragraph that doesn't say what the memo claims it says. This happens because the model is pattern-matching on plausible-sounding text, not actually reading the codification.
For every ASC reference in the memo, open the actual codification and verify four things:
- The citation exists. ASC 606-10-25-1 should actually be a real paragraph.
- It says what the memo says it says. Read the actual language, not just the reference number.
- It's the most relevant citation. Sometimes a model cites a real, accurate paragraph that's adjacent to the most important guidance — close, but not quite right.
- It hasn't been superseded. This one is easy to miss. AI models have a training data cutoff, which means they can confidently cite guidance that was accurate at some point but has since been amended or replaced. The paragraph exists, it says roughly what the memo claims — but it's no longer the operative guidance. Always confirm you're looking at the current version of the standard, not a prior iteration of it.
This step takes time. But it's the step that separates using AI as a starting point versus using it as a substitute for research. The citations are the foundation. If they're wrong — or out of date — everything built on them is wrong.
Step 3: Evaluate the application to your specific facts
This is where most AI memos get soft, and where your judgment matters most.
The guidance section of a memo — what the standard says — is usually the strongest part of an AI-generated draft. The application section — how the standard applies to your specific transaction — is usually the weakest.
Ask yourself:
Did the memo actually use your facts? A lot of AI-generated application sections are generic. They describe how the standard applies to a hypothetical similar transaction rather than the actual one. Look for specific references to your terms, your structure, your numbers.
Did it identify the judgment calls? Technical accounting rarely has clean, algorithmic answers. If the memo presents a definitive conclusion without acknowledging any complexity, that's a flag. Real memos acknowledge where judgment was required and explain why you landed where you did.
Did it miss any relevant facts? You provided the information, but did the AI use all of it? Think about whether there's anything in your transaction that the memo didn't address. Missing facts can lead to missing analysis.
Step 4: Check the conclusion for internal consistency
Read the memo's conclusion against the rest of the memo. Does it actually follow from the analysis?
This sounds like it should always be true, but it isn't. AI tools can sometimes produce memos where the guidance section points one direction, the application section introduces some nuance, and the conclusion ignores that nuance and lands somewhere that doesn't quite follow.
A few things to look for:
- Does the conclusion address all the considerations raised in the analysis, or does it quietly drop some?
- If the analysis flagged uncertainty or judgment calls, does the conclusion acknowledge them?
- If you changed one of the key facts, would the conclusion change? If not, the analysis might not be as fact-specific as it looks.
Step 5: Apply the "auditor question" test
Before you finalize any memo — AI-generated or not — ask yourself: what's the first question an auditor would ask about this?
Then look for the answer in the memo.
If you can predict a reasonable follow-up question and the memo doesn't address it, that's a gap you need to fill before the memo is done. Auditors will ask about the things that aren't there just as much as the things that are.
This is also a useful test for identifying where the AI's analysis is thin. The places where a good auditor would push back are usually the places where the reasoning is least developed.
What to fix vs. what to flag
Not everything that needs work in an AI memo is a fix you make yourself. Some things are flags — issues that need to be escalated or addressed before the memo moves forward.
Fix it yourself if:
- A citation is accurate but could be more specific
- The application section is correct but too generic — needs more fact-specific language
- The structure needs reorganization
- The tone or level of detail needs adjustment
Flag it (don't just fix it) if:
- A citation is wrong or doesn't exist
- The conclusion doesn't align with the guidance
- A key fact wasn't addressed and you're not sure how it affects the analysis
- You're not sure the right standard was applied
- The memo reaches a conclusion on a genuinely ambiguous question without acknowledging the ambiguity
The distinction matters because "fixing" something you're not sure about is how you introduce errors. When in doubt, slow down.
The right way to think about AI memos
AI-generated memos are drafts. They're starting points. A well-generated draft from a purpose-built tool can save you significant time and give you a better starting point than a blank page.
But the memo that goes in the file — the one the auditor reviews, the one with your name on it — has to reflect your analysis. Not the AI's.
That means reviewing the draft rigorously, filling in the gaps, correcting what's wrong, and making sure the conclusion is yours. The AI got you there faster. The quality of the final product is still on you.
Q1 close is a bad time to learn this lesson the hard way.
Gaapio is designed to make this review process easier — the analysis is structured for documentation from the start, citations link directly to the codification, and the output is built to be edited rather than just copied. If you're working through technical accounting questions this close season, it's worth trying.