How to evaluate legal AI for a plaintiff firm

Every legal AI vendor will show you a demo that works. The demo is the least informative hour of the buying process: it runs on the vendor's best example, chosen in advance. What tells you whether a product will hold up on your matters is how the vendor answers a short list of plain questions, in writing, before you sign anything. Here are ten, with what a good answer sounds like. Ask them of us too.

1. Can every figure be traced to the page it came from?

A chronology or a demand is only as good as your ability to check it. A good answer: every entry links back to the source document, and a reviewer can open that page in one step. A weak answer talks about accuracy percentages instead. A percentage is an average; your case is one matter, and the error that matters is the one in it.

2. What happens when the system is unsure?

Every extraction system meets a smudged fax, a handwritten note or a bill in an unfamiliar format. The question is whether it guesses or asks. A good answer: uncertain fields are flagged for a person to confirm, and the confirmation is recorded. If the vendor says the system is never unsure, keep looking.

3. Is our data used to train models other firms use?

Your files contain medical records, privileged communications and settlement figures. A good answer is a plain no, in the contract as well as on the website. Ask where the data is processed, which outside model providers see it, and on what terms.

4. Which security attestation do you hold, and can we read it?

"SOC 2 compliant" is not an answer. Ask which report — Type I examines the design of the controls at a point in time, Type II examines how they operated over a period — when it was issued, and whether you can read it under NDA. A vendor with nothing to hide will point you to a trust center.

5. Will you sign a business associate agreement?

If the product will touch medical records, HIPAA applies, and the vendor should sign a BAA without a negotiation. Hesitation here tells you how the rest of the relationship will go.

6. How is data encrypted, and who can reach it?

A good answer names encryption in transit and at rest, the strength of the encryption at rest, multi-factor sign-in, role-based access and an audit log of who opened what. Be wary of any promise that nobody but you can ever read your files from a product that has to read them to work: a server that runs OCR on a document can, by definition, read it.

7. How many places will my staff still re-key the client?

Count them. A tool that saves an hour on chronologies but adds a new place to type the client's name, the policy number and the date of loss has moved the work, not removed it. Ask to see how a new intake reaches the chronology, the demand and the settlement statement without being typed again.

8. Can we see the price before we talk to sales?

Ask for the whole bill at your volume: seats, per-matter or per-page charges, what is included and what is extra. Ask which parts are recoverable as case expenses and which are firm overhead. A price that only exists after three calls is a price designed to be negotiated upward.

9. How long is the contract, and how do we leave?

Ask for the term, the notice period and, above all, the export: what format your matters, documents and work product come out in, how long it takes, and what it costs. The time to find out is before your data is inside.

10. Who do we call in the middle of a case?

Software fails at inconvenient moments — the night before a filing, the week of a mediation. Ask who answers, how fast, and whether that person can see your account. Then ask a firm the vendor did not choose for you how that went.

How to use the answers

Send the same ten questions to every vendor on your list and ask for written replies. Lay them side by side. The product with the best demo is rarely the one with the best answers, and the answers are what you will be living with.