I’ve seen this movie before

Commercial legal LLMs are trained on statutes, case law, and legal documents (contracts, filings, briefs), all of which have been proofread and edited by experts. This creates a high-quality, highly consistent training set. Nothing like knowing you can be sued or disbarred for a single mistake to sharpen your focus! This training set has enabled impressive accuracy and major productivity gains. In many firms, they’re already displacing much of the work junior lawyers once did.

Code-generating LLMs, by contrast, are trained on hundreds of millions of lines of public code, much of it outdated, mediocre, or outright wrong. Their output quality reflects this. When such models are trained on consistently high-quality code, something now possible as mechanically generated and verified codebases grow, their performance could rise dramatically, probably rivaling the accuracy and productivity of today’s best legal LLMs. “Garbage in, garbage out” has been the training rule. Soon, it will be “Good in, good out.”

I’ve seen this before. When compilers began replacing assembler for enterprise applications, the early generated code was slow and ugly. We hard-core bare metal types sneered. But compilers improved, hardware got faster and cheaper, and in a shockingly short time, assembler became a niche skill. Don’t dismiss new tools just because v1 is crude; v3 will eat your lunch just as compilers, back in the day, ate mine.