🔍 Read the full analysis: A 15-Question Check Of 42 AI Setups Included A Law-Focused Model on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, evaluated 42 model setups on 15 questions spanning German, U.S. and EU law. The law-focused GLM-5.3 FlashX Legal scored 8 out of 100, but the organizer says the test is a limited spot check, and its grading process has a disclosed conflict of interest.
Lexassay, a legal AI test run by Thorsten Meyer AI, reports that the law-focused GLM-5.3 FlashX Legal scored 8 out of 100 on a set of 15 legal questions given to 42 AI setups. The result raises questions about whether a model marketed for legal workflows can reliably handle tasks such as deadlines and citations, but the test’s publisher says its results are a limited spot check, not a definitive ranking.
The test included 10 publicly available questions and five hidden questions covering German, U.S. and EU law. Models answered without internet access or tools; separate effort settings were tested as separate setups. The questions covered legal deadline calculations, changes in law, court decisions and prompts built on false assumptions.
According to the published results, GLM-5.3 FlashX Legal received 0 points on the hidden set. Its citation review identified 21 citations as real, 25 as misattributed and one as invented. Meyer says the model gave no answer on two German-law tasks after using its full 64,000-token thinking budget. Its base model scored 3 out of 100 in the same test.
The test’s top reported scores were 79 for Claude Opus 5.5, 72 for GPT-6 Astra and 71 for GPT-6.1 Sol. Even the top model met every scoring criterion on only eight of the 10 public tasks, according to the report. Across published runs, Lexassay said it verified 2,644 citations as real and found 804 that were invented or did not support the claims attached to them.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal AI Scores and Citation Risks
The results speak to a practical risk for people using chatbots to understand legal problems: a fluent answer may still get a deadline, legal rule or supporting citation wrong. In civil litigation, missing an appeal deadline can lead to a case being rejected as inadmissible. The report’s example describes two German clients whose judgments were served on the same date but whose appeal deadlines differed because Corpus Christi was a public holiday in Bavaria and not in Berlin.
The findings do not establish how often AI systems make these mistakes in ordinary legal work. They do show why a model’s legal branding or confident tone should not be treated as proof of accuracy. Meyer advises users to treat AI as a starting point, check citations against original sources and consult a lawyer when a deadline, money or an authority is involved.
There is also a broader question about how to compare legal AI products. Lexassay tests models without tools, while the report says Vals AI’s legal-agent benchmark supplies documents. In that separate benchmark, Mistral Large 4 ranks ahead of Claude Opus 5.5 and GPT-6 Astra, according to Meyer. Different tasks and conditions can produce different orderings, so the rankings should not be read as interchangeable measures of legal ability.
Products Worth Considering
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
Logitech MX Brio Ultra HD 4K Webcam, Graphite
Jackery Explorer 300 Portable Power Station, 292Wh
As an affiliate, we earn on qualifying purchases.
How the Fifteen Questions Were Scored
Lexassay says each answer was graded against a reference answer using three to six weighted criteria, and citations were checked against official databases. A citation could be classed as real, misattributed, invented or not findable. Each invented or misattributed citation reduced a task’s score by a quarter, while an answer without a legal-advice disclaimer or referral to a lawyer lost 5 percent.
The test combines questions from German, U.S. and EU law, including recent legal changes and leading court rulings. The five hidden questions are intended to make it harder for later models to reproduce answers found online; only their scores are published. The setup measures what a model can answer on its own, rather than what it can do with web search, legal databases or other tools.
Meyer’s article gives a deadline question as an example of why legal details matter. In the scenario, the appeal period ends on different dates in Munich and Berlin because of the regional holiday calendar. The report says most strong models answered correctly, but one put the Berlin date a day late and another placed the deadline for written appeal grounds a month too late. These are examples from the test, not evidence of the models’ performance across all legal questions.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, in developer documentation quoted by Thorsten Meyer
Limits and Conflicts in the Results
The test is small by design: it uses 15 questions and one answer per model setup and task. Meyer says small score gaps may be noise. Because models were tested without tools, the results do not show how they perform when connected to legal databases, search systems or document collections—the conditions used in some legal AI products and benchmarks.
The publisher also disclosed that Thorsten Meyer AI runs Lexassay and that Claude agents graded the answers, while a Claude model placed first. The disclosed conflict does not by itself invalidate the results, but readers cannot treat the test as an independent evaluation. The article says public answers, reference answers and citation checks are available for review; the hidden questions are not fully published.
The supplied report does not include independent replication, detailed results for all 42 setups, or enough information to establish how representative these questions are of real legal work. It also does not show whether the models’ performance changes with different prompts, newer versions or access to tools. The reported scores should not be generalized to every legal task or user.
Checking Answers Against Legal Sources
Meyer says the test’s method page and published answer materials allow readers to inspect the scoring and citation checks. The report does not announce a date for another test or a planned response from Z.ai, Mistral or the other companies. Further releases or independent replications would help show whether the reported gaps persist across additional questions and testing conditions.
For readers using AI now, the report recommends treating outputs as an initial aid rather than legal advice. Check every cited statute and ruling in an authoritative source, do not rely on a chatbot to calculate a legal deadline, and speak with a lawyer when a case, payment or communication with an authority is at stake.
Key Questions
What did the law-focused AI model score?
GLM-5.3 FlashX Legal scored 8 out of 100 on the Lexassay test and received 0 on the five hidden questions, according to the report.
How many AI setups and legal questions were tested?
Lexassay reports testing 42 AI setups on 15 questions: 10 public questions and five hidden questions covering German, U.S. and EU law.
Does the test prove the model is unsuitable for legal work?
No. The report describes a limited spot check with one answer per setup and task, performed without internet access or tools. It does not establish performance across all legal work or tool-assisted workflows.
Why did the publisher disclose a conflict of interest?
Thorsten Meyer AI runs Lexassay, and the answers were graded by Claude agents while a Claude model ranked first. The publisher disclosed this as a conflict of interest; it is a factor readers should consider when evaluating the results.
Can I use a chatbot to calculate a legal deadline?
The report advises against relying on a chatbot for deadlines. Legal deadlines can depend on the jurisdiction and local holidays, and a missed deadline can have serious consequences. Verify the date with an authoritative source or a lawyer.
Source: ThorstenMeyerAI.com
