🔍 Read the full analysis: The AI Model Built For Law Faced 15 Questions And Scored 8/100 on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A test published by ThorstenMeyerAI.com says GLM-5.3 FlashX Legal scored 8 out of 100 across 15 legal questions and fully solved none. The test operator also says the model produced no answer to two tasks and had substantial citation problems; the results come from a limited, disclosed test and are not an independent benchmark.
GLM-5.3 FlashX Legal, a model Z.ai describes as optimized for professional legal workflows, scored 8 out of 100 on a 15-question test run by ThorstenMeyerAI.com, according to results published November 4. The test operator says the model fully solved none of the tasks, gave no answer to two, and scored zero on the hidden questions, raising questions about how well a legal-focused chatbot handles legal research without internet access or tools.
The assessment, called Lexassay, covered questions involving German, U.S. and European Union law. Its tasks included calculating legal deadlines, identifying recent changes in law, applying leading court decisions and spotting questions built on a false premise. Ten questions are public and five are hidden; the operator publishes only the hidden-set score.
Thorsten Meyer, who runs the site and Lexassay, reported that GLM-5.3 FlashX Legal received 21 citations marked real, alongside 25 citations classified as misattributed and one as invented. Under the test’s scoring rules, a citation that exists but does not support the model’s claim is misattributed; invented citations do not exist. Each misattributed or invented citation costs a quarter of a task’s points, the method page says.
The results article gives examples of errors. In one answer about compensation following data scraping, Meyer says the model described a four-figure sum as conceivable, whereas the test’s account of German Federal Court of Justice practice puts compensation for the mere loss of control over personal data at around €100. In another, the model reportedly described a challenge to the EU–US Data Privacy Framework as pending, although the EU General Court had dismissed it on September 3, 2025.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Errors Can Change Legal Decisions
The test focuses on mistakes that could matter to people handling a court deadline, compensation claim or legal citation. An incorrect deadline may affect whether an appeal is accepted, while a misleading account of damages could distort a person’s expectations. A citation that looks authoritative but does not support the stated conclusion can also make an answer difficult for a non-lawyer to verify.
Lexassay’s opening example illustrates the point: it describes two German civil cases served on the same day, one in Munich and one in Berlin. The appeal period would ordinarily run for a month, but the test says the Munich deadline shifts because Corpus Christi is a public holiday in Bavaria, while the Berlin deadline does not. The example is part of the test material, not evidence that a chatbot should be relied on to calculate an actual case deadline.
The results are a warning about using a model’s legal branding as a substitute for checking its work. They do not establish how the model performs with a lawyer reviewing its answers, with access to current legal databases, or on the full range of professional tasks.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky
Dell XPS 13 9350 Copilot+ Laptop, 13.4-inch 120Hz Display, Intel Core Ultra 7 256V, 16GB RAM, 512GB SSD, Ubuntu Linux 24.04 LTS
Apple iMac 27-inch Retina 5K Display (2020, 8GB RAM, 512GB SSD)
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay says models receive the same 15 questions without internet access or external tools. The test therefore measures answers generated from the model itself, rather than a system that can retrieve current statutes or court documents. Questions are graded against a “gold answer” using three to six weighted criteria, and citations are checked against official databases.
The operator says the test is a spot check, not a benchmark in the strict sense: each model and task receives one answer, so small score differences may reflect variation in a single run. Different “effort” settings, which control how long a model may reason before answering, are tested separately. Meyer also says an answer loses 5% if it lacks a legal-advice disclaimer or a suggestion to consult a lawyer.
The test’s published leaderboard places Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer says even the leading model fully met every criterion on only eight of the ten public tasks. Mistral Large 4 scored 10 in Lexassay, according to the article, though a separate Vals AI legal-agent benchmark reportedly ranked it eighth, ahead of several models that scored higher on Lexassay. The different results reflect different test designs and should not be treated as directly comparable.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation by ThorstenMeyerAI.com
Limits of the Published Score
The findings come from a test run and published by Thorsten Meyer, who operates Lexassay. Meyer discloses that several Claude agents graded the answers and that a Claude model ranks first, calling this a conflict of interest. He says the test method and public answers, gold answers and citation checks are available for review. The provided results do not include an independent replication.
The test also has a narrow setup: one answer per model per task, no access to the internet or tools, and a fixed set of 15 questions. Its score does not show how the model would perform with legal research tools, a human reviewer, a different prompt or a larger set of cases. The source says GLM-5.3 FlashX Legal scored zero on the hidden set but does not provide the individual hidden answers, which are not publicly released.
The supplied article cuts off while discussing how the model ranked on another test, so its full account of that comparison is unavailable here. The reported scores should be read as results from this particular test, not as a general measure of legal competence or proof about every version and use of the model.
Check Citations Before Acting
For readers, the practical next step is to treat chatbot legal answers as starting points, not verified advice. Meyer recommends checking every cited law and ruling in the original source, not relying on a chatbot for a deadline, and consulting a lawyer when a matter involves money, a deadline or communication with an authority.
The source does not identify a response from Z.ai or say whether the company plans to address the reported score. A follow-up from the company, independent reruns of the tasks, and evaluations that include current legal research tools could help clarify how the result applies beyond Lexassay’s no-tools test.
Lexassay says it publishes the public questions, answers, grading criteria and citation checks so readers can inspect the scoring. The operator says hidden-task scores are published without revealing those questions. Whether future model versions improve on the reported performance remains to be seen.
Key Questions
Which legal AI model scored 8 out of 100?
GLM-5.3 FlashX Legal, made by Z.ai, received the score in Lexassay’s 15-question test, according to ThorstenMeyerAI.com.
What did the test cover?
The questions covered German, U.S. and EU law, including legal deadline calculations, recent legal changes, court decisions and questions containing false assumptions.
Was the score independently verified?
The supplied source describes results from a test run by Thorsten Meyer, who operates Lexassay. He disclosed that Claude agents graded the answers and that a Claude model ranked first. The material does not report an independent replication.
Does this mean the model always performs poorly on legal work?
No. The result reflects a 15-question test without internet access or tools, with one answer per task. It does not establish performance across other prompts, larger datasets, tool-enabled workflows or human-reviewed work.
Should someone use a chatbot to calculate a legal deadline?
The test operator advises readers to check legal citations against original sources and never take a deadline from a chatbot. For a real deadline or legal matter, consult a lawyer.
Source: ThorstenMeyerAI.com
