🔍 Read the full analysis: How Z.ai’s Legal AI Measures Up When It Passes About 1 In 5 Tasks on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived version. Vals also reported a 92.5% pass rate across individual criteria; the figures measure different things, and the task score means at least one required element was missed on most tasks.
Z.ai’s GLM-5.3-FlashX-Legal passed every required criterion on 19.17% of tasks in the archived Vals AI leaderboard for Harvey’s Legal Agent Benchmark, placing seventh among 81 models. The score is an all-or-nothing task measure: the model could satisfy most elements of a task and still not pass if it missed even one required item.
Vals AI’s archived leaderboard lists the legal model at $0.15 per test and reports a 92.5% pass rate on individual criteria. That higher figure does not mean the model completed 92.5% of tasks. It reflects performance on individual rubric items, while the 19.17% task score requires all criteria for a task to pass.
The leaderboard places GLM Legal seventh, behind six models, including Muse Spark 1.2 at 25.42% and Gemini 4 Argon at 19.58%. Z.ai’s documentation rounds its result to 19.2% and says it improved by 12.5 percentage points over the underlying GLM-5.3-Flash model. The Vals table lists the base model at 6.67%, which yields that 12.50-point difference.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its developer documentation says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of modeled task environments. The company recommends using it as an agent connected to services for laws, cases and company information.
What the Task Score Means
The result matters because legal work products are judged on completeness, not just whether most of the answer looks plausible. A missed citation, deadline, risk or monetary figure can make a draft unsuitable for use, even when the rest of the work is sound. Harvey’s benchmark makes that distinction explicit: a task passes only when every required rubric criterion passes.
That design explains how GLM Legal can record a 92.5% individual-criterion rate and still pass fewer than one in five full tasks. The task result is a warning against reading the partial-credit figure as a measure of finished legal work. It is also not a measure of how often the model’s output would be accepted by a particular law firm or used without revision.
The price listed by Vals may attract attention: $0.15 per test, compared with $10.55 for Gemini 4 Argon in the same archived table. But cost per benchmark test alone does not establish the cost of a reliable workflow. Human review, corrections, data handling, and connections to external legal sources can all affect procurement decisions. The leaderboard is evidence about one evaluation, not a substitute for firms’ own checks.
Products Worth Considering
LG PuriCare AeroHit Smart Air Purifier for Large Rooms up to 1000 sq ft, True HEPA Filter with 360° Airflow, PM1.0 Sensor & LED Air Quality
Apple iMac 27-inch Retina 5K Display (2020, 8GB RAM, 512GB SSD)
Apple iMac Pro (27-inch Retina 5K Display, 3.2GHz 8-core Intel Xeon W, 32GB RAM, 1TB SSD) – Space Gray
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. It includes 1,250 tasks across 24 areas of legal practice. Each task gives an agent an instruction and client matter materials, then asks it to produce a work product for review. Harvey says the evaluation uses more than 75,000 expert-written, binary criteria covering details such as facts, conclusions, citations, recommendations, deadlines and formatting.
The public leaderboard uses a private holdout set of 120 tasks, according to Vals AI, which operates the leaderboard using Harvey’s grading protocol. Internet access is disabled during the evaluation. Two language-model judges grade submissions, and Vals averages their task pass rates. Its archived leaderboard was updated October 18, 2026; Vals published a later HLAB-Vals-1.1 listing updated October 30. Z.ai’s AutoClaw blog announcement, dated October 13, said the model was built on GLM-5.3-Flash and opened a beta by application. Vals lists the release date as October 12.
Harvey’s earlier baseline offers a sense of the challenge but is not a direct comparison with the Vals run. In a May 26 baseline, Harvey reported a best overall all-pass rate of 7.1% among the frontier models it tested and said those models completed fewer than 10% of tasks end to end in aggregate. The runs and evaluation settings differ, so those figures should not be treated as a head-to-head result.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its grading rule
Limits of the Published Results
The benchmark does not establish how the model performs across all legal matters, jurisdictions or firm workflows. It reports results on a 120-task private holdout scored by two AI judges; the source material does not provide task-by-task results or a full account of variation between judges. The leaderboard score is a test result, not proof that the model can safely produce final legal work without professional review.
Z.ai calls the result state-of-the-art among open-source models in the evaluation, but the comparison set is not specified in the supplied materials, and the legal model’s weights were not identified as publicly available. A separate small spot check described by Thorsten Meyer found citations to rulings that could not be located in four tasks. That sample is too small to estimate a general error rate. The same check found that two Claude models cited real rulings for the opposite of what they held, underscoring that a limited spot check cannot rank systems reliably.
It is also not clear from the cited Z.ai pages how client data would be processed in a firm deployment, what contractual or technical controls would apply, or what review process Z.ai expects before generated work leaves a legal team. The company describes outputs as drafts for legal professionals; the available material does not establish a complete deployment or data-governance policy.
What Firms Should Check Next
The next useful evidence would include results from the updated leaderboard, fuller details of the model’s evaluation and access, and testing by legal teams on their own documents and workflows. Firms considering a trial will need to verify citations and substantive conclusions, examine data-processing terms, and establish who reviews every output before it is sent to clients, courts or counterparties.
Z.ai’s AutoClaw post said access to the beta was available by application, while its documentation recommends connecting the model to legal information and company-data services. The supplied sources do not state a general release date or describe a complete deployment schedule. Until those details emerge, the benchmark remains a useful but narrow measure: a relatively low rate of fully completed tasks alongside a much higher rate of passed individual criteria.
Key Questions
What does the 19.17% score mean?
It means the model passed every required criterion on 19.17% of benchmark tasks in the archived Vals AI leaderboard. A task with even one failed criterion does not count as a pass.
Why is the criterion pass rate 92.5%?
The 92.5% figure covers individual rubric criteria, not complete tasks. A model can pass many items within a task but fail the task-level score by missing one required item.
How did GLM Legal rank?
It ranked seventh among 81 models in Vals AI’s archived leaderboard updated October 18, 2026. The ranking applies to that version of the leaderboard, not necessarily later versions.
Does this show the model is ready to handle legal work independently?
No. The score is a benchmark result and does not show that the model can replace lawyer review or firm-specific testing. Z.ai describes its outputs as drafts for legal professionals.
What is still unknown about using it at a law firm?
The available material does not fully explain client-data processing, deployment controls or review requirements, and it gives no general release date. Firms would need those details before judging whether the model fits their systems and obligations.
Source: ThorstenMeyerAI.com
