TL;DR
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai says GLM-5.3-FlashX-Legal achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s leaderboard lists it seventh of 81 models, while the benchmark’s strict all-or-nothing scoring means the result does not show that the other tasks were wholly unsuccessful.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark, placing it seventh among 81 models on the independent Vals AI leaderboard. The result means the model met every required criterion on about one in five tasks; it does not mean it earned no credit on the rest.
The score is an all-or-nothing measure. Harvey’s benchmark marks a task complete only when every item in its expert-written rubric passes. A missed citation, deadline or other required detail can therefore fail the entire task, even if the rest of the work product meets the rubric. In the same run, GLM Legal met 92.5% of individual criteria, according to the source material.
Vals AI’s archived leaderboard records the model at 19.17%, rounded to 19.2%, and seventh of 81 models, at a listed test cost of $0.15. Z.ai says the result improved by 12.5 percentage points over the base model. That comparison is consistent with the base-model result on the same leaderboard, according to the source material.
Z.ai describes GLM-5.3-FlashX-Legal as a model for legal document understanding, reasoning and task execution, including research, contract review and drafting initial drafts. The company says it underwent legal-domain training and recommends using it as an agent connected to services for laws, cases and company data. Z.ai lists API pricing at $0.37 per million input tokens and $1.25 per million output tokens.
Legal AI · Benchmark briefing
Why a 1-in-5 Pass Rate Still Puts Z.ai’s Legal AI Near the Top
GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s leaderboard. Its 19.2% score reflects a strict all-criteria threshold, while the same run met 92.5% of individual rubric checks.
One task. Two different ways to count.
A strict task pass rate measures complete work products. Criterion-level scoring measures how often individual requirements were met.
Same run, different lenses
The all-or-nothing benchmark can turn one missed citation, deadline, or required detail into a failed task—even when much of the work meets the rubric.
These figures describe different units: whole tasks versus individual rubric checks.
“A task is marked complete only if every criterion passes.”Harvey · Benchmark scoring method
From matter materials to a graded work product
Harvey’s open benchmark tests legal-agent tasks using instructions, client-matter materials, and expert-written grading rubrics.
Task assigned
An instruction asks the agent to complete legal work for a matter.
Materials reviewed
The agent works from client-matter materials; leaderboard internet access is disabled.
Criteria checked
Binary rubric items cover facts, conclusions, citations, deadlines, amounts, and format.
Task scored
Two language-model judges grade submissions; the final score averages task pass rates.
Harvey released the benchmark on May 6, 2026. Vals AI reports a 120-task private holdout mirroring the public mix.
A leaderboard signal, with practical limits
Seventh place is a useful comparison under this protocol. It does not answer whether a draft is ready for a live matter or how it performs in every workflow.
Built for legal tasks
Z.ai describes document understanding, legal reasoning, and task execution, including research, contract review, and initial drafting.
Connect trusted sources
Z.ai recommends connecting services for laws, cases, and company data. API pricing is listed at $0.37 per million input tokens and $1.25 per million output tokens.
Verify citations and facts
A four-task spot check reported rulings the reviewer could not find. That small check cannot estimate a general error rate, but it reinforces citation review.
What the reported result does—and does not—establish
Dates and claims below reflect the supplied source material and the archived leaderboard details.
| Evidence | What it supports | What remains open |
|---|---|---|
| Vals AI listing | 19.17% (rounded to 19.2%), seventh of 81; listed October 12, 2026. | Performance beyond this benchmark and its holdout. |
| Z.ai announcement | Beta by application announced October 13, 2026; Z.ai reports a 12.5-point gain over the base model. | No broader release date is provided. “State-of-the-art” comparison set is unspecified. |
| Model availability | Z.ai positions the model for professional legal workflows. | Source material reports no published weights for the legal model. |
| Buyer diligence | Benchmark gives comparative evidence under a stated protocol. | Client-data processing, privacy, security, integration, and review responsibilities need procurement review. |
Four checks before relying on a legal AI draft
Use the benchmark as one input to evaluation. Operational readiness depends on workflow-specific evidence and controls.
Which requirements fail?
The supplied material has no detailed breakdown of GLM Legal’s failed tasks, so misses by citation, analysis, formatting, or other criteria remain unclear.
Can lawyers use drafts without checking?
The benchmark does not establish that. Review authorities, matter-specific facts, deadlines, and other required details before use.
Is the legal model open source?
Z.ai makes an open-source performance claim, but the supplied material reports no published weights and names no comparison set.
What does the spot check show?
It flags citation verification as a practical concern. Four tasks are too few to estimate overall reliability or compare systems broadly.
From benchmark result to deployment decision
A strong relative rank is a starting point for a controlled, matter-specific assessment.
Compare
Read the rank alongside both task and criterion-level scores.
Trial
Evaluate representative work in the intended legal workflow.
Control
Set data, source integration, and human review requirements.
Reassess
Track errors and review new independent results as they appear.
Why the Ranking Needs Careful Reading
The leaderboard position gives buyers a comparative signal: under the same benchmark protocol, the model ranked near the top of the 81 submissions listed. But an all-criteria pass rate is a demanding threshold, and the 19.2% score alone does not describe partial performance. The 92.5% individual-criteria figure offers another view of the same run, though it also does not establish that a lawyer could use the resulting documents without review.
For legal teams, the practical issue is reliability in specific workflows. A task that misses one required citation or deadline may still need substantial correction before it can support a matter. Firms considering the model would need to check its work, especially authorities and matter-specific facts, and establish how client data is handled. The benchmark result does not answer those procurement and oversight questions.
The source material also describes a four-task spot check in which the legal model cited rulings the reviewer could not find. That small check is not a benchmark and cannot establish a general error rate, but it underscores why citation verification matters. The same check found different citation problems in other models, including Claude models that cited a real ruling for the opposite of its holding.
Products Worth Considering
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, Space Black
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. The assignments combine an instruction with client-matter materials and ask an AI agent to produce a work product for review. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, deadlines, dollar amounts and formatting.
Harvey says each criterion is binary and that a task passes only when every criterion passes. The public leaderboard uses a private holdout set that mirrors the public mix. Vals AI says that holdout contains 120 tasks, with internet access disabled. Two language-model judges grade each submission, and the final score averages their task pass rates.
Z.ai’s AutoClaw blog announcement, dated October 13, 2026, opened a beta by application and said the model was built on GLM-5.3-Flash. Vals lists the release on October 12. Z.ai calls its result state-of-the-art among open-source models in the evaluation, but the source material does not identify the comparison set, and it reports no published weights for the legal model.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai, in its developer documentation
Questions Beyond the Leaderboard
The leaderboard establishes a result under its stated benchmark protocol, but it does not show how the model performs across every legal practice, jurisdiction or live client workflow. The source material gives no detailed error breakdown for GLM Legal’s failed tasks, so it is unclear how often misses involved citations, analysis, formatting or other criteria.
Z.ai’s claim of state-of-the-art performance among open-source models is also difficult to assess from the information provided: the company names no comparison set, and the source material reports no published weights for the legal model. The four-task spot check is too small to estimate overall reliability. The source material says the Z.ai pages reviewed describe drafts for legal professionals but carry no “not legal advice” notice; it does not establish how the product is presented in every setting.
Beta Access and Buyer Checks
Z.ai’s October 13 announcement opened a beta by application. The source material does not give a broader release date or say when the beta will expand. Vals AI’s archived leaderboard provides a current point of comparison, while further results or a larger independent evaluation could clarify how the model performs across legal tasks.
Before adopting the system, law firms and other buyers will need answers on client-data processing, applicable privacy and security rules, integration with trusted legal sources, and who checks citations and other requirements before a draft is used. The benchmark score provides comparative evidence, but those operational details remain matters for procurement and deployment review.
Key Questions
What does the 19.2% pass rate mean?
It means GLM-5.3-FlashX-Legal met every required rubric criterion on about 19.2% of tasks in the evaluation. Under Harvey’s all-pass rule, one missed criterion fails the whole task.
Why does the model rank seventh if it passed only about one in five tasks?
The rank compares its all-criteria pass rate with the other models on Vals AI’s leaderboard. Seventh place among 81 reflects that comparison; it does not mean the model passed most tasks.
What was the model’s score on individual criteria?
The source material reports that the same run met 92.5% of individual criteria. That figure counts rubric checks rather than whole tasks, so it answers a different question from the all-or-nothing task pass rate.
Is GLM-5.3-FlashX-Legal open source?
Z.ai claims state-of-the-art performance among open-source models in this evaluation. The source material says it found no published weights for the legal model and does not specify the company’s comparison set, so the claim cannot be independently assessed from the details provided.
Can lawyers rely on its legal drafts without checking them?
The benchmark does not establish that. The source material describes drafts for legal professionals and reports a four-task spot check in which the model cited rulings the reviewer could not find. That small check is not a measure of general performance; buyers should assess citations and other required details in their own review process.
Source: Thorsten Meyer AI
