🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment edited the output weights of a local Ornith model to reduce looping. On the 10 published Lexassay tasks, it answered 10 instead of 4 on the same setup, but its legal score stayed at zero; the results are preliminary and the comparison has important limitations.
A home experiment using a direct edit to a local AI model’s weights helped it finish more legal benchmark answers, but did not improve its legal score. On the same experimental setup, Ornith 1.5 35B-A3B answered 10 of 10 published Lexassay tasks after the edit, compared with 4 of 10 before; its score remained zero, showing that fewer loops did not translate into more accurate legal reasoning.
The experiment, conducted from Wednesday evening to Thursday morning and described by Thorsten Meyer, was inspired by an MIT CSAIL report on what its researchers call weight-space programming. The home test used a related but different method: a Claude Opus 5.5 session wrote code to run Ornith, measure a direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. It did not train the model, add legal data, or have an AI read and directly edit the weights by reasoning over them.
On the 10 published tasks, the edited model completed every answer, but the report says the legal score stayed at 0. In 50 checked criteria, it met seven after editing, up from three. The 10 edited answers contained 55 wrong or invented citations, according to the experiment’s citation checks. The benchmark tests questions about German, U.S. and EU law, with citations checked against primary sources.
The comparison needs qualification. The unchanged model answered eight of 10 tasks in a previously published run, but answered only four of 10 on the setup used for the edit comparison. The report also says that using the model card’s recommended sampling settings, without changing the weights, raised completion to nine of 10 on the original setup. The figures therefore show a result on one particular setup, not a stable estimate of performance across conditions.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Fewer Loops, Not Better Law
The experiment illustrates a distinction that matters for anyone evaluating AI systems: finishing an answer is not the same as answering correctly. A model that stops repeating itself may be easier to use, but completion counts alone cannot establish that its output is accurate, especially in a high-stakes area such as law.
The result also offers a small, concrete test of the idea behind the MIT work. If a model’s failure comes from getting stuck, modifying its internal behavior may remove that obstacle. But the home test did not add legal knowledge, and the benchmark score did not rise. The reported citation errors underline why a technical improvement in output behavior should not be treated as evidence of dependable legal advice.
For readers interested in local AI, the report suggests that a weight edit can change how a model behaves without retraining it. It does not establish that the technique generalizes to other models, tasks or hardware setups. The experiment’s authors explicitly describe it as limited, and the article says its results are not legal advice.
Products Worth Considering
Samsung Galaxy Tab S7 11-inch Android Tablet 128GB Wi‑Fi with S Pen Mystic Black
Samsung Galaxy Tab A11+ 8GB RAM, 256GB Storage, Gray
Jackery Explorer 1000 v2 Portable Power Station, 1070Wh, 1500W
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Home Test
The MIT CSAIL study described in the source material involved researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. In a reported Qwen case, GPT-6 Astra analyzed a model’s weights and identified a direction associated with looping; the researchers subtracted it from a small portion of the weights. The home experiment took inspiration from that result, but used a more established approach called directional ablation: run the model, record internal signals, estimate a direction and project it out of selected weights.
The chosen model, Ornith 1.5 35B-A3B, is a mixture-of-experts model whose publisher describes it as intended for coding and agent tasks, not legal work. The report says two of the benchmark’s other local models did not show the same looping problem, while another looping model was too large for the available machine and ran under a different token budget. Ornith was therefore selected as the workable candidate for this test.
Lexassay asks models 15 questions across German, U.S. and EU law, without internet access or tools. Ten tasks are published and five are held back. The article’s updated figures refer to those 10 published tasks, and it says the runs, answers, grades and citation checks are public on Lexassay.
“The model finishes is not the answer is right.”
— Thorsten Meyer, describing the home experiment
Limits of the One-Run Test
The findings come from one run per setting, a small development set and a loop detector that changed during the experiment, according to the report. That makes it difficult to tell how much of the difference came from the weight edit, sampling choices or other setup details. The gap between the unchanged model’s eight completed answers in one run and four in another illustrates that sensitivity.
The report also discloses that Claude carried out the edit and Claude agents did the blind grading. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers and claims and found faults, according to the author. That independent review is a check, but it does not make this a second experiment or settle the results. Three graders came from the same model family, the article says.
It remains unclear whether the edit would produce similar results across repeated runs, different models or the five unpublished benchmark tasks. The published account also does not establish that the model’s legal answers are safe to use in practice; its score remained zero and the reported citation checks found substantial errors.
Public Runs and Further Checks
The experiment’s immediate next step is scrutiny of the public record: the article points readers to the Lexassay runs, including the original model, model-card sampling, weight-edited model and edit control. Those materials allow readers to inspect the answers, grading and citation checks rather than relying only on the summary figures.
To establish whether the edit itself caused the change, further testing would need repeated runs under matched settings and a clearly fixed loop-detection method. Testing on additional tasks, including the held-back questions, could help show whether the effect extends beyond the published set. Until such results are available, the finding is best treated as a limited demonstration of changed output behavior, not evidence of improved legal capability.
Key Questions
What changed in the local model?
A Claude Opus 5.5 session wrote code that measured a direction associated with looping and removed it at half strength from the output weights of 27 of Ornith’s 40 layers. The model was not retrained and received no new legal data.
Did the edit improve the model’s legal accuracy?
No improvement was reported in the legal score: it remained zero. The edited answers met seven of 50 checked criteria, compared with three before, and contained 55 wrong or invented citations, according to the article.
Why did it answer more tasks after editing?
The report links the change to reduced looping: the model completed all 10 published tasks on the edited setup, compared with four on the same comparison setup before editing. But the result is limited by one run per setting and by differences across setups; recommended sampling settings also raised completion without an edit.
Was this the same method used by the MIT researchers?
No. The home test used directional ablation, which measures model activity by running the model and then modifies selected weights. The MIT example described in the article involved GPT-6 Astra reasoning directly about the weights. The home experiment was inspired by that work but did not reproduce its method.
Can the edited model be used for legal advice?
The experiment does not support using it for legal advice. Its legal score stayed at zero, and the reported citation checks found errors. The source article also explicitly says the experiment is not legal advice.
Source: ThorstenMeyerAI.com
