MIT Uses GPT-6 Astra to Boost Alibaba’s Qwen Math Performance by Editing Just 0.016% of Its Weights

Researchers at the Massachusetts Institute of Technology (MIT) have used OpenAI’s GPT-6 Astra to improve the mathematical performance of Alibaba’s Qwen3.5-0.8B AI model by directly modifying its internal neural network weights. The experiment increased Qwen’s score on the MATH-500 benchmark from 7.9% to 48.1%, while changing just 0.016% of its approximately 800 million parameters.

The researchers call the technique Weight-Space Programming. Instead of using conventional training methods, gradient descent or backpropagation, GPT-6 Astra inspected Qwen’s internal weights, identified a problem causing repetitive reasoning and modified selected parameters to correct it. The experiment suggests that advanced AI models may eventually help diagnose and repair other neural networks directly, although the researchers describe the results as early-stage research.

Key Takeaways

• MIT CSAIL researchers demonstrated a technique called Weight-Space Programming.
• They used OpenAI’s GPT-6 Astra to modify Alibaba’s Qwen3.5-0.8B model.
• Only 0.016% of Qwen’s weights were changed.
• MATH-500 performance increased from 7.9% to 48.1%.
• GSM8K performance improved from 13.8% to 67.1%.
• Instruction-following performance increased from 21% to 62.3%.
• The researchers identified repetitive reasoning as the main problem.
• The experiment did not use gradient descent or conventional model training.
• A separate experiment showed that the approach could not significantly improve a model that lacked the underlying mathematical capability.

What Is Weight-Space Programming?

Neural networks contain millions or billions of numerical parameters called weights.

These weights determine how an AI model processes information, recognises patterns and generates responses.

Traditionally, developers improve AI models through training and fine-tuning. These processes use datasets, optimisation algorithms and feedback to adjust the weights.

MIT’s Weight-Space Programming experiment explores a different approach.

Instead of asking an AI model to generate training code or run a conventional optimisation process, researchers gave GPT-6 Astra direct access to another model’s numerical weights.

The goal was to see whether a capable language model could understand how those weights influenced behaviour and make targeted changes.

In this experiment, GPT-6 Astra was asked to improve Qwen’s mathematical performance.

It inspected the model, investigated its failures, tested possible explanations and directly modified selected parameters.

The researchers believe this could eventually create a new way to debug and improve neural networks.

How GPT-6 Astra Improved Alibaba’s Qwen Model

The experiment focused on Alibaba’s Qwen3.5-0.8B, a relatively small language model containing approximately 800 million parameters.

The researchers noticed that Qwen frequently became trapped in repetitive reasoning loops when solving mathematical problems.

Instead of completing a calculation and producing an answer, the model sometimes repeated the same reasoning until it reached its maximum output limit of 32,768 tokens.

During the initial investigation, 85 out of 96 practice answers reached this limit.

The researchers gave GPT-6 Astra access to Qwen’s weights through a Python environment.

It could inspect the model’s parameters and use a limited number of forward passes to test its behaviour. However, it was not allowed to use gradient information or conventional training procedures to update the model.

Astra eventually identified a direction associated with repetitive behaviour in the model’s internal representations.

It traced the problem to the multilayer perceptron components in layers 15 and 19.

The AI then modified selected weight matrices to weaken the signal responsible for the repeated reasoning.

This intervention changed 131,072 weights, representing approximately 0.016% of Qwen’s total parameters.

The result was a substantial improvement in the model’s ability to complete mathematical responses.

Qwen’s Mathematical Performance Improved Significantly

The most important results came from three benchmarks used to evaluate mathematical reasoning and instruction-following capabilities.

Benchmark	Original Qwen3.5-0.8B	After Astra’s edit
MATH-500	7.9%	48.1%
GSM8K	13.8%	67.1%
IFEval	21.0%	62.3%

<text weight=”medium” color=”default”>Qwen’s performance before and after weight editing</text>

<Chart content={{“chartType”:”bar”,”meta”:{“description”:”Scores reported by MIT researchers under their experimental evaluation settings.”},”xKey”:”benchmark”,”series”:[{“dataKey”:”original”,”label”:”Original Qwen”,”valueSuffix”:”%”},{“dataKey”:”edited”,”label”:”After Astra’s edit”,”valueSuffix”:”%”}],”yAxisMin”:0,”yAxisMax”:100,”data”:[{“benchmark”:”MATH-500″,”original”:7.9,”edited”:48.1},{“benchmark”:”GSM8K”,”original”:13.8,”edited”:67.1},{“benchmark”:”IFEval”,”original”:21,”edited”:62.3}]}}/>

The MATH-500 result represents an improvement of more than six times over the original score.

The GSM8K benchmark, which evaluates mathematical problem-solving, also showed a substantial increase.

Interestingly, the changes improved instruction-following performance as well.

The researchers attributed this broader improvement to the fact that repetitive reasoning was not limited to mathematical questions. Removing the underlying problem helped Qwen complete other types of responses.

The share of MATH-500 answers that reached the 32,768-token limit also fell from 93% to 33.1%.

Did GPT-6 Astra Train Qwen From Scratch?

No. This distinction is important.

GPT-6 Astra did not retrain Qwen using a new dataset or teach it additional mathematical concepts through conventional fine-tuning.

Instead, it corrected a weakness that prevented Qwen from effectively using capabilities it already possessed.

The experiment still involved computation and testing. Astra received access to practice questions from the GSM8K and MATH datasets and was allowed up to 50 forward passes in its strongest attempt.

A forward pass means running information through the model to observe its output.

However, the researchers did not allow Astra to use gradients, backpropagation or a traditional training loop to calculate the weight updates.

The result is therefore better understood as targeted model repair rather than creating new mathematical intelligence from scratch.

Researchers Also Tested the Technique on Other AI Models

The MIT team conducted several experiments beyond Qwen.

They explored whether GPT-6 Astra could identify what a neural network had been trained to do simply by examining its weights.

In some experiments, Astra correctly identified the games that small neural networks were designed to play.

It also examined diffusion-model weights and identified the shapes those models were trained to generate.

Another experiment involved creating small game-playing neural networks directly by writing their weights.

However, the researchers also identified important limitations.

In one test involving two versions of Google’s Gemma 3 270M model, Astra could not reliably identify which model had been trained to agree with users simply by examining the weights.

It succeeded only after receiving additional information about the models’ attention patterns.

The team also attempted to improve Gemma’s mathematical performance. Because that model did not suffer from the same repetitive reasoning problem, Astra could not produce a meaningful improvement.

Its MATH-500 score moved only from 6.8% to at most 7.7%, a gain the researchers did not establish as significant.

This finding is important: removing an obstacle can help a model use existing abilities, but it does not automatically give the model capabilities it never learned.

Why This Research Matters for AI Development

The research could have implications for how developers maintain and improve AI systems.

Today, improving an existing model often requires collecting data, running experiments, fine-tuning parameters and evaluating the results.

Weight-Space Programming introduces the possibility of using an advanced AI model to inspect another neural network and directly propose targeted repairs.

Potential applications include:

• Model debugging: Identifying specific internal components responsible for repetitive or unwanted behaviour.
• AI interpretability: Understanding how neural network weights relate to model capabilities.
• Targeted optimisation: Correcting certain failures without running a complete training process.
• Model auditing: Investigating potential weaknesses or hidden behaviours within neural networks.
• Model maintenance: Helping developers diagnose problems in existing AI systems.

However, the research has not yet established that this approach will consistently be faster or cheaper than conventional fine-tuning. The Qwen experiment itself required substantial investigation and approximately 20.3 hours of elapsed experimental time, including waiting through a service outage.

The Bigger Picture

The MIT experiment highlights a potentially important development in AI research: advanced language models may become capable of working directly with the internal structures of other neural networks.

Until now, large language models have primarily been used to write code, generate data, assist with research and help automate training workflows.

Weight-Space Programming explores a more direct role in which an AI system can inspect and modify the numerical parameters that determine another model’s behaviour.

If this approach becomes reliable, it could create new possibilities for model debugging, interpretability and optimisation.

However, the current findings are early evidence rather than proof that AI systems can independently improve any neural network.

Looking Ahead

The MIT researchers say they are developing a dedicated benchmark and testing framework to evaluate Weight-Space Programming more systematically. Further experiments across larger models and different failure types will be necessary to establish whether the technique can work beyond carefully selected examples.

For the wider AI industry, the research raises an interesting possibility: future AI systems may not only help people build software and train models but also inspect, diagnose and directly repair the neural networks powering other AI applications. The ability to do this reliably could change how developers maintain and improve AI models.