📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM was built from scratch with extensive native-language data, yet underperformed on Italian academic benchmarks. This challenges assumptions about the relationship between data scale and language understanding in sovereign models.
Italy’s Minerva-3B, a large-scale European sovereign language model trained entirely from scratch on 2.5 trillion tokens, scored just 4.9% on the INVALSI Italian school-benchmark, revealing significant limitations despite extensive native-language data and institutional backing.
Minerva, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, was designed to produce a high-performing Italian language model through large-scale training. The project trained models ranging from 350 million to 7 billion parameters, using approximately 50% Italian data across 2.5 trillion tokens. Despite this, the 3B parameter model achieved only 4.9% accuracy on the INVALSI Italian school exams, a result near chance level, which is a stark contrast to its impressive technical benchmarks.
Researchers from the Minerva team emphasized that while dataset composition and size are important, the overall scale of parameters and training data remain critical for complex language tasks. The results suggest that even substantial native-language investment may not be sufficient at current parameter scales to achieve meaningful country-specific knowledge depth. The project’s open weights, data, and methodology have become a reference for European sovereign-LLM efforts, but the empirical findings highlight fundamental challenges in scaling language models for complex, real-world applications.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Multi-Agent Systems Engineering: Design architecture with evidence: metrics, risk gating, failure modes, and tested reference code—benchmarks, debugging, and production hardening for AI agents
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Advanced Language Tool Kit: Teaching the Structure of the English Language
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale on Sovereign-Language Models
The results from Minerva demonstrate that extensive native-language data and large-scale training alone may not guarantee high performance on complex academic or societal tasks. This raises critical questions for European nations investing heavily in sovereign AI: how much scale is truly necessary to develop models capable of understanding and reasoning within their own languages? The findings suggest that current parameter scales may be insufficient, prompting a reevaluation of strategic investments and expectations for national AI initiatives.
For policymakers and researchers, this underscores the importance of balancing data quantity, model size, and task complexity. It also signals that achieving a deep, country-specific knowledge base in language models might require even larger models or innovative architectural approaches, beyond simply increasing data and parameters.
European Sovereign-LLM Strategies and Challenges
Italy’s Minerva project represents a significant effort to build a European sovereign language model from scratch, utilizing extensive national resources, including the CINECA supercomputer and PNRR funding. The project aimed to demonstrate that a dedicated, native-language model could outperform multilingual counterparts on language-specific benchmarks. This approach contrasts with models like Portugal’s AMÁLIA, which focused on continuation pre-training with limited European data. Despite the heavy investment—training on 2.5 trillion tokens with half Italian content—Minerva’s performance on real-world academic tests has been disappointing, with a 4.9% score on INVALSI exams.
Prior European efforts have often emphasized multilingual or continuation strategies, but Minerva’s results highlight the potential limitations of scale alone. The project’s open data and weights have made it a reference case, but the empirical findings challenge assumptions that native data and large models automatically lead to high country-specific knowledge.
“Our results suggest that even with 2.5 trillion tokens, the model struggles with academic benchmarks, indicating a need for even larger or more specialized models.”
— Research team member, Minerva project
What Factors Limit Minerva’s Academic Performance?
It is not yet clear whether the low INVALSI scores are due to fundamental limitations of model architecture, insufficient scale relative to task complexity, or other factors such as training methodology. The ongoing research aims to clarify whether larger models, different training strategies, or architectural innovations could improve performance.
Next Steps for European Sovereign-Language Model Development
The Minerva team plans to continue iterating on training methodology, including ongoing experiments with larger models and different data strategies. Future evaluations will focus on whether increased scale or new architectures can bridge the gap between technical benchmarks and real-world language understanding. Policymakers and researchers will need to reassess investment levels and strategic priorities based on these empirical insights.
Key Questions
Why did Minerva perform poorly on the Italian school benchmark?
The low performance suggests that even large-scale native-language training may not be sufficient at current parameter scales to develop deep country-specific knowledge and reasoning abilities.
Does this mean European sovereign models are not worth pursuing?
Not necessarily. It indicates that scale and investment must be carefully balanced, and that innovative approaches may be required to achieve desired performance levels.
What are the implications for future AI investments in Europe?
Investments should consider not only data and scale but also architectural innovation and task-specific training to ensure models meet real-world needs.
Will larger models improve Minerva’s performance?
Theoretically, increasing model size could help, but empirical results suggest that scale alone may not be enough without methodological improvements.
What does this mean for other countries developing sovereign LLMs?
It highlights the importance of realistic expectations about the relationship between data scale, model size, and task complexity in building effective country-specific models.
Source: ThorstenMeyerAI.com
