On the evening of July 23rd Beijing time, Meta officially released the latest open-source model Llama 3.1 series, further narrowing the gap between open-source models and closed source models. Llama 3.1 includes three parameter scales of 8B, 70B, and 450B, with the 450B parameter model surpassing OpenAI's GPT-4o in multiple benchmark tests and comparable to leading closed source models such as Claude 3.5 Sonnet.
Meta founder and CEO Mark Zuckerberg also posted a blog on the official website at the same time to promote the release. He stated that Llama 3.1 version will be a turning point in the industry, and most developers will begin to primarily use open source. Open source AI is the future direction of development.
NVIDIA Senior Research Scientist Jim Fan congratulated the Meta team on X, stating, "The power of GPT-4 is in our hands, and this is a truly historic moment
In terms of specific details, the context windows of the three versions of Llama 3.1 have increased from 8K to 128K, a 16 fold expansion, and support 8 languages simultaneously. The Llama 3.1-405B model was trained using over 15 trillion tokens, and in order to achieve this training scale, the team used 16000 H100 GPUs. Officially, the 405B model is the first Llama model trained at this scale.
Open source large-scale language models often lag behind closed source models in terms of functionality and performance, but now we are ushering in a new era led by open source
In the official blog, Meta evaluated the performance of over 150 benchmark datasets and compared the performance of Llama 3.1 with other models. The flagship model Llama 3.1-405B is comparable to GPT-4, GPT-4o, and Claude 3.5 Sonnet in a range of tasks such as common sense, operability, and mathematics. In addition, the 8B and 70B small models are competitive with closed source and open source models with similar numbers of parameters.
In real-world scenarios, Llama 3.1 405B performed better overall than GPT-4o and Claude 3.5 Sonnet compared to manual evaluations.
Meta has also updated its open source license this time, allowing developers to use the output of the Llama model (including 405B) for the first time to improve other models. Compared to GPT-4o, the official statement states that they will also use a combination approach to integrate image, video, and voice functions into Llama 3, enabling the model to recognize images and videos and support interaction through voice. However, this feature is still under development and is not yet ready for release.
In the official blog, Meta stated that the total download volume of all Llama versions has exceeded 300 million times so far.
In addition to this model release, Zuckerberg also posted a long article on the official website titled "Open Source AI Is the Path Forward", which mentioned the importance of open source. He believes that open source is good for all developers, Meta, and the world.
Zuckerberg used the example of open-source system Linux defeating closed source system Unix, believing that artificial intelligence will develop in a similar way. Several technology companies are developing leading closed models, but open source is quickly narrowing the gap. He mentioned that last year, Llama 2 could only be compared to the old generation models. And this year, Llama 3 has competitiveness in some fields, even leading the most advanced models in some aspects.
Zuckerberg believes that open source can promote innovation, reduce costs, and improve security. For developers, using open source can train, fine tune, and distill their own models. Each organization has different needs, and it is best to use models of different sizes to meet these needs, which are trained or fine tuned with specific data.
Meanwhile, developers can avoid being locked into closed vendors to protect data security. Open source software is often more secure because its development is more transparent and can be widely reviewed, "said Zuckerberg.
Zuckerberg also mentioned that open-source models have lower costs and higher efficiency, allowing developers to run inference on Llama 3.1 405B on their own infrastructure at a cost of approximately 50% of using closed models like GPT-4o, suitable for user interface and offline inference tasks.
Open source artificial intelligence represents the world's best opportunity. In Zuckerberg's view, utilizing this technology can create the greatest economic opportunity and security.