Google unveils its latest Gemini 1.5 Pro model, utilizing the MoE architecture. With a context window of up to 1 million tokens, it matches the performance of the Gemini 1.0 Ultra model. Currently in preview, developers can test it via AI Studio and Vertex AI.
Following the recent launch of Gemini 1.0 Ultra under the Bard rebrand, Google introduces the Gemini 1.5 Pro model to rival GPT-4. As the successor to Gemini 1.0 Pro, it powers the free version of Gemini (formerly Bard). While the Gemini 1.0 model family offers a context window of up to 32K tokens, the 1.5 Pro model elevates the standard context length to 128K tokens. Notably, it supports a massive context window of up to 1 million tokens, surpassing GPT-4 Turbo’s 128K and Claude 2.1’s 200K tokens.
Gemini 1.5 Pro Built on Mixture-of-Experts (MoE) Architecture
Google describes the Gemini 1.5 Pro as a mid-size model, yet it boasts performance almost on par with the Gemini 1.0 Ultra while requiring less compute. This feat is achieved because the 1.5 Pro model is constructed on the Mixture-of-Experts (MoE) architecture, akin to OpenAI’s GPT-4 model. It marks Google’s debut of an MoE model, departing from a single dense model approach.
In case you’re unfamiliar with the MoE architecture concept, it comprises multiple smaller expert models activated based on the task at hand. Utilizing specialized models for specific tasks leads to improved and more efficient outcomes.
Regarding the expansive context window of Gemini 1.5 Pro, it can process substantial amounts of data in a single operation. According to Google, the 1 million context length can handle 700,000 words, 1 hour of video, 11 hours of audio, or codebases with over 30,000 lines of code.
To evaluate Gemini 1.5 Pro’s retrieval capability, given its extensive context window, Google conducted the Needle In A Haystack challenge. According to the company, the model successfully recalled the needle (text statement) 99% of the time.
Just to clarify, the 1.5 Pro model is currently in preview, and only developers and customers can test the new model using AI Studio and Vertex AI. You can click on the link to join the waitlist. Access to the model will be free during the testing period.
