Gemini 2.0 Flash

Natural language to SQL conversion evaluation

Automatic speech translation (BLEU score) across 21 languages

Video analysis across multiple domains

Ability to provide factuality correct responses given documents and diverse user requests

Challenging dataset of questions written by domain experts in biology, physics, and chemistry

Competition-level math problems, Held out dataset AIME/AMC-like

Code generation in Python. Code Generation subset covering more recent examples: 06/01/2024 - 10/05/2024

Challenging math problems including algebra, geometry, pre-calculus, and others

Enhanced version of MMLU dataset evaluation

Multi-discipline college-level multimodal understanding and reasoning problems

Novel, diagnostic long-context understanding evaluation

Code generation evaluation across multiple languages

Visual understanding in chat models with challenging everyday examples