New Finance Agent Benchmark Released

Released date

Model

Last Updated 9/24/2024

google/gemini-1.5-pro-002

Gemini 1.5 Pro (002)

Gemini 1.5 Pro, 002 model.

Released Date: 9/24/2024

Avg. Accuracy:

64.4%

Latency:

10.29s

Performance by Benchmark

Benchmarks

Accuracy

Rankings

50.3%

( 35 / 44 )

50.3%

35 / 44

67.6%

( 63 / 73 )

67.6%

63 / 73

68.0%

( 26 / 76 )

68.0%

26 / 76

64.9%

( 49 / 60 )

64.9%

49 / 60

71.0%

( 18 / 35 )

71.0%

18 / 35

82.8%

( 31 / 54 )

82.8%

31 / 54

18.7%

( 35 / 50 )

18.7%

35 / 50

89.2%

( 34 / 53 )

89.2%

34 / 53

71.6%

( 50 / 75 )

71.6%

50 / 75

76.5%

( 41 / 56 )

76.5%

41 / 56

58.3%

( 31 / 52 )

58.3%

31 / 52

75.3%

( 29 / 50 )

75.3%

29 / 50

41.7%

( 38 / 51 )

41.7%

38 / 51

65.5%

( 21 / 32 )

65.5%

21 / 32

Academic Benchmarks

Proprietary Benchmarks (contact us to get access)

Cost Analysis

Input Cost

$1.25 / M Tokens

Output Cost

$5.00 / M Tokens

Input Cost (per char)

$0.33 / M chars

Output Cost (per char)

N/A

Join our mailing list to receive benchmark updates on

Stay up to date as new benchmarks and models are released.