OVHcloud GPU benchmark - Llama 3

Go back to list
eval_rate_mean prompt_eval_rate_mean real_duration total_duration
mean std mean std mean std mean std
model llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b llama3 llama3.1:70b-instruct-q8_0 llama3.1:8b-instruct-q8_0 llama3:70b
provider_name flavor_name flavor__gpu_model
Amazon Web Services g5.xlarge NVIDIA A10G 74.111 49.467 0.116 0.061 1790.283 6057.143 280.503 1159.488 1.829 2.633 0.052 0.044 1815.187 2618.853 28.654 3.292
p3.2xlarge Tesla V100-SXM2-16GB 84.257 61.688 3.922 2.441 1367.541 4530.832 195.036 985.301 1.633 2.125 0.086 0.106 1628.533 2120.049 85.989 104.691
Google Cloud a2-highgpu-1g NVIDIA A100-SXM4-40GB 94.259 82.421 14.015 0.886 0.537 0.049 1827.559 6559.392 282.322 302.885 1315.156 46.402 1.471 1.603 9.370 0.040 0.050 0.042 1461.260 1586.136 9367.130 30.321 10.091 42.357
a2-ultragpu-1g NVIDIA A100-SXM4-80GB 98.542 17.473 91.599 25.292 0.666 0.062 0.232 0.089 1869.744 489.769 6765.741 341.193 325.284 82.318 1132.676 47.641 1.396 7.457 1.435 5.266 0.015 0.038 0.008 0.032 1392.049 7453.604 1429.843 5262.839 14.032 37.515 4.218 31.476
g2-standard-16 NVIDIA L4 44.201 27.007 0.133 0.082 1441.690 4051.748 327.826 897.972 3.009 4.795 0.053 0.054 2994.856 4776.347 9.981 14.964
n1-highmem-8 Intel Skylake Tesla V100 Tesla V100-SXM2-16GB 88.481 64.671 0.711 0.210 1333.016 4659.428 186.582 818.092 1.531 2.025 0.027 0.039 1527.067 2013.731 26.998 7.155
Microsoft Azure Standard_NC40ads_H100_v5 NVIDIA H100 NVL 191.022 27.382 159.899 40.712 0.561 0.138 0.999 0.232 2934.673 805.944 10733.333 566.184 594.193 147.350 2140.316 114.136 0.739 4.756 0.852 3.286 0.035 0.030 0.108 0.029 729.699 4753.478 821.442 3283.044 19.831 29.796 5.120 28.875
Standard_NC6s_v3 Tesla V100-PCIE-16GB 85.940 63.158 1.109 0.159 1227.860 4360.000 171.588 714.683 1.597 2.067 0.022 0.005 1593.439 2062.110 20.897 5.956
Standard_NV36ads_A10_v5 NVIDIA A10 79.827 0.217 1817.759 266.258 1.695 0.036 1684.996 12.166
OVHcloud A10-45 NVIDIA A10 52.236 0.176 5883.968 1422.360 2.499 0.015 2493.031 12.606
H100-380 NVIDIA H100 PCIe 120.238 21.133 124.898 29.748 9.947 0.115 1.918 0.254 2492.966 704.116 6200.741 513.873 462.381 139.743 1888.363 95.752 975.793 6.172 1.091 4.522 1266.185 0.036 0.063 0.041 5837.149 6166.975 1069.573 4516.216 26506.018 36.312 19.267 39.990
L4-90 NVIDIA L4 48.929 28.929 0.078 0.038 1438.877 3975.031 317.703 999.894 2.750 4.491 0.046 0.054 2729.533 4472.413 9.672 7.771
L40S-90 NVIDIA L40S 115.282 72.121 16.647 0.764 0.464 0.065 2909.430 7412.037 510.268 785.319 2645.250 127.983 1.194 1.826 7.858 0.047 0.014 0.028 1181.888 1819.426 7853.295 25.157 12.093 27.646
T1-45 Tesla V100-PCIE-16GB 60.492 0.639 3543.024 810.268 2.202 0.058 2182.698 20.545
T1-LE-45 Tesla V100-PCIE-16GB 84.138 1.506 1122.191 175.381 1.663 0.040 1648.902 31.604