George Cameron

georgewritescode

AI & ML interests

None yet

Articles

Organizations

Posts 2

view post
Post
997
Visualization of GPT-4o breaking away from the quality & speed trade-off curve the LLMs have followed thus far ✂️

Key GPT-4o takeaways
‣ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
‣ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
‣ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)

👉 How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)

Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)

Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
view post
Post
2300
Excited to bring our benchmarking leaderboard of >100 LLM API endpoints to HF!

Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.

Coverage:
‣ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
‣ Throughput (tokens/s: median, P5, P25, P75, P95)
‣ Latency (TTFT: median, P5, P25, P75, P95)
‣ Context window
‣ OpenAI library compatibility

Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard

Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis

models

None public yet

datasets

None public yet