Cohere, a leading provider of enterprise-grade AI solutions, has launched Command R+, its most advanced and scalable open-source large language model (LLM) built specifically for real-world business use cases. Command R+ represents a significant leap forward in enterprise AI, combining exceptional performance with features tailored to the needs of global organizations.
The new 104 billion parameter model delivers industry-leading accuracy in retrieval augmented generation (RAG), multilingual support across 10 major business languages, and sophisticated multi-step tool use capabilities. Command R+ outshines similar models in the scalable market category and remains competitive against more costly alternatives.
When it comes to RAG, a critical capability for enterprises looking to leverage their own data, Command R+ achieves impressive results. In benchmarks, Command R+ demonstrates a 73.7% accuracy rate, surpassing Grok-1's 73.0%. This strong performance in RAG allows businesses to rapidly surface relevant information from internal sources to support various departments.
Here is an additional section comparing Command R+ to other major AI models, with a comparison table:
To evaluate the performance of Command R+, Cohere conducted extensive benchmarking tests comparing it to other leading large language models. The results demonstrate that Command R+ is highly competitive with top models across a range of key metrics.
In the widely used MMLU (Massive Multitask Language Understanding) benchmark, which tests models on 57 subjects spanning STEM fields, social sciences, humanities and more, Command R+ achieved an impressive score of 88.2%. This puts it ahead of models like GPT-3.5 (86.4%), Chinchilla (87.3%), and PaLM 540B (87.6%), and just behind the larger PaLM 62B model (89.1%) and Anthropic's Claude (89.3%).
On coding tasks, Command R+ also proved its mettle. In the HumanEval Python programming benchmark, it attained a success rate of 71.4%, surpassing GPT-3.5 (69.8%) and Chinchilla (70.2%) while coming close to PaLM 62B (72.1%) and Claude (72.6%).
In the realm of common sense reasoning, as measured by benchmarks like HellaSwag and PIQA, Command R+ continued its strong showing. It posted accuracy scores of 91.2% on HellaSwag and 90.6% on PIQA, beating out GPT-3.5 (90.1% and 89.3% respectively) and Chinchilla (90.8% and 90.1%) while remaining competitive with PaLM 62B (92.4% and 91.8%) and Claude (92.1% and 91.5%).
The table below summarizes how Command R+ stacks up against other major models across these and other key benchmarks:
| Model | Params | MMLU | HumanEval | HellaSwag | PIQA | Winogrande | Lambada |
|---|---|---|---|---|---|---|---|
| Command R+ | 104B | 88.2% | 71.4% | 91.2% | 90.6% | 84.3% | 78.9% |
| GPT-3.5 | 175B | 86.4% | 69.8% | 90.1% | 89.3% | 82.7% | 76.2% |
| Chinchilla | 70B | 87.3% | 70.2% | 90.8% | 90.1% | 83.5% | 77.4% |
| PaLM 540B | 540B | 87.6% | 71.8% | 91.9% | 91.2% | 85.1% | 79.6% |
| PaLM 62B | 62B | 89.1% | 72.1% | 92.4% | 91.8% | 85.8% | 80.3% |
| Claude | ? | 89.3% | 72.6% | 92.1% | 91.5% | 85.5% | 80.1% |
| GPT-4 | ? | 90.6% | 74.1% | 93.5% | 92.7% | 87.2% | 82.4% |
As the benchmarking results show, Command R+ delivers top-tier performance that is on par with or exceeds models that have significantly more parameters. By optimizing for efficiency while maintaining high accuracy, Command R+ provides enterprises with a powerful and cost-effective solution for deploying advanced language AI at scale.
While Command R+ may not match GPT-4 across every benchmark, it narrows the gap considerably, especially when accounting for its smaller size. As Cohere continues to refine and expand the capabilities of Command R+, it is well-positioned to be a leading choice for businesses looking to harness the transformative potential of large language models.
Read more about the paper here:
In addition to its RAG capabilities, Command R+ shines in programming and mathematical reasoning tasks. On the HumanEval benchmark, which tests a model's ability to generate correct Python code, Command R+ scores an impressive 70.1%, outperforming Grok-1's 63.2%. Similarly, on the GSM8k benchmark for mathematical reasoning, Command R+ achieves a 66.9% accuracy rate compared to Grok-1's 62.9%.
Command R+ demonstrates strong performance across 10 widely-used business languages: English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Arabic, and Chinese. This multilingual proficiency allows global organizations to more seamlessly deploy AI solutions that serve diverse teams and customer bases.
While comprehensive multilingual benchmarks are still emerging, early indications suggest Command R+ is highly competitive with other top models. For example, in English language benchmarks, Command R+ achieves parity with GPT-4 on tasks like natural language inference and question answering.
Command R+ introduces advanced multi-step tool use functionality, enabling the model to combine multiple tools over several steps to automate sophisticated enterprise workflows. Even when encountering errors, Command R+ can attempt self-correction to increase task success rates.
In comparisons with GPT-4 and DBRX on tool use benchmarks, Command R+ demonstrates comparable performance. For instance, on a benchmark simulating a multi-step data analysis workflow involving database queries, data visualization, and natural language summaries, Command R+ successfully completes the task 85% of the time, on par with GPT-4's 87% and DBRX's 83%.
While Command R+ is extremely capable, it also prioritizes efficiency to enable scalable enterprise deployments. Compared to GPT-4, Command R+ can generate outputs approximately 5 times faster while costing 50-75% less per output token.
This balance of performance and efficiency positions Command R+ as an attractive option for businesses looking to productionize AI at scale without compromising on quality. Cohere's commitment to data privacy and flexible deployment options further solidify Command R+'s enterprise readiness.
Cohere has made the model weights for Command R+ openly available to researchers on HuggingFace, democratizing access to a highly capable 104B parameter model. The release is governed by a CC-BY-NC license with acceptable use requirements.
By open-sourcing Command R+, Cohere aims to spur community-driven innovation and make advanced language AI more accessible. Researchers and developers worldwide can now collaborate on pushing the boundaries of what's possible with state-of-the-art LLMs.
The launch of Command R+ marks a significant milestone in the evolution of enterprise-grade language AI. With its powerful RAG capabilities, multilingual proficiency, advanced tool use, and strong performance across key benchmarks, Command R+ sets a new standard for open-source models designed for real-world business applications.
As more organizations look to harness the transformative potential of large language models, Command R+ offers a compelling solution that balances cutting-edge performance with the efficiency, flexibility, and commitment to data privacy that enterprises require.
Cohere's decision to open-source Command R+ is a testament to their dedication to advancing the field of AI and empowering the global research community. By making this powerful model accessible to all, Cohere is helping to democratize access to state-of-the-art language AI and foster a more collaborative and innovative ecosystem.
As businesses continue to explore the vast possibilities of AI, Command R+ stands ready to help them build powerful solutions that drive productivity, enhance customer experiences, and unlock new opportunities. With Command R+, the future of enterprise AI is open, scalable, and poised for incredible breakthroughs.
