From Databricks with Love: DBRX 132B instruct, an Open Source LLM Challenges GPT-4

What is DBRX? What is DBRX 132B instruct?

DBRX is a state-of-the-art open LLM from Databricks that outperforms other open models and rivals top closed models across a range of benchmarks, making powerful generative AI more accessible to enterprises and researchers.

Databricks, a leading data and AI company, has just released DBRX - a powerful new open-source large language model (LLM) that is setting new benchmarks for open LLMs. DBRX comes in two versions:

• DBRX Base: The pre-trained base model that serves as a general text-completion model
• DBRX Instruct: A fine-tuned version of the base model optimized for instruction-following tasks

What makes DBRX stand out is its impressive performance across a wide range of standard benchmarks, where it outperforms other established open LLMs and even rivals some of the best closed-source models. Let's dive into the details of this exciting new model.

Architecture and Training of DBRX 132B instruct

DBRX utilizes a transformer-based decoder-only architecture with a mixture-of-experts (MoE) setup. Here are some key numbers:

• 132 billion total parameters
• 36 billion active parameters on any given input
• Uses 16 experts, of which 4 are activated per token
• Trained on a massive 12 trillion token dataset
• 32k token maximum context length

The MoE architecture allows DBRX to achieve high quality output with faster inference compared to dense models of similar size. Techniques like gated linear units, grouped query attention, and rotary position encodings further boost its performance.

Databricks leveraged their suite of tools to train DBRX over a period of 3 months using 3072 NVIDIA H100 GPUs. Their training pipeline is now nearly 4x more efficient than earlier models, which will benefit customers looking to train customized models on their own data.

Benchmarks and Comparisons of DBRX 132B instruct

DBRX showcases state-of-the-art performance on several key benchmarks:

1. MMLU (Massive Multitask Language Understanding): DBRX achieves better performance than all other open models tested
2. HumanEval (Code generation): Rivals specialized code models like CodeLLaMA-70B
3. GSM8K (Math word problems): Outperforms other open models and is competitive with top closed models

Compared to other open LLMs, DBRX beats the likes of LLaMA2-70B, Mixtral, and Grok-1 on most benchmarks. It generates responses 2-3x faster than a 132B non-MoE model like LLaMA2-70B.

Even more impressively, DBRX goes toe-to-toe with some of the best closed-source models:

• Outperforms GPT-3.5 on language understanding, programming, and math benchmarks
• Competitive with Gemini 1.0 Pro and Mistral Medium in many areas

This positions DBRX as a powerful open alternative to state-of-the-art closed models.

Getting Started with DBRX

There are several ways to start using DBRX:

1. Hugging Face: Download the off-the-shelf DBRX Instruct model from the Hugging Face Hub
2. Databricks APIs: Available for Databricks customers to pre-train custom models or continue training existing checkpoints
3. Databricks Playground: Experiment with the model using Databricks' chat interface demo

DBRX particularly shines in retrieval-augmented generation (RAG) tasks, making it a top choice for applications like SQL generation.

DBRX 132B instruct: Best Open Source LLM from Databricks

Databricks has made DBRX freely available on GitHub and Hugging Face under a non-commercial license similar to LLaMA. This open approach lowers barriers to entry and allows enterprises to develop their own GenAI applications without exorbitant costs.

The release of DBRX is a major milestone in Databricks' ongoing investment in open LLMs. As they continue to refine their models and make them more accessible, we can expect to see even more impressive capabilities emerge.

Conclusion

With DBRX setting a new bar for open LLM performance, the future looks bright for enterprises and researchers looking to harness the power of generative AI. As the technology continues to advance at a rapid pace, models like DBRX will play a key role in democratizing access to cutting-edge language AI.