GPT-4o-2024-08-06: What's New and How to Use Structure Data?

OpenAI has once again pushed the boundaries of artificial intelligence with the release of GPT-4o-2024-08-06, introducing Structured Outputs to the API. This groundbreaking feature marks a significant leap forward in the reliability and usability of AI-generated content, particularly for developers and businesses relying on structured data generation.

GPT-4o-2024-08-06: What's New?

Historically, large language models (LLMs) have excelled at generating human-like text but struggled with consistently producing structured data adhering to specific formats. GPT-4o-2024-08-06 addresses this challenge head-on with the introduction of Structured Outputs, ensuring model-generated outputs exactly match JSON Schemas provided by developers.

GPT-4o-2024-08-06 Is Really Good At JSON Schema

First things first:

• The performance improvement of GPT-4o-2024-08-06 with Structured Outputs is remarkable.
• In OpenAI's evaluations of complex JSON schema following, this new model achieves a perfect score of 100%, compared to its predecessor, GPT-4-0613, which scored less than 40% on the same tests.

GPT-4o-2024-08-06 Is Better, And Cheaper Than GPT-4O

Let's compare GPT-4o-2024-08-06 with its predecessors and the more compact GPT-4o-mini:

MMLU (Massive Multitask Language Understanding)

• GPT-4o-2024-08-06: 88.7% (5-shot)
• GPT-4o: 88.7% (5-shot)
• GPT-4o-mini: 82.0% (5-shot)
• GPT-4 (June 2023 version): 86.4% (5-shot)

MMMU (Massive Multitask Multimodal Understanding)

• GPT-4o-2024-08-06: 69.1%
• GPT-4o: 69.1%
• GPT-4o-mini: 59.4%
• GPT-4 (June 2023 version): 34.9%

Pricing (per million tokens)

• GPT-4o-2024-08-06: $5.00 input, $15.00 output
• GPT-4o: $5.00 input, $15.00 output
• GPT-4o-mini: $0.15 input, $0.60 output
• GPT-4 (June 2023 version): $30.00 input, $60.00 output

Context Window

• GPT-4o-2024-08-06: 128K tokens
• GPT-4o: 128K tokens
• GPT-4o-mini: 128K tokens
• GPT-4 (June 2023 version): 8,192 tokens

These benchmarks demonstrate that GPT-4o-2024-08-06 maintains the high performance of GPT-4o while introducing the new Structured Outputs feature. GPT-4o-mini, while not as powerful, offers a more cost-effective solution for many applications.

Structured Outputs: The New Trick from OpenAI

OpenAI has introduced Structured Outputs in two primary forms within the API:

1. Function Calling
2. Response Format Parameter

Let's explore how to use these features with step-by-step guides and sample codes.

Step-by-Step Guide: Using Structured Outputs

1. Function Calling with Structured Outputs

Step 1: Define your function with a strict schema

import openai

function_schema = {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location",
    "parameters": {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
            },
            "temperature_unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"]
            }
        },
        "required": ["location", "temperature_unit"]
    },
    "strict": True  # This enables Structured Outputs
}

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "What's the weather like in Boston?"}
    ],
    functions=[function_schema],
    function_call={"name": "get_current_weather"}
)

print(response.choices[0].message.function_call)

2. Response Format Parameter with Structured Outputs

Step 1: Define your JSON schema

json_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "cities_visited": {
            "type": "array",
            "items": {"type": "string"}
        }
    },
    "required": ["name", "age", "cities_visited"]
}

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "Generate a profile for a world traveler named John who is 30 years old."}
    ],
    response_format={"type": "json_object", "json_schema": json_schema}
)

print(response.choices[0].message.content)

3. Using Structured Outputs with SDKs

OpenAI has updated its Python and Node SDKs with native support for Structured Outputs. Here's an example using the Python SDK with Pydantic:

from pydantic import BaseModel, Field
from typing import List
from openai import OpenAI

class Traveler(BaseModel):
    name: str
    age: int
    cities_visited: List[str] = Field(min_items=1)

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": "Generate a profile for a world traveler named Sarah who is 28 years old."}
    ],
    response_format={"type": "json_object", "schema": Traveler.model_json_schema()}
)

traveler = Traveler.model_validate_json(response.choices[0].message.content)
print(f"Name: {traveler.name}, Age: {traveler.age}, Cities visited: {', '.join(traveler.cities_visited)}")

Best Practices for Using Structured Outputs

1. Define Clear Schemas: Ensure your JSON schemas are well-defined and cover all possible outputs.
2. Handle Refusals: Implement logic to handle cases where the model refuses to generate output due to safety concerns.
3. Validate Outputs: Although Structured Outputs guarantees schema compliance, always validate the content for accuracy.
4. Optimize for Performance: Cache preprocessed schemas to reduce latency on subsequent requests.
5. Combine with Function Calling: Use Structured Outputs in conjunction with function calling for more complex applications.

Limitations and Considerations

Despite its advancements, GPT-4o-2024-08-06 and Structured Outputs have some limitations:

• Only a subset of JSON Schema is supported.
• The first API response with a new schema incurs additional latency.
• While structure is guaranteed, content accuracy is not.
• Structured Outputs is not compatible with parallel function calls.
• JSON Schemas used are not eligible for Zero Data Retention (ZDR).

Conclusion

GPT-4o-2024-08-06 and its Structured Outputs feature represent a significant advancement in AI-generated content reliability. By solving the challenge of consistently producing structured data, OpenAI has unlocked new possibilities for developers and businesses. As the AI landscape continues to evolve, GPT-4o-2024-08-06 sets a new standard for precision and structure in AI-powered applications, paving the way for more sophisticated and dependable AI systems across various industries.