| Before | After | ||
|---|---|---|---|
| 0 | August 8, 2024 Publication GPT‑4o System Card View PDF version Share GPT-4o Scorecard Key Areas of Risk Evaluation & Mitigation Unauthorized voice generation Speaker identification Ungrounded inference & sensitive trait attribution Generating disallowed audio content Generating erotic & violent speech Preparedness Framework Scorecard Cybersecurity Low Biological Threats Low Persuasion Medium Model Autonomy Low Scorecard ratings Low Medium High Critical Only models with a post-mitigation score of "medium" or below can be deployed. | 0 | August 8, 2024 Publication GPT‑4o System Card View PDF version GPT-4o Scorecard Key Areas of Risk Evaluation & Mitigation Unauthorized voice generation Speaker identification Ungrounded inference & sensitive trait attribution Generating disallowed audio content Generating erotic & violent speech Preparedness Framework Scorecard Cybersecurity Low Biological Threats Low Persuasion Medium Model Autonomy Low Scorecard ratings Low Medium High Critical Only models with a post-mitigation score of "medium" or below can be deployed. |
| 80 | Speaker identification We post-trained GPT-4o to refuse to comply with requests to identify someone based on a voice in an audio input, while still complying with requests to identify people associated with famous quotes. | 80 | Speaker identification We post-trained GPT‑4o to refuse to comply with requests to identify someone based on a voice in an audio input, while still complying with requests to identify people associated with famous quotes. |
| 81 | Generating copyrighted content We trained GPT-4o to refuse requests for copyrighted content, including audio, consistent with our broader practices. | 81 | Generating copyrighted content We trained GPT‑4o to refuse requests for copyrighted content, including audio, consistent with our broader practices. |
| 82 | To account for GPT-4o's audio modality, we also updated certain text-based filters to work on audio conversations, built filters to detect and block outputs containing music, and for our limited alpha of ChatGPT's Advanced Voice Mode, instructed the model to not sing at all. | 82 | To account for GPT‑4o's audio modality, we also updated certain text-based filters to work on audio conversations, built filters to detect and block outputs containing music, and for our limited alpha of ChatGPT's Advanced Voice Mode, instructed the model to not sing at all. |
| 83 | Ungrounded inference / Sensitive trait attribution We post-trained GPT-4o to refuse requests for ungrounded inference, such as "how intelligent is this speaker?". | 83 | Ungrounded inference / Sensitive trait attribution We post-trained GPT‑4o to refuse requests for ungrounded inference, such as "how intelligent is this speaker?". |
| 84 | We post-trained GPT-4o to safely comply with requests for sensitive trait attribution by hedging answers, such as "what is this speaker's accent" → "Based on the audio, they sound like they have a British accent." Disallowed content in audio output We run our existing moderation classifier over text transcriptions of audio prompts and generations, and block the output for certain high-severity categories. | 84 | We post-trained GPT‑4o to safely comply with requests for sensitive trait attribution by hedging answers, such as "what is this speaker's accent" → "Based on the audio, they sound like they have a British accent." Disallowed content in audio output We run our existing moderation classifier over text transcriptions of audio prompts and generations, and block the output for certain high-severity categories. |
| 107 | GPT-4o-early GPT-4o-deployed should_refuse 0.83 0.98 should_comply 0.70 0.83 Disparate performance on voice inputs Risk Description: Models may perform differently with users speaking with different accents. | 107 | GPT‑4o‑early GPT‑4o‑deployed should_refuse 0.83 0.98 should_comply 0.70 0.83 Disparate performance on voice inputs Risk Description: Models may perform differently with users speaking with different accents. |
| 127 | GPT-4o-early GPT-4o-deployed Accuracy 0.60 0.84 Violative & disallowed content Risk Description: GPT‑4o may be prompted to output harmful content through audio that would be disallowed through text, such as audio speech output that gives instructions on how to carry out an illegal activity. | 127 | GPT‑4o‑early GPT‑4o‑deployed Accuracy 0.60 0.84 Violative & disallowed content Risk Description: GPT‑4o may be prompted to output harmful content through audio that would be disallowed through text, such as audio speech output that gives instructions on how to carry out an illegal activity. |
| 237 | GPT-4T (May 2024) GPT-4o MedQA USMLE 4 Options (0-shot) 0.78 0.89 MedQA USMLE 4 Options (5-shot) 0.81 0.89 MedQA USMLE 5 Options (0-shot) 0.75 0.86 MedQA USMLE 5 Options (5-shot) 0.78 0.87 MedQA Taiwan (0-shot) 0.82 0.91 MedQA Taiwan (5-shot) 0.86 0.91 MedQA Mainland China (0-shot) 0.72 0.84 MedQA Mainland China (5-shot) 0.78 0.86 MMLU Clinical Knowledge (0-shot) 0.85 0.92 MMLU Clinical Knowledge (5-shot) 0.87 0.92 MMLU Medical Genetics (0-shot) 0.93 0.96 MMLU Medical Genetics (5-shot) 0.95 0.95 MMLU Anatomy (0-shot) 0.79 0.89 MMLU Anatomy (5-shot) 0.85 0.89 MMLU Professional Medicine (0-shot) 0.92 0.94 MMLU Professional Medicine (5-shot) 0.92 0.94 MMLU College Biology (0-shot) 0.93 0.95 MMLU College Biology (5-shot) 0.95 0.95 MMLU College Medicine (0-shot) 0.74 0.84 MMLU College Medicine (5-shot) 0.80 0.89 MedMCQA Dev (0-shot) 0.70 0.77 MedMCQA Dev (5-shot) 0.72 0.79 Limitations While text-based evaluations appear promising, additional future work is needed to test whether text-audio transfer, which occurred for refusal behavior, extends to these evaluations. | 237 | GPT‑4T (May 2024) GPT‑4o MedQA USMLE 4 Options (0-shot) 0.78 0.89 MedQA USMLE 4 Options (5-shot) 0.81 0.89 MedQA USMLE 5 Options (0-shot) 0.75 0.86 MedQA USMLE 5 Options (5-shot) 0.78 0.87 MedQA Taiwan (0-shot) 0.82 0.91 MedQA Taiwan (5-shot) 0.86 0.91 MedQA Mainland China (0-shot) 0.72 0.84 MedQA Mainland China (5-shot) 0.78 0.86 MMLU Clinical Knowledge (0-shot) 0.85 0.92 MMLU Clinical Knowledge (5-shot) 0.87 0.92 MMLU Medical Genetics (0-shot) 0.93 0.96 MMLU Medical Genetics (5-shot) 0.95 0.95 MMLU Anatomy (0-shot) 0.79 0.89 MMLU Anatomy (5-shot) 0.85 0.89 MMLU Professional Medicine (0-shot) 0.92 0.94 MMLU Professional Medicine (5-shot) 0.92 0.94 MMLU College Biology (0-shot) 0.93 0.95 MMLU College Biology (5-shot) 0.95 0.95 MMLU College Medicine (0-shot) 0.74 0.84 MMLU College Medicine (5-shot) 0.80 0.89 MedMCQA Dev (0-shot) 0.70 0.77 MedMCQA Dev (5-shot) 0.72 0.79 Limitations While text-based evaluations appear promising, additional future work is needed to test whether text-audio transfer, which occurred for refusal behavior, extends to these evaluations. |
| 295 | Translated ARC-Easy (%, higher is better), 0-shot Model English (n=523) Amharic (n=518) Hausa (n=475) Northern Sotho (n=520) Swahili (n=520) Yoruba (n=520) GPT 3.5 Turbo 80.3 6.1 26.1 26.9 62.1 27.3 GPT-4o mini 93.9 42.7 58.5 37.4 76.9 43.8 GPT-4 89.7 27.4 28.8 30 83.5 31.7 GPT-4o 94.8 71.4 75.4 70 86.5 65.8 Translated TruthfulQA (%, higher is better), 0-shot Model English (n=809) Amharic (n=808) Hausa (n=808) Northern Sotho (n=809) Swahili (n=808) Yoruba (n=809) GPT 3.5 Turbo 53.6 26.1 29.1 29.3 40 28.3 GPT-4o mini 66.5 33.9 42.1 36.1 48.4 35.8 GPT-4 81.3 42.6 37.6 42.9 62 41.3 GPT-4o 81.4 55.4 59.2 59.1 64.4 51.1 Uhura (New Reading Comprehension Evaluation), 0-shot Model Amharic (n=77) Hausa (n=155) Yoruba (n=258) GPT 3.5 Turbo 22.1 32.3 28.3 GPT-4o mini 33.8 43.2 44.2 GPT-4 41.6 41.9 41..9 GPT-4o 44.2 59.4 60.5 Conclusion and next steps OpenAI has implemented various safety measurements and mitigations throughout the GPT‑4o development and deployment process. | 295 | Translated ARC-Easy (%, higher is better), 0-shot Model English (n=523) Amharic (n=518) Hausa (n=475) Northern Sotho (n=520) Swahili (n=520) Yoruba (n=520) GPT 3.5 Turbo 80.3 6.1 26.1 26.9 62.1 27.3 GPT‑4o mini 93.9 42.7 58.5 37.4 76.9 43.8 GPT‑4 89.7 27.4 28.8 30 83.5 31.7 GPT‑4o 94.8 71.4 75.4 70 86.5 65.8 Translated TruthfulQA (%, higher is better), 0-shot Model English (n=809) Amharic (n=808) Hausa (n=808) Northern Sotho (n=809) Swahili (n=808) Yoruba (n=809) GPT 3.5 Turbo 53.6 26.1 29.1 29.3 40 28.3 GPT‑4o mini 66.5 33.9 42.1 36.1 48.4 35.8 GPT‑4 81.3 42.6 37.6 42.9 62 41.3 GPT‑4o 81.4 55.4 59.2 59.1 64.4 51.1 Uhura (New Reading Comprehension Evaluation), 0-shot Model Amharic (n=77) Hausa (n=155) Yoruba (n=258) GPT 3.5 Turbo 22.1 32.3 28.3 GPT‑4o mini 33.8 43.2 44.2 GPT‑4 41.6 41.9 41..9 GPT‑4o 44.2 59.4 60.5 Conclusion and next steps OpenAI has implemented various safety measurements and mitigations throughout the GPT‑4o development and deployment process. |
| 303 | Current GPT-4o Text New GPT-4o–Text New GPT-4o–Audio not_unsafe 0.99 0.99 1.0 not_overrefuse 0.91 0.89 0.91 sexual_minors_not_unsafe 0.95 0.98 0.98 sexual_illegal_not_unsafe 0.97 0.98 0.99 extrimism_propoganda_not_unsafe 1.0 1.0 1.0 illicit_violent_not_unsafe 1.0 1.0 1.0 ilicit_non_violent_not_unsafe 0.99 0.97 1.0 self_harm_not_unsafe 1 1 1 B - Sample tasks from METR Evaluations Task Family Time for human with domain expertise Example prompt excerpts Converting JSON data from one structure to another 5-55 mins In /home/agent there are two JSON files. | 303 | Current GPT‑4o Text New GPT‑4o–Text New GPT‑4o–Audio not_unsafe 0.99 0.99 1.0 not_overrefuse 0.91 0.89 0.91 sexual_minors_not_unsafe 0.95 0.98 0.98 sexual_illegal_not_unsafe 0.97 0.98 0.99 extrimism_propoganda_not_unsafe 1.0 1.0 1.0 illicit_violent_not_unsafe 1.0 1.0 1.0 ilicit_non_violent_not_unsafe 0.99 0.97 1.0 self_harm_not_unsafe 1 1 1 B - Sample tasks from METR Evaluations Task Family Time for human with domain expertise Example prompt excerpts Converting JSON data from one structure to another 5-55 mins In /home/agent there are two JSON files. |
Follow unlimited companies, monitor the clauses that matter across every platform, and get the full institutional analysis on what each change obligates you to do.