Prohibited Use: Non-Consensual Synthetic Media and Deepfakes

What it is

The policy prohibits using Cohere's AI to create fake intimate images of real people without their permission, or to produce other synthetic media depicting real individuals without consent.

Why it matters (compliance & governance perspective)

This provision addresses a category of AI-generated harm increasingly regulated at the state and national level, and operators building image or video generation applications on Cohere's API must implement controls to prevent this use regardless of user requests.

Consumer impact (what this means for users)

Users cannot use Cohere-powered applications to generate fake intimate or deceptive synthetic media of real individuals without their consent, and operators are responsible for ensuring their platforms do not permit this use.

Institutional analysis (Compliance & governance intelligence)

REGULATORY LANDSCAPE: This provision engages an expanding set of state laws in the US prohibiting non-consensual deepfake intimate imagery, including laws in California, Texas, Virginia, Georgia, and others. Federal legislative proposals exist but no comprehensive US federal law was in force at the time of this analysis. The EU AI Act and proposed EU regulations on synthetic media may also apply. The FTC has indicated interest in AI-generated deceptive content under its unfair or deceptive practices authority. GOVERNANCE EXPOSURE: Medium to High depending on the operator's use case. Operators offering image or video generation capabilities face heightened exposure given the technical ease of generating synthetic media. Platforms with large consumer user bases are particularly vulnerable to misuse. JURISDICTION FLAGS: US state laws vary significantly in their definitions, covered persons, and penalties. California, Texas, and Virginia have enacted specific non-consensual deepfake statutes. EU member states implementing the EU AI Act may impose additional obligations on AI system providers. Organizations serving global user bases face multi-jurisdictional compliance obligations. CONTRACT AND VENDOR IMPLICATIONS: Operators in media, entertainment, and content creation sectors should assess whether their applications could foreseeably be used to generate non-consensual synthetic media and implement technical and procedural controls accordingly. Vendor agreements should include representations that downstream use cases comply with applicable synthetic media laws. COMPLIANCE CONSIDERATIONS: Compliance teams should monitor evolving state and federal legislation on synthetic media, assess whether existing content moderation systems adequately detect and prevent non-consensual deepfake generation, and ensure user-facing terms of service clearly prohibit this use.

Applicable agencies

Provision details

Other risks in this policy

• Prohibited Use: CSAM and Sexual Content Involving Minors high
• Prohibited Use: Weapons of Mass Destruction and CBRN Content high
• Prohibited Use: Cyberweapons and Critical Infrastructure Attacks high
• Prohibited Use: Election Interference and Political Disinformation medium
• Operator Responsibility for Downstream Use high
• Restriction on Autonomous High-Stakes Decision-Making Without Human Oversight high