The Dirty Dozen vs the Magnificent Seven

Trustworthy and responsible Generative AI (GenAI) is tough — full stop. Agreeing on what it is, or more importantly what it isn't, is also not easy. Perhaps that is the root of all the confusion. Without discussing the merits of any one stakeholder's position, perhaps we can pick one definition and then compare that against real-world mission statements and service level agreements (SLAs), warranties and guarantees.

The Dirty Dozen: Threat catalogs and controls 

I have become fond of calling the 12 risk categories associated with GenAI from NIST AI 600-1 "the Dirty Dozen." Distilled down to its essence, the document describes in detail how human beneficiaries could be harmed if a GenAI system fails. It has become my lens of choice when assessing these systems.
 
The 12 listed out:

1. CBRN Information
2. Confabulation
3. Dangerous or Violent Recommendations
4. Data Privacy
5. Environmental
6. Human-AI Configuration
7. Information Integrity
8. Information Security
9. Intellectual Property
10. Obscene, Degrading and/or Abusive Content
11. Harmful Bias or Homogenization
12. Value Chain and Component Integration

Humans love stories. Armed with the Dirty Dozen, I can have impactful and productive conversations with various stakeholders when discussing curated threat catalogs and control affinities. This approach has proven to be very effective when communicating complex concepts like AI hallucinations (i.e. confabulation) to the people responsible for securing these systems. Further, it allows me to be very prescriptive when discussing reasonable ways to address residual risk with compensating controls.
 
A curated threat catalog is simply a list of bad things that have happened or could happen to an organization that would cause harm to its stakeholders. Historically, organizations have focused more on risk management than threat catalogs. However, from a storytelling perspective, people seem to gravitate toward the threats regardless of the likelihood that bad things could happen. A proper threat catalog distills the world of threats into "stories" (aka threat scenarios) of the most relevant threats to your organization and stakeholders. What's in your threat catalog?
 
System confidence is a combination of trust and control. In the absence of trust, control is all you have. By assessing specific threat catalog items against the harm they could cause, we can develop "structured choice" by supplying controls that can most effectively address the potential harm. 
 
Once an organization decides to address its threat catalog items, it must actually choose the controls it will use to address residual risk. [Residual risk is the difference between the organization's current risk profile and the risk profile the organization wants.] Then, the organization can leverage its understanding of controls to choose the best measures to mitigate the possibility or impact of bad outcomes. This allows organizations to "treat" residual risk.
 
When we put it all together, these are the types of informed conversations I can now have:
 
Client: "We want to use GenAI to do something cool. But we want to make sure our system doesn't tell people to hurt themselves or others (bad things). We want to make sure that our system does not discriminate, exclude or insult its users (our stakeholders). We want to make sure we are good stewards of the world's limited resources (see Hammers & Nails). We also want it to be cost effective, safe, secure and easy to operate." [No tall order here ;)]
 
Me: "It sounds like you want to implement a new productivity tool and have a holistic view on trustworthy and responsible AI. Assuming you already have a mature governance foundation in place, you should start by validating your business case, agreeing to a list of bad things you want to protect against and putting controls in place that will provide a high degree of certainty in how it is operated."
 
Client: "Yeah, that sounds about right."
 
We now have a reasonable starting point and can move on to control selection. It's beyond the scope of this blog to talk about all the types of controls available to organizations. Suffice it to say, one size does not fit all and there are many controls that can be used to provide the required system confidence. Much like threat catalogs, organizations should consider building their own curated control catalogs. These catalogs contain lists of controls that currently exist in the organization and some insight into their cost and maturity. 
 

The Magnificent Seven: The market-dominating tech companies 

One set of controls that is NOT often discussed but should be considered is service level agreements (SLAs), warranties and guarantees. These controls attempt to boost system confidence via commercial remediates and assertions. This is where it gets interesting.
 
Besides being a most excellent Western, The Magnificent Seven is also what Bank of America analyst Michael Hartnett calls the market-dominating tech companies. I wondered what these leaders in technological change and influence consider trustworthy and responsible. See for yourself: 
 
1.     Alphabet
2.     Amazon
3.     Apple
4.     Meta
5.     Microsoft
6.     NVIDIA
7.     TESLA
 
As the market matures, expect this type of review to become more important. We will explore the type of due diligence organizations can perform in future blogs because…AI Matters!