Words of Estimative Probability, Analytic Confidences, and Structured Analytic Techniques
In line with Intelligence Community Directives 203 and 206, the products of the Center for Internet Security® (CIS®) Cyber Threat Intelligence (CTI) team leverage specific verbiage to express probability and confidence associated with analytical assessments. These products account for such uncertainties, particularly related to forecasts and predictions, using words of estimative probability (WEPs) and analytic confidences.
The chart below outlines the WEPs commonly used by CIS CTI in member engagements to communicate likelihoods and probabilities. Note: The word “High” is often exchanged for “Very.”
|
Almost no chance |
Very unlikely | Unlikely | Roughly even chance | Likely | Very likely | Almost certain(ly) |
| Remote | Highly improbably | Improbable (improbably) |
Roughly even odds | Probable (probably) |
Highly probable | Nearly certain |
| 1-5% | 5-20% | 20-45% | 45-55% | 55-80% | 80-95% | 95-99% |
Analytic Confidences
Disseminated products with a formal assessment also explicitly state analytic confidence based on the number, variety, and reliability of sources. The following confidence levels articulate the CIS CTI team’s assessment of the quality and quantity of source information supporting judgments within a given product.
High Confidence
High Confidence generally indicates that CIS CTI’s judgments are based on high-quality information from multiple sources, most or all of which are considered trustworthy, with minimal to no conflict among sources. High confidence in source information does not imply that the assessment is a fact or a certainty; there is always a chance that an assessment might be wrong.
Moderate Confidence
Moderate Confidence generally means that the information is credibly sourced and interpreted to be plausible but is not of sufficient quality or corroboration to warrant a higher level of confidence. For example, multiple sources may have opposing or alternative views, and while the CIS CTI team or partners may have evidence to support one assessment over an alternative, it may not be sufficient enough to claim high confidence.
Low Confidence
Low Confidence generally means that the source information’s credibility or plausibility is uncertain – that is, the source information is scant, questionable, fragmented, or poorly corroborated to the point where it is difficult to make solid analytic inferences. Low confidence could also indicate that the CIS CTI team has concerns with the reliability of the source data.
SAT Categories
In addition to WEPs and analytic confidences, the CIS CTI team regularly leverages Structured Analytic Techniques (SATs) to enrich, guide, and formalize U.S. State, Local, Tribal, and Territorial (SLTT)-focused threat assessments. SATs are methodical analytic procedures that analysts can apply to hone the accuracy and relevancy of assessments. The SATs are generally categorized under three umbrella techniques – Diagnostic, Contrarian, and Imaginative – but categorization may vary across organizations.
Diagnostic SATs
Diagnostic SATs focus on assessing what analysts can effectively determine based on the evidence to answer a series of analytic questions.
Key Assumptions Check (KAC)
Definition: This tool enables analysts to lay out and directly challenge their assumptions based on the available evidence.
Application: KAC is most useful in the early stages of an analytic assessment after the CIS CTI team has begun collecting evidence. When beginning research, the team needs to pore through a variety of resources spanning open-source, industry analysis, and partner submissions. Sorting through this context requires analysts to make determinations, often passively, about what information is relevant or reliable. The challenge is that bias inevitably influences humans' perceptions of the information they consume. By forcing the team to explicitly list those assumptions and identify which ones hold up to the available evidence, they ensure the substantiation for their assessments are rigorous and stress-tested.
Quality of Information Check
Definition: This technique requires analysts to vet the strength of the evidence informing their assessments.
Application: The CIS CTI team assesses a variety of sources, including claims from cyber threat actors (CTAs) themselves. For the team to ensure they only convey rigorously reviewed and vetted information, they must vet the sources. This includes factoring in the source's history of reliability, driving motivations, medium of communication, implications of their claims, as well as potential risks of relying on that information. Based on this analysis, the team will then provide an assessment in alignment with the standards we outline in the confidence section above.
Indicators or Signposts of Change
Definition: Indicators are tools that enable analysts to call out specific conditions suggesting a future scenario is more likely. SLTTs can use indicators to make preparatory decisions for that emerging scenario.
ApplicationThe job of the CIS CTI team when forecasting is to inform customers of the likely outcomes following an event. Tools like an Alternative Futures Analysis help develop a matrix covering the spectrum of potential scenarios. (See figure 1 for reference.)
For example, if the team identified that vendors began developing affordable defensive tooling proven to effective against artificial intelligence (AI)/machine learning (ML)-derived threats, this would indicate the trend is shifting towards a scenario where the security community is likely to be generally prepared for the emerging threats with the proper defenses in place. This would then provide decision-makers with evidence that if they implement these defensive measures, they can effectively defend against the emerging threats.
Analysis of Competing Hypothesis (ACH)
Definition: ACH enables analysts to score and rank their hypotheses according to the evidence.
Application: When an incident impacts SLTTs, the CIS CTI team may have limited information to inform analysis. If context is limited but SLTTs require timely analysis, then the team can perform an ACH. This technique requires analysts to form hypotheses based on likely explanations (e.g., Threat Actor A conducted a ransomware attack and leaked the data as part of a financially motivated campaign) and score the hypotheses against the available evidence, including when there's a lack of evidence that analysts would expect to see. This enables the team to eliminate less likely hypotheses and drive their assessment according to the available evidence.
Contrarian SATs
Contrarian SATs force analysts to assume opposing views to challenge and poke holes in their assessments and assumptions.
Devil's Advocacy
Definition: Devil's Advocacy can be a powerful tool for analysts to explore alternative explanations by assuming their analysis turned out to be wrong and why that might have been the case.
Application: It is critical that the CIS CTI team rigorously evaluates the available evidence to formulate assessments. Despite that commitment, assessments can still miss the mark due to a failure to consider alternative scenarios, a failure to account for information gaps, and a failure to check analysts’ own biases. By flipping the incentives, analysts can poke holes in their own assessments and identify potential errors before publishing. This enables them to work backwards and adjust findings to account for those gaps or errors.
Team A/Team B
Definition: Analysts can engage in a Team A/Team B exercise to break through gridlock in the form of a structured debate.
Application: It's not uncommon that analysts hold opposing views about an analytic question or its implications. Using this technique forces the team to see both the merits and shortcomings of each perspective and evaluate their findings accordingly. This exercise can inform new information requirements as well as follow-up steps, such as an ACH, to further evaluate findings.
High-Impact/Low-Probability Analysis (HILP)
Definition: A High-Impact/Low-Probability Analysis forces analysts to identify major shifts in the current operating environment and assume they'll lead to a significant change.
Application: Because humans often perceive the world based on trends and immediately available context, analysts often struggle to accurately recognize the long-term impact of recent shifts. It can be very difficult for anyone to anticipate these sorts of events and when they occur. Analysts and decision-makers alike are often left unprepared to handle them. This tool can be equally useful for executive-level decision makers aiming to assess a complex set of circumstances like an organization’s plan to bring an AI/ML-derived cybersecurity tool to market.
“What If?” Analysis
Definition: Like the HILP exercise, this tool focuses on potentially cataclysmic outcomes that analysts and decision-makers may not have anticipated.
Application: Instead of focusing on an emerging trend and how that trend could lead to an eventual major impact like in the HILP technique, analysts start by identifying a hypothetical negative outcome and work backwards to identify how it could have come about. This tool is particularly useful for planning a new initiative or strategy. This then enables them to identify specific indicators suggesting that outcome is more likely as well as develop strategies to better avoid missteps that could lead to that negative outcome.
Imaginative SATs
Imaginative SATs aid analysts in coming up with alternative and even out-there perspectives to stress-test their assessments and account for potential outcomes they haven’t yet considered.
Brainstorming
Definition: Brainstorming enables analysts to begin an assessment by identifying a list of potential variables, key drivers, alternative scenarios, and solutions that they can use to generally steer their analysis.
Application: Whenever the CIS CTI team begins an analysis, they must confront a potential endless list of possibilities and approaches for tackling an issue. Brainstorming is a fundamental analytic technique that can aid in identifying key questions and pathways to drive analytic assessments. It also serves well as a precursor to a KAC.
Outside-in Thinking
Definition: This technique forces analysts looking at a complex issue like a geopolitical conflict to identify key conditions like global, economic, technological, or social forces that are likely to influence the outcome.
Application: This is a particularly useful framing tool in the early stages of an assessment, as it ensures that analysts’ understanding of the conflict will be shaped by these easily overlooked but critical conditions before diving into the issue.
Red Hat Analysis
Definition: Since analysts cannot enter the minds of the enemy, this tool forces them to imagine they’re in the adversary’s position and envision how the adversary would likely see the world, what their likely concerns would be, and how they would likely address the issue at hand.
Application: This technique is designed to combat cognitive biases like "mirror imaging" where analysts fall into the error of assuming an adversary or target of analysis is likely to think and perceive an issue as they would. This technique can also be useful to trade places with the customers to better tailor analysis to their needs and perspectives.
Alternative Futures Analysis
Definition: This technique enables analysts to develop a forecast by identifying a spectrum of potential critical future scenarios and ranking them based on likelihood and impact.
Application: Anytime the team is performing anticipatory analysis (aiming to assess an event or outcome that has yet to happen), they're inherently at an information disadvantage simply because it hasn’t yet happened. To fill this gap, this technique forces analysts to first identify key drivers likely to determine the eventual outcome. The analysts append these drivers on an axis and develop four mutually exclusive but comprehensive scenarios, break them down in detail, assess their likelihood, and then identify a list of indicators likely to signal each scenario is coming to fruition. (See figure 1 for reference.)
Analysis Components in Practice
All three leveraged analysis components discussed above – WEPs, Analytic Confidences, and SATs – can be found within the body of CIS CTI team's products or summarized near the end of the product under the “Analytic Confidence” header. The CIS CTI team can be reached at [email protected]
