Background
Investigations involve hypothesis generation and evaluation of observations in light of competing hypotheses. Evaluation assigns a value to observations in light of each hypothesis that can be fed into a decision process. As described in NIST OSAC Tech Pub 0002, the value produced by evaluation can be a measure of continuous nature, such as a real number between 0 and 1 (probability), or a real number between -1 and 1 (measure of distrust and trust). Alternatively, this value can a mark of a discrete nature, such as {A+, A, A-, B+, B, B-, C,D, F}... or a verbal scale.
Requirements
Requirement 1
State a Hypothesis and one or more alternative Hypotheses being considered. Selecting a single hypothesis without consideration of alternatives increases risk of confirmation bias.
Requirement 2
Evaluate observations under multiple hypotheses and assign a value (numeric, verbal, etc.) to the observations, not the hypothesis.
Requirement 3
Allow for different forms of evaluation, including verbal (more probable), numeric (0.9), pseudonumeric (C5.5)
Requirement 4
Represent who performed the evaluation, using which tools and methods.
Requirement 5
Group a set of hypotheses being considered
Risk / Benefit analysis
Benefits
The ability to generate and evaluate hypotheses is fundamental to cyber-investigations.
Risks
The analysis of competing hypotheses approach represented here are incompatible with conclusion scales (e.g., Identification, Inconclusive, and Exclusion), which typically only consider one hypothesis (problematic because of confirmation bias), are categorical (problematic because of lack of agreement and cliff-edge effect), and do not make a clear distinction between hypotheses evaluation (prosector's fallacy) and evaluation of observations in light of opposing hypotheses.
The evaluative approach represented here is incompatible with non-scientific domains (e.g., intelligence analysis) approaches that assign a likelihood of occurrence of an event or development, and the confidence in the basis for this judgment. Assigning a value to an analytic judgment evaluates the hypotheses not the observations (prosector's fallacy), increases risk of confirmation bias, and is not balanced with evaluation of opposing hypothesis.
Competencies demonstrated
Competency 1
Evaluate the observations given opposing hypotheses.
Competency Question 1.1
Are the [Observations] more probable given [Hypothesis 1] or [Hypothesis 2]?
Result 1.1
The [Observations] are more probable given [Hypothesis 1] rather than [Hypothesis 2].
Competency Question 1.2
What [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2]?
Result 1.2
The [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2].
Competency 2
Track provenance and ensure accountability for evaluation of observations given opposing hypotheses.
Competency Question 2.1
Who performed the evaluation of the observations given opposing hypotheses?
Result 2.1
Eoghan Casey performed the evaluation of the observations given opposing hypotheses.
Solution suggestion
Add Hypothesis (Assertion): A statement that is either true or false, the truth of which is uncertain
Add EvaluationResult (AnalyticResult): An opinion that is formed on the basis of observations in light of a given hypothesis.
Add HypothesisGroup (Collection): A collection of hypotheses being considered.
This illustrative example demonstrates Requirement 1:
https://github.com/casework/CASE-Examples/tree/ONT-434/examples/illustrations/inference
Background
Investigations involve hypothesis generation and evaluation of observations in light of competing hypotheses. Evaluation assigns a value to observations in light of each hypothesis that can be fed into a decision process. As described in NIST OSAC Tech Pub 0002, the value produced by evaluation can be a measure of continuous nature, such as a real number between 0 and 1 (probability), or a real number between -1 and 1 (measure of distrust and trust). Alternatively, this value can a mark of a discrete nature, such as {A+, A, A-, B+, B, B-, C,D, F}... or a verbal scale.
Requirements
Requirement 1
State a Hypothesis and one or more alternative Hypotheses being considered. Selecting a single hypothesis without consideration of alternatives increases risk of confirmation bias.
Requirement 2
Evaluate observations under multiple hypotheses and assign a value (numeric, verbal, etc.) to the observations, not the hypothesis.
Requirement 3
Allow for different forms of evaluation, including verbal (more probable), numeric (0.9), pseudonumeric (C5.5)
Requirement 4
Represent who performed the evaluation, using which tools and methods.
Requirement 5
Group a set of hypotheses being considered
Risk / Benefit analysis
Benefits
The ability to generate and evaluate hypotheses is fundamental to cyber-investigations.
Risks
The analysis of competing hypotheses approach represented here are incompatible with conclusion scales (e.g., Identification, Inconclusive, and Exclusion), which typically only consider one hypothesis (problematic because of confirmation bias), are categorical (problematic because of lack of agreement and cliff-edge effect), and do not make a clear distinction between hypotheses evaluation (prosector's fallacy) and evaluation of observations in light of opposing hypotheses.
The evaluative approach represented here is incompatible with non-scientific domains (e.g., intelligence analysis) approaches that assign a likelihood of occurrence of an event or development, and the confidence in the basis for this judgment. Assigning a value to an analytic judgment evaluates the hypotheses not the observations (prosector's fallacy), increases risk of confirmation bias, and is not balanced with evaluation of opposing hypothesis.
Competencies demonstrated
Competency 1
Evaluate the observations given opposing hypotheses.
Competency Question 1.1
Are the [Observations] more probable given [Hypothesis 1] or [Hypothesis 2]?
Result 1.1
The [Observations] are more probable given [Hypothesis 1] rather than [Hypothesis 2].
Competency Question 1.2
What [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2]?
Result 1.2
The [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2].
Competency 2
Track provenance and ensure accountability for evaluation of observations given opposing hypotheses.
Competency Question 2.1
Who performed the evaluation of the observations given opposing hypotheses?
Result 2.1
Eoghan Casey performed the evaluation of the observations given opposing hypotheses.
Solution suggestion
Add Hypothesis (Assertion): A statement that is either true or false, the truth of which is uncertain
Add EvaluationResult (AnalyticResult): An opinion that is formed on the basis of observations in light of a given hypothesis.
Add HypothesisGroup (Collection): A collection of hypotheses being considered.
This illustrative example demonstrates Requirement 1:
https://github.com/casework/CASE-Examples/tree/ONT-434/examples/illustrations/inference