ms_calcu

ms_calcu#

Uncertainty-Aware Mutation Score (UA-MS) calculation.

Consumes the per-repetition detection outputs (prediction_{t}.json) produced by running a model under MC-Dropout / MC-DropBlock mutation (see docs/uq4ma.md for how those mutants are generated) and computes the Uncertainty-Aware Mutation Score between an original model and each of its mutants, at both the image level and the object level.

Attributes#

uq

Functions#

process_match_metrics(repetitions)

Calculates metrics for Spatial (Box), Classification (Label), and Probabilistic (Entropy) stability.

process_missing_set(repetitions)

Analyzes the 'Missing Set' to determine if misses are due to

process_ghost_set_dbscan(repetitions[, iou_threshold])

Clusters ghost detections using DBSCAN with (1 - IoU) distance metric.

ms_per_test_case_mutant(test_case, org_model, ...[, T])

Computes the UA-MS components (match/miss/ghost metrics, image- and

calcu_mutation_score(test_set, org_model, ...[, ...])

Computes the UA-MS for every mutant of mutation_operator (MC-Dropout

ms_calcu_exec(case_study_name, case_study, save_folder)

Runs calcu_mutation_score() for both the MC-DropBlock and MC-Dropout

Module Contents#

ms_calcu.uq#
ms_calcu.process_match_metrics(repetitions)#

Calculates metrics for Spatial (Box), Classification (Label), and Probabilistic (Entropy) stability.

ms_calcu.process_missing_set(repetitions)#

Analyzes the ‘Missing Set’ to determine if misses are due to instability (flickering) or blindness, weighted by original confidence.

ms_calcu.process_ghost_set_dbscan(repetitions, iou_threshold=0.5)#

Clusters ghost detections using DBSCAN with (1 - IoU) distance metric.

ms_calcu.ms_per_test_case_mutant(test_case, org_model, mutation_operator, mutation_rate, case_study, T=10)#

Computes the UA-MS components (match/miss/ghost metrics, image- and object-level kill rates) for a single test case, comparing the original model’s predictions against one mutant (identified by mutation_rate).

Expects T repetitions of prediction_{t}.json files under:

{case_study}/experiment_results_{mutation_operator}/{org_model}/dataset/orig/

for both the original run (rate/size 0) and the mutant run.

ms_calcu.calcu_mutation_score(test_set, org_model, mutation_operator, case_study_p, case_study_n, save_folder, mutation_rates=None)#

Computes the UA-MS for every mutant of mutation_operator (MC-Dropout dropout rates, or MC-DropBlock (dropout_rate, block_size) pairs) across a whole test set, and writes one CSV per mutant under {save_folder}/{case_study_n}/{org_model}/{mutation_operator}/.

ms_calcu.ms_calcu_exec(case_study_name, case_study, save_folder)#

Runs calcu_mutation_score() for both the MC-DropBlock and MC-Dropout mutants of every model registered under case_study[case_study_name].

case_study is a dict of the form:

{
    "<case_study_name>": {
        "dataset": "<path to a folder with common_images_{model}_filtered.pkl per model>",
        "raw_result": "<path to the experiment_results_{mutation_operator} folders>",
        "models": ["<model_1>", ...],
    },
    ...
}
Contents