ms_calcu
========

.. py:module:: ms_calcu

.. autoapi-nested-parse::

   Uncertainty-Aware Mutation Score (UA-MS) calculation.

   Consumes the per-repetition detection outputs (``prediction_{t}.json``) produced
   by running a model under MC-Dropout / MC-DropBlock mutation (see
   ``docs/uq4ma.md`` for how those mutants are generated) and computes the
   Uncertainty-Aware Mutation Score between an original model and each of its
   mutants, at both the image level and the object level.



Attributes
----------

.. autoapisummary::

   ms_calcu.uq


Functions
---------

.. autoapisummary::

   ms_calcu.process_match_metrics
   ms_calcu.process_missing_set
   ms_calcu.process_ghost_set_dbscan
   ms_calcu.ms_per_test_case_mutant
   ms_calcu.calcu_mutation_score
   ms_calcu.ms_calcu_exec


Module Contents
---------------

.. py:data:: uq

.. py:function:: process_match_metrics(repetitions)

   Calculates metrics for Spatial (Box), Classification (Label), and Probabilistic (Entropy) stability.


.. py:function:: process_missing_set(repetitions)

   Analyzes the 'Missing Set' to determine if misses are due to
   instability (flickering) or blindness, weighted by original confidence.


.. py:function:: process_ghost_set_dbscan(repetitions, iou_threshold=0.5)

   Clusters ghost detections using DBSCAN with (1 - IoU) distance metric.


.. py:function:: ms_per_test_case_mutant(test_case, org_model, mutation_operator, mutation_rate, case_study, T=10)

   Computes the UA-MS components (match/miss/ghost metrics, image- and
   object-level kill rates) for a single test case, comparing the original
   model's predictions against one mutant (identified by ``mutation_rate``).

   Expects ``T`` repetitions of ``prediction_{t}.json`` files under:
       ``{case_study}/experiment_results_{mutation_operator}/{org_model}/dataset/orig/``
   for both the original run (rate/size ``0``) and the mutant run.


.. py:function:: calcu_mutation_score(test_set, org_model, mutation_operator, case_study_p, case_study_n, save_folder, mutation_rates=None)

   Computes the UA-MS for every mutant of ``mutation_operator`` (MC-Dropout
   dropout rates, or MC-DropBlock (dropout_rate, block_size) pairs) across a
   whole test set, and writes one CSV per mutant under
   ``{save_folder}/{case_study_n}/{org_model}/{mutation_operator}/``.


.. py:function:: ms_calcu_exec(case_study_name, case_study, save_folder)

   Runs :func:`calcu_mutation_score` for both the MC-DropBlock and MC-Dropout
   mutants of every model registered under ``case_study[case_study_name]``.

   ``case_study`` is a dict of the form::

       {
           "<case_study_name>": {
               "dataset": "<path to a folder with common_images_{model}_filtered.pkl per model>",
               "raw_result": "<path to the experiment_results_{mutation_operator} folders>",
               "models": ["<model_1>", ...],
           },
           ...
       }