General Usage

General Usage#

This page describes how to use the metrics provided by deepluq. Examples below are adapted from the test suite in tests/deepluq/, which is the best place to see each metric exercised with concrete inputs and expected output shapes/ranges.

Follow the installation and setup instructions in the Installation Page.

Main functionality:

deepluq.metrics_dl.DLMetrics: UQ metrics for general deep learning models (classification and object detection), see test_metrics.py.
deepluq.metrics_vla: UQ metrics for Vision-Language-Action (VLA) models (TokenMetrics, OutputMetrics), see test_token_based_metrics.py and test_output_based_metrics.py.
deepluq.utils: clustering and box-fusion helpers used to prepare Monte-Carlo Dropout detections before UQ metrics are computed, see test_clustering.py and test_iou.py.
deepluq.metrics_mut: Uncertainty-Aware Mutation Score (UA-MS) computation for MC-Dropout / MC-DropBlock mutants of object detection models.

Deep Learning UQ Metrics (`DLMetrics`)#

DLMetrics computes classification-uncertainty metrics from repeated stochastic predictions (e.g. multiple MC-Dropout forward passes), and geometric metrics from repeated bounding-box predictions.

from deepluq.metrics_dl import DLMetrics

uq = DLMetrics()

Classification uncertainty metrics#

Variation Ratio (VR): proportion of predictions that disagree with the modal (most frequent) class across N stochastic forward passes. Each row in events is a per-pass probability distribution over classes.

events = [[0.2, 0.8], [0.1, 0.9], [0.6, 0.4]]
vr = uq.cal_vr(events)  # -> float in [0, 1]

Shannon Entropy: entropy of a single probability distribution, using natural log by default converted to the given base (default base 2).

entropy = uq.calcu_entropy([0.5, 0.5])  # -> 1.0 for a uniform 2-class distribution

Mutual Information (MI): combines the entropy of the mean prediction across passes with the average per-pass entropy, giving a measure of predictive uncertainty that accounts for disagreement between passes.

import numpy as np

events = np.array([[0.7, 0.3], [0.6, 0.4], [0.5, 0.5]])
mi = uq.calcu_mi(events)  # -> float

Total Variance (TV)#

Total variance: trace of the covariance matrix of repeated predictions, either for bounding-box corners ("bounding_box") or box center points ("center_point"). An invalid tag raises ValueError.

import numpy as np

matrix = np.array([[1, 2], [3, 4], [5, 6]])
tv_center = uq.calcu_tv(matrix, "center_point")
tv_box = uq.calcu_tv(matrix, "bounding_box")

Mutual Information between three variables#

calcu_mutual_information computes MI between three discrete random variables (e.g. class label, mutation type, and detection outcome across mutation-based robustness testing).

import numpy as np

X = np.array([0, 0, 1, 1])
Y = np.array([0, 1, 0, 1])
Z = np.array([1, 1, 0, 0])
mi = uq.calcu_mutual_information(X, Y, Z)  # -> float >= 0

Geometric uncertainty: prediction surface#

Prediction surface: sum of convex-hull areas formed by each corner (top-left, top-right, bottom-left, bottom-right) of repeated bounding-box predictions for the same object. Larger surfaces indicate greater localization uncertainty. At least 3 non-collinear boxes are required per corner set; otherwise the metric returns -1.

boxes = [
    [0, 0, 1, 1],
    [1, 0, 2, 1],
    [0, 1, 1, 2],
    [1, 1, 2, 2],
]
surface = uq.calcu_prediction_surface(boxes)  # -> float >= 0

# Too few points to form a hull:
uq.calcu_prediction_surface([[0, 0, 1, 1]])  # -> -1

VLA Metrics (`deepluq.metrics_vla`)#

Token-based metrics (`TokenMetrics`)#

TokenMetrics computes token-level uncertainty metrics directly from a VLA model’s output logits (shape (batch_size, num_classes)).

import torch
from deepluq.metrics_vla import TokenMetrics

tm = TokenMetrics()

logits = torch.tensor([
    [1.0, 2.0, 0.5, -0.5],
    [2.0, 1.0, -1.0, 0.0],
    [0.0, 0.0, 0.0, 0.0],
])

entropy, max_prob, pcs, deepgini = tm.calculate_metrics(logits)

calculate_metrics returns four lists (one value per sample):

Shannon entropy of the softmax distribution.
Max token probability: confidence of the top predicted token.
PCS (Prediction Confidence Score): gap between the top-1 and top-2 probabilities.
DeepGini: 1 - sum(p^2).

For a confident (peaked) prediction, entropy/DeepGini are close to 0 and max-probability/PCS are close to 1:

logits = torch.tensor([[10.0, -10.0]])
entropy, max_prob, pcs, deepgini = tm.calculate_metrics(logits)
# entropy[0] < 0.1, max_prob[0] > 0.9, pcs[0] > 0.9, deepgini[0] < 0.1

compute_norm_inv_token_metrics returns the same four metrics normalized to [0, 1] and inverted where necessary, so that higher values always mean higher uncertainty:

logits = torch.tensor([[1.0, 2.0, 0.5], [2.0, 1.0, -1.0]])
entropy_norm, max_prob_inv, pcs_inv, deepgini_norm = tm.compute_norm_inv_token_metrics(logits)

Internally, TokenMetrics accumulates results in shannon_entropy_list, token_prob/token_prob_inv, pcs/pcs_inv, and deepgini. Call clear() to reset the accumulated state between batches:

tm.calculate_metrics(torch.tensor([[1.0, 2.0, 3.0]]))
tm.clear()
assert tm.shannon_entropy_list == []

Output-based instability and variability metrics (`OutputMetrics`)#

OutputMetrics quantifies how unstable a VLA model’s predicted actions or tool-center-point (TCP) poses, and how much variability there is across an ensemble of models given the same observation.

from deepluq.metrics_vla import OutputMetrics

om = OutputMetrics()

Action-based instability: each action is a dict with "world_vector", "rot_axangle", and "gripper" keys. Instability is the mean absolute successive difference (position = 1st order, velocity = 2nd order, acceleration = 3rd order) across the rollout, returned per action dimension.

actions = [
    {"world_vector": [1, 2], "rot_axangle": [0, 0, 1], "gripper": [0.5]}
    for _ in range(5)
]
position_instability = om.compute_position_instability(actions)      # requires >= 2 steps
velocity_instability = om.compute_velocity_instability(actions)      # requires >= 3 steps
acceleration_instability = om.compute_acceleration_instability(actions)  # requires >= 4 steps

TCP instability: same idea, but computed from a list of [x, y, z, ...] poses (only the first three coordinates are used).

poses = [[i, i + 1, i + 2] for i in range(10)]

tcp_position = om.compute_TCP_position_instability(poses)
tcp_velocity = om.compute_TCP_velocity_instability(poses)
tcp_acceleration = om.compute_TCP_acceleration_instability(poses)

# Jerk instability via numerical gradients (one value per time step):
jerk = om.compute_TCP_jerk_instability_gradient(poses)

Execution variability: standard deviation of actions produced by an ensemble of models (or repeated stochastic rollouts of the same model) given the same observation, used to quantify epistemic uncertainty in action selection.

variability = om.compute_execution_variability(
    variability_models=[model_1, model_2, model_3],
    image=image,
    action_space=action_space,
    instruction=instruction,
    obs=obs,               # dict with obs["agent"]["eef_pos"]
    model_name="pi0",      # selects the model.step(...) calling convention
)

Detection Utilities (`deepluq.utils`)#

These helpers support MC-Dropout–style UQ pipelines for object detection, where a single object may receive several overlapping predictions across passes that need to be clustered/fused before UQ metrics (e.g. calcu_tv, calcu_prediction_surface) are computed on them.

Weighted Boxes Fusion clustering#

wbf_clustering groups raw per-pass detections into clusters using Weighted Boxes Fusion (WBF), and returns, for each cluster, both the original member detections and the fused/merged detection.

from deepluq.utils import wbf_clustering

predictions = {
    "pred_0": {"box": [822.0, 301.6, 902.1, 340.7], "box_n": [0.642, 0.419, 0.704, 0.473],
               "label": 0, "score": 0.932, "logit": [0.932, 1.107e-08, 0.0002]},
    "pred_1": {"box": [865.9, 309.1, 892.9, 332.6], "box_n": [0.676, 0.429, 0.697, 0.462],
               "label": 2, "score": 0.877, "logit": [0.0005, 7.545e-06, 0.877]},
    "pred_2": {"box": [822.0, 301.6, 902.1, 340.7], "box_n": [0.642, 0.419, 0.704, 0.473],
               "label": 0, "score": 0.900, "logit": [0.900, 2.107e-08, 0.0003]},
}

clusters = wbf_clustering(predictions, iou_thr=0.5, skip_box_thr=0.01)

Each entry in the returned dict (keyed cluster_0, cluster_1, …) contains the member box/box_n/score/label/logit lists plus a detection entry with the fused box, score, label, and averaged logit.

compute_iou(box1, box2) is the plain IoU helper (normalized [x1, y1, x2, y2] boxes) used internally by wbf_clustering to match original boxes back to their fused cluster.

Density-based clustering#

For scenarios where WBF is not suitable, deepluq.utils also provides density-based clustering on box corner points via DBSCANCluster (HDBSCAN-backed) and the lower-level cluster(mc_locations) function, which clusters MC-Dropout box predictions with DBSCAN and reports the convex-hull surface per cluster.

from deepluq.utils import DBSCANCluster

# x: array of shape (N, >=4), e.g. [x1, y1, x2, y2, center_x, center_y]
clustering = DBSCANCluster(x)
clustered_preds = clustering.cluster_preds(preds)

get_kdist_plot(X, k) is a diagnostic helper that plots the sorted k-nearest-neighbor distances for a point set, useful for choosing a DBSCAN eps value.

Intersection over Union (object detection quality)#

For evaluating detection accuracy alongside UQ metrics, this project relies on torchmetrics.detection rather than reimplementing IoU-based detection metrics. test_iou.py shows the expected usage pattern with preds/targets dictionaries of boxes, scores, and labels:

import torch
from torchmetrics.detection import IntersectionOverUnion
from torchmetrics.functional.detection import intersection_over_union

metric = IntersectionOverUnion(class_metrics=True, respect_labels=True)
result = metric(preds, targets)

per_box_iou = intersection_over_union(preds[0]["boxes"], targets[0]["boxes"], aggregate=False)

Uncertainty-Aware Mutation Analysis (`deepluq.metrics_mut`)#

deepluq.metrics_mut computes the Uncertainty-Aware Mutation Score (UA-MS) for an object detection model: it compares the original model’s predictions against predictions from MC-Dropout / MC-DropBlock mutants, and scores how much each mutant’s outputs and predictive uncertainty diverge from the original. Unlike the metrics above, this module reads prediction files from disk (the output of running a model repeatedly under mutation) rather than working on in-memory arrays, so see Uncertainty-Aware Mutation Analysis for the required file layout and the full concepts (matches/misses/ghosts, Img-MS, Obj-MS, MS_iou, UA-MS). The entry point for scoring one test case against one mutant is:

from pathlib import Path
from deepluq.metrics_mut.ms_calcu import ms_per_test_case_mutant

iskill_miss, iskill_ghost, iskill_miss_ghost, ms_obj_level, \
    match_metrics, miss_metrics, ghost_metrics = ms_per_test_case_mutant(
        test_case=Path("image_0001"),
        org_model="fasterrcnn_resnet50_fpn",
        mutation_operator="mc_dropout",   # or "mc_dropblock"
        mutation_rate=0.3,                # or (dropout_rate, block_size) for mc_dropblock
        case_study="/path/to/experiment_results_root",
        T=10,
    )

calcu_mutation_score scores a whole test set against every mutant of an operator and writes one CSV per mutant, and ms_calcu_exec runs that for every model in a case study.

Running the tests#

The examples above mirror the test suite. To run all metric tests locally:

pytest tests/deepluq

General Usage

Contents