Metrics

MetricsManager

class torchok.metrics.metrics_manager.MetricsManager(params: List[MetricParams])

Bases: Module

Manages all metrics for a Task.

__init__(params: List[MetricParams])

Initialize MetricManager.

Parameters: params – Metric parameters.

update(phase: Phase, dataloader_idx: int = 0, **kwargs)

Update states of all metrics on phase loop.

MetricsManager update method use only update method of metrics. Because metric forward method increases computation time (see MetricWithUtils forward method for more information).

Parameters

phase – Phase Enum.
dataloader_idx – index of the dataloader

static is_number(num: Any) → bool

on_epoch_end(phase: Phase) → Dict[str, Tensor]

Summarize epoch values and return log.

Parameters: phase – Run metric phase.
Returns: Logging dictionary, where key is <phase>/<metric_name> and value is metric value for a given phase.
Return type: log
Raises: ValueError – If metric.compute() returns not numerical value.

training: bool

class torchok.metrics.metrics_manager.MetricWithUtils(metric: Metric, mapping: Dict[str, str], log_name: str, dataloader_idx: int)

Bases: Module

Union class for metric and metric utils parameters.

__init__(metric: Metric, mapping: Dict[str, str], log_name: str, dataloader_idx: int)

Initialize MetricWithUtils.

Parameters

metric – Metric written with TorchMetrics.
mapping – Dictionary for mapping Metric forward input keys with Task output dictionary keys.
log_name – The metric name used in logs.
dataloader_idx – Dataloader index on which metrics are calculated.

map_arguments(task_output: Dict[str, Any]) → Dict[str, Any]

Map arguments between metric target_fields and task output dictionary.

Parameters

mapping – Dictionary for mapping Metric forward input keys with Task output dictionary keys.
task_output – Output after task forward pass.

Returns

Metric input dictionary like **kwargs for metric forward pass.

Return type

metric_input

Raises

ValueError – If not found mapping_source in task_output keys.

update(dataloader_idx: int = 0, **kwargs)

Update metric states if current dataloader_idx equal self.dataloader_idx.

Add *args and **kwargs (usually it is batch) to current state.

Parameters: dataloader_idx – Current dataloader index.

compute(): Compute metric on the whole current state.

reset(): Reset metric states.

training: bool

General

Everything from TorchMetrics under regression or image metrics

Classification Metrics

Everything from TorchMetrics under classification metrics

Representation Metrics

class torchok.metrics.representation_ranx.PrecisionAtKMeter(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

Bases: RanxBasedMeter

__init__(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_ranx.RecallAtKMeter(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

Bases: RanxBasedMeter

__init__(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_ranx.MeanAveragePrecisionAtKMeter(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

Bases: RanxBasedMeter

__init__(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_ranx.NDCGAtKMeter(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

Bases: RanxBasedMeter

__init__(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_ranx.HitAtKMeter(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

Bases: RanxBasedMeter

__init__(dataset_type: str, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, group_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalFallOutMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalHitRateMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalMAPMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalMRRMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalNormalizedDCGMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalPrecisionMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalPrecisionRecallCurveMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalRPrecisionMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

class torchok.metrics.representation_torchmetrics.RetrievalRecallMeter(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Bases: TorchMetricBaseMetr

__init__(dataset_type: str, metric_params: Optional[Dict[str, Any]] = None, exact_index: bool = True, metric_distance: str = 'IP', k: Optional[int] = None, search_batch_size: Optional[int] = None, normalize_vectors: bool = False, target_averaging: bool = False, k_as_target_len: bool = False, use_batching_search: bool = True, group_averaging: bool = False, raise_empty_query: bool = True, **kwargs)

Metrics

MetricsManager

General

Classification Metrics

Representation Metrics

Detection Metrics