consnet.api.evaluation

consnet.api.evaluation.hico_det_eval(blob, anno, split='test', mode=['def', 'ko'], zero_shot=None, logger=None)[source]

Perform standard evaluation on HICO-DET dataset using mean average precision (mAP) as introduced in [1].

Parameters:

blob (list[Tensor[N, 10]]) – Human-object pairs and their detection scores to be evaluated. The length of the list should be 600 and each item should be an N * 10 tensor in (batch_id, x1, y1, x2, y2, ..., score) format.
anno (torch.Tensor or str) – The annotations object or path to the anno_bbox.mat file.
split (str, optional) – The dataset split to be evaluated. Expected values are 'train' and 'test'.
mode (list[str] or str, optional) – Mode of evaluation. Expected values are 'def', 'ko' or a list containing these terms, denoting the default mode and known-object mode introduced in [1].
zero_shot (dict or None, optional) –
Configurations for zero-shot settings. It should contain the following fields:
- type (str): Expected values include 'uc', 'ub' and 'ua', representing unseen action-object combination, unseen object and unseen action scenarios introduced in [2].
- id (int, optional): Only valid when type='uc'. Expected values are in the range of 0 ~ 4, indicating the 5 groups of unseen action-object combination settings in [2, 3].
logger (logging.Logger or str or None, optional) – The potential logger or name of the logger to be used.

Returns:

Evaluation results including mean average precision (mAP) and mean recall (mRec) values of multiple dataset splits under different evaluation modes.

Return type:

dict

Example

>>> results = hico_det_eval(blob, '<path-to-anno_bbox.mat>')
>>> print(results)
... {'def_mAP': xxx, 'def_mRec': xxx, ... }

References