consnet.api.evaluation

consnet.api.evaluation.hico_det_eval(blob, anno, split='test', mode=['def', 'ko'], zero_shot=None, logger=None)[source]

Perform standard evaluation on HICO-DET dataset using mean average precision (mAP) as introduced in [1].

Parameters

blob (list[Tensor[N, 10]]) – Human-object pairs and their detection scores to be evaluated. The length of the list should be 600 and each item should be an N * 10 tensor in (batch_id, x1, y1, x2, y2, ..., score) format.
anno (torch.Tensor or str) – The annotations object or path to the anno_bbox.mat file.
split (str, optional) – The dataset split to be evaluated. Expected values are 'train' and 'test'.
mode (list[str] or str, optional) – Mode of evaluation. Expected values are 'def', 'ko' or a list containing these terms, denoting the default mode and known-object mode introduced in [1].
zero_shot (dict or None, optional) –
Configurations for zero-shot settings. It should contain the following fields:
- type (str): Expected values include 'uc', 'ub' and 'ua', representing unseen action-object combination, unseen object and unseen action scenarios introduced in [2].
- id (int, optional): Only valid when type='uc'. Expected values are in the range of 0 ~ 4, indicating the 5 groups of unseen action-object combination settings in [2, 3].
logger (logging.Logger or str or None, optional) – The potential logger or name of the logger to be used.

Returns

Evaluation results including mean average precision (mAP) and mean recall (mRec) values of multiple dataset splits under different evaluation modes.

Return type

dict

Example

>>> results = hico_det_eval(blob, '<path-to-anno_bbox.mat>')
>>> print(results)
... {'def_mAP': xxx, 'def_mRec': xxx, ... }

References