Benchmark

Benchmark Protocol

Metrics

In this benchmark, we follow the convention by adopting the following four metrics to evaluate the model performances:

Evaluation Datasets

In this benchmark, we adopt the following two recent datasets for evaluation:

Participation

To participate in the benchmark, please send the following information to cat.cosaliency@gmail.com:

  • Author(s).
  • Model name.
  • Evaluation results.
  • Paper URL.
  • Project URL (if any).
  • Method description (if any).


Benchmark: CoSOD3k

Last updated: Sept 2021

# Model Venue MAE Fmax Favg Emax Eavg S
1CBCS (Fu et al. 2013) TIP'13 0.2280.4680.3220.6330.5090.529
2CSHS (Liu et al. 2014) SPL'14 0.3090.484-0.656-0.563
3ESMG (Li et al. 2014) SPL'14 0.2390.364--0.6150.532
4CODR (Ye et al. 2015) SPL'15 0.2290.458--0.6450.630
5GW (Wei et al. 2017) IJCAI'17 0.1470.649-0.777-0.716
6PoolNetβ—‹ (Liu et al. 2019) CVPR'19 0.1710.5610.5280.7510.6890.636
7CSMG (Zhang et al. 2019) CVPR'19 0.1410.7290.5960.8210.6750.727
8BASNet (Qin et al. 2019) CVPR'19 0.1220.696--0.7910.753
9EGNetβ—‹ (Zhao et al. 2019) ICCV'19 0.0880.7450.7350.8400.8300.791
10SCRN (Wu et al. 2019) ICCV'19 0.1180.717--0.8060.773
11RCAN (Li et al. 2019) IJCAI'19 0.1300.688-0.808-0.744
12SSNM (Zhang et al. 2020a) AAAI'20 0.1200.675--0.7560.726
13F3Net (Wei et al. 2020) AAAI'20 0.1140.717-0.802-0.772
14GCAGC (Zhang et al. 2020b) CVPR'20 0.1000.740--0.8310.778
15MINet (Pang et al. 2020) CVPR'20 0.1220.707-0.782-0.754
16ICNet-Vβ—‹ (Jin et al. 2020) NeurIPS'20 0.0680.7910.7820.8700.8660.811
17ICNet-Rβ—‹ (Jin et al. 2020) NeurIPS'20 0.0670.7920.7830.8720.8670.817
18CoADNet (Zhang et al. 2020c) NeurIPS'20 0.0780.786-0.874-0.822
19GICDβ—‹ (Zhang et al. 2020) ECCV'20 0.0700.7860.7790.8860.8630.812
20CoEGNetβ—‹ (Fan et al. 2021a) TPAMI'21 0.0920.736-0.825-0.762
21DeepACG (Zhang et al. 2021) CVPR'21 0.0890.756--0.8380.792
22GCoNetβ—‹ (Fan et al. 2021b) CVPR'21 0.0650.7890.7840.8650.8620.813
Notations: Benchmarking results on CoSOD3k (Fan et al. 2020). Models trained by CAT are highlighted with superscript β—‹. β€œ-V” and β€œ-R” denote models with VGG and ResNet backbones, respectively.

Benchmark: CoCA

Last updated: Sept 2021

# Model Venue MAE Fmax Favg Emax Eavg S
1CBCS (Fu et al. 2013) TIP'13 0.1800.3130.2300.6410.5310.523
2CSHS (Liu et al. 2014) SPL'14 ------
3ESMG (Li et al. 2014) SPL'14 ------
4CODR (Ye et al. 2015) SPL'15 ------
5GW (Wei et al. 2017) IJCAI'17 0.1660.408-0.701-0.602
6PoolNetβ—‹ (Liu et al. 2019) CVPR'19 0.1810.3460.3230.7070.6250.561
7CSMG (Zhang et al. 2019) CVPR'19 0.1240.508-0.735-0.632
8BASNet (Qin et al. 2019) CVPR'19 0.1950.397--0.6230.589
9EGNetβ—‹ (Zhao et al. 2019) ICCV'19 0.1430.4490.4390.6910.6780.626
10SCRN (Wu et al. 2019) ICCV'19 0.1660.416--0.6580.610
11RCAN (Li et al. 2019) IJCAI'19 0.1600.422-0.702-0.616
12SSNM (Zhang et al. 2020a) AAAI'20 0.1160.482--0.7410.628
13F3Net (Wei et al. 2020) AAAI'20 0.1780.437-0.678-0.614
14GCAGC (Zhang et al. 2020b) CVPR'20 0.1110.523--0.7540.669
15MINet (Pang et al. 2020) CVPR'20 0.2210.387-0.634-0.550
16ICNet-Vβ—‹ (Jin et al. 2020) NeurIPS'20 0.1080.5290.5220.7440.7340.673
17ICNet-Rβ—‹ (Jin et al. 2020) NeurIPS'20 0.1010.5330.5260.7510.7430.678
18CoADNet (Zhang et al. 2020c) NeurIPS'20 ------
19GICDβ—‹ (Zhang et al. 2020) ECCV'20 0.1160.5230.5160.7270.7160.668
20CoEGNetβ—‹ (Fan et al. 2021a) TPAMI'21 0.1040.499-0.717-0.616
21DeepACG (Zhang et al. 2021) CVPR'21 0.1020.552--0.7710.688
22GCoNetβ—‹ (Fan et al. 2021b) CVPR'21 0.0890.5310.5280.7300.7280.673
Notations: Benchmarking results on CoCA (Zhang et al. 2019). Models trained by CAT are highlighted with superscript β—‹. β€œ-V” and β€œ-R” denote models with VGG and ResNet backbones, respectively.