🐈CAT

Benchmark

Benchmark Protocol

Metrics

In this benchmark, we follow the convention by adopting the following four metrics to evaluate the model performances:

Mean absolute error (MAE), Cheng et al. 2013.
F-measure (F^max and F^avg), Achanta et al. 2019.
Enhanced-alignment measure (E^max and E^avg), Fan et al. 2018.
Structure measure (S), Fan et al. 2017.

Evaluation Datasets

In this benchmark, we adopt the following two recent datasets for evaluation:

CoSOD3k, Fan et al. 2020.
CoCA, Zhang et al. 2019.

Participation

To participate in the benchmark, please send the following information to cat.cosaliency@gmail.com:

Author(s).
Model name.
Evaluation results.
Paper URL.
Project URL (if any).
Method description (if any).

Benchmark: CoSOD3k

Last updated: Sept 2021

#	Model	Venue	MAE	F^max	F^avg	E^max	E^avg	S
1	CBCS (Fu et al. 2013)	TIP'13	0.228	0.468	0.322	0.633	0.509	0.529
2	CSHS (Liu et al. 2014)	SPL'14	0.309	0.484	-	0.656	-	0.563
3	ESMG (Li et al. 2014)	SPL'14	0.239	0.364	-	-	0.615	0.532
4	CODR (Ye et al. 2015)	SPL'15	0.229	0.458	-	-	0.645	0.630
5	GW (Wei et al. 2017)	IJCAI'17	0.147	0.649	-	0.777	-	0.716
6	PoolNet^○ (Liu et al. 2019)	CVPR'19	0.171	0.561	0.528	0.751	0.689	0.636
7	CSMG (Zhang et al. 2019)	CVPR'19	0.141	0.729	0.596	0.821	0.675	0.727
8	BASNet (Qin et al. 2019)	CVPR'19	0.122	0.696	-	-	0.791	0.753
9	EGNet^○ (Zhao et al. 2019)	ICCV'19	0.088	0.745	0.735	0.840	0.830	0.791
10	SCRN (Wu et al. 2019)	ICCV'19	0.118	0.717	-	-	0.806	0.773
11	RCAN (Li et al. 2019)	IJCAI'19	0.130	0.688	-	0.808	-	0.744
12	SSNM (Zhang et al. 2020a)	AAAI'20	0.120	0.675	-	-	0.756	0.726
13	F3Net (Wei et al. 2020)	AAAI'20	0.114	0.717	-	0.802	-	0.772
14	GCAGC (Zhang et al. 2020b)	CVPR'20	0.100	0.740	-	-	0.831	0.778
15	MINet (Pang et al. 2020)	CVPR'20	0.122	0.707	-	0.782	-	0.754
16	ICNet-V^○ (Jin et al. 2020)	NeurIPS'20	0.068	0.791	0.782	0.870	0.866	0.811
17	ICNet-R^○ (Jin et al. 2020)	NeurIPS'20	0.067	0.792	0.783	0.872	0.867	0.817
18	CoADNet (Zhang et al. 2020c)	NeurIPS'20	0.078	0.786	-	0.874	-	0.822
19	GICD^○ (Zhang et al. 2020)	ECCV'20	0.070	0.786	0.779	0.886	0.863	0.812
20	CoEGNet^○ (Fan et al. 2021a)	TPAMI'21	0.092	0.736	-	0.825	-	0.762
21	DeepACG (Zhang et al. 2021)	CVPR'21	0.089	0.756	-	-	0.838	0.792
22	GCoNet^○ (Fan et al. 2021b)	CVPR'21	0.065	0.789	0.784	0.865	0.862	0.813

Notations: Benchmarking results on CoSOD3k (Fan et al. 2020). Models trained by CAT are highlighted with superscript ○. “-V” and “-R” denote models with VGG and ResNet backbones, respectively.

Benchmark: CoCA

Last updated: Sept 2021

#	Model	Venue	MAE	F^max	F^avg	E^max	E^avg	S
1	CBCS (Fu et al. 2013)	TIP'13	0.180	0.313	0.230	0.641	0.531	0.523
2	CSHS (Liu et al. 2014)	SPL'14	-	-	-	-	-	-
3	ESMG (Li et al. 2014)	SPL'14	-	-	-	-	-	-
4	CODR (Ye et al. 2015)	SPL'15	-	-	-	-	-	-
5	GW (Wei et al. 2017)	IJCAI'17	0.166	0.408	-	0.701	-	0.602
6	PoolNet^○ (Liu et al. 2019)	CVPR'19	0.181	0.346	0.323	0.707	0.625	0.561
7	CSMG (Zhang et al. 2019)	CVPR'19	0.124	0.508	-	0.735	-	0.632
8	BASNet (Qin et al. 2019)	CVPR'19	0.195	0.397	-	-	0.623	0.589
9	EGNet^○ (Zhao et al. 2019)	ICCV'19	0.143	0.449	0.439	0.691	0.678	0.626
10	SCRN (Wu et al. 2019)	ICCV'19	0.166	0.416	-	-	0.658	0.610
11	RCAN (Li et al. 2019)	IJCAI'19	0.160	0.422	-	0.702	-	0.616
12	SSNM (Zhang et al. 2020a)	AAAI'20	0.116	0.482	-	-	0.741	0.628
13	F3Net (Wei et al. 2020)	AAAI'20	0.178	0.437	-	0.678	-	0.614
14	GCAGC (Zhang et al. 2020b)	CVPR'20	0.111	0.523	-	-	0.754	0.669
15	MINet (Pang et al. 2020)	CVPR'20	0.221	0.387	-	0.634	-	0.550
16	ICNet-V^○ (Jin et al. 2020)	NeurIPS'20	0.108	0.529	0.522	0.744	0.734	0.673
17	ICNet-R^○ (Jin et al. 2020)	NeurIPS'20	0.101	0.533	0.526	0.751	0.743	0.678
18	CoADNet (Zhang et al. 2020c)	NeurIPS'20	-	-	-	-	-	-
19	GICD^○ (Zhang et al. 2020)	ECCV'20	0.116	0.523	0.516	0.727	0.716	0.668
20	CoEGNet^○ (Fan et al. 2021a)	TPAMI'21	0.104	0.499	-	0.717	-	0.616
21	DeepACG (Zhang et al. 2021)	CVPR'21	0.102	0.552	-	-	0.771	0.688
22	GCoNet^○ (Fan et al. 2021b)	CVPR'21	0.089	0.531	0.528	0.730	0.728	0.673

Notations: Benchmarking results on CoCA (Zhang et al. 2019). Models trained by CAT are highlighted with superscript ○. “-V” and “-R” denote models with VGG and ResNet backbones, respectively.