The largest benchmark of video-compression-related quality metrics, with industry-wide adoption.
Role: Project Head
Period: Mar 2021 – Jun 2025
Developed the biggest benchmark of video-compression-related quality metrics (Antsiferova et al., 2022). The benchmark received positive feedback from industry leaders, such as Huawei, Tencent, Yandex, Google (YouTube Media Algorithms), raised funding and received new methods submissions from companies. The dataset is published online, having an average of 15 downloads per month by leading industry companies and research institutes.
The work was continued with the LEHA-CVQAD dataset for generalized quality assessment of compression artifacts (Gushchin et al., 2025) and the AIM 2024 challenge on compressed video quality assessment (Smirnov et al., 2024). Earlier, we studied the limits of no-reference metrics for compressed video (Zvezdakova et al., 2019) and proposed the ERQA metric for video super-resolution (Kirillova et al., 2021).
We propose the LEHA-CVQAD (Large-scale Enriched Human Annotated) dataset, which comprises 6,240 clips for compression-oriented video quality assessment. 59 source videos are encoded with 186 codec-preset variants, ≈1.8M pairwise, and ≈1.5k MOS ratings are fused into a single quality scale; part of the videos remains hidden for blind evaluation. We also propose Rate-Distortion Alignment Error (RDAE), a novel evaluation metric that quantifies how well VQA models preserve bitrate-quality ordering, directly supporting codec parameter tuning. Testing IQA/VQA methods reveals that popular VQA metrics exhibit high RDAE and lower correlations, underscoring the dataset’s challenges and utility. The open part and the results of LEHA-CVQAD are available at https://aleksandrgushchin.github.io/lcvqad/
@article{antsiferovaLehacvqad2025,title={{LEHA-CVQAD: Dataset To Enable Generalized Video Quality Assessment of Compression Artifacts}},author={Gushchin, Aleksandr and Smirnov, Maksim and Vatolin, Dmitriy S and Antsiferova, Anastasia},year={2025},}
2024
ECCVW
AIM 2024 challenge on compressed video quality assessment: methods and results
Maksim Smirnov, Aleksandr Gushchin, Anastasia Antsiferova, Dmitry Vatolin, Radu Timofte, Ziheng Jia, Zicheng Zhang, Wei Sun, Jiaying Qian, Yuqin Cao, Yinan Sun, Yuxin Zhu, Xiongkuo Min, Guangtao Zhai, Kanjar De, Qing Luo, Ao-Xiang Zhang, Peng Zhang, Haibo Lei, Linyan Jiang, Yaqing Li, Wenhui Meng, Xiaoheng Tan, Haiqiang Wang, Xiaozhong Xu, Shan Liu, Zhenzhong Chen, Zhengxue Cheng, Jiahao Xiao, Jun Xu, Chenlong He, Qi Zheng, Ruoxi Zhu, Min Li, Yibo Fan, and Zhengzhong Tu
Video quality assessment (VQA) is a crucial task in the development of video compression standards, as it directly impacts the viewer experience. This paper presents the results of the Compressed Video Quality Assessment challenge, held in conjunction with the Advances in Image Manipulation (AIM) workshop at ECCV 2024. The challenge aimed to evaluate the performance of VQA methods on a diverse dataset of 459 videos, encoded with 14 codecs of various compression standards (AVC/H.264, HEVC/H.265, AV1, and VVC/H.266) and containing a comprehensive collection of compression artifacts. To measure the methods performance, we employed traditional correlation coefficients between their predictions and subjective scores, which were collected via large-scale crowdsourced pairwise human comparisons. For training purposes, participants were provided with the Compressed Video Quality …
@article{tuAim2024,title={{AIM 2024 challenge on compressed video quality assessment: methods and results}},author={Smirnov, Maksim and Gushchin, Aleksandr and Antsiferova, Anastasia and Vatolin, Dmitry and Timofte, Radu and Jia, Ziheng and Zhang, Zicheng and Sun, Wei and Qian, Jiaying and Cao, Yuqin and Sun, Yinan and Zhu, Yuxin and Min, Xiongkuo and Zhai, Guangtao and De, Kanjar and Luo, Qing and Zhang, Ao-Xiang and Zhang, Peng and Lei, Haibo and Jiang, Linyan and Li, Yaqing and Meng, Wenhui and Tan, Xiaoheng and Wang, Haiqiang and Xu, Xiaozhong and Liu, Shan and Chen, Zhenzhong and Cheng, Zhengxue and Xiao, Jiahao and Xu, Jun and He, Chenlong and Zheng, Qi and Zhu, Ruoxi and Li, Min and Fan, Yibo and Tu, Zhengzhong},year={2024},}
2022
NeurIPS
Video compression dataset and benchmark of learning-based video-quality metrics
Anastasia Antsiferova, Sergey Lavrushkin, Maksim Smirnov, Aleksandr Gushchin, Dmitriy Vatolin, and Dmitriy Kulikov
In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022
Video-quality measurement is a critical task in video processing. Nowadays, many implementations of new encoding standards-such as AV1, VVC, and LCEVC-use deep-learning-based decoding algorithms with perceptual metrics that serve as optimization objectives. But investigations of the performance of modern video-and image-quality metrics commonly employ videos compressed using older standards, such as AVC. In this paper, we present a new benchmark for video-quality metrics that evaluates video compression. It is based on a new dataset consisting of about 2,500 streams encoded using different standards, including AVC, HEVC, AV1, VP9, and VVC. Subjective scores were collected using crowdsourced pairwise comparisons. The list of evaluated metrics includes recent ones based on machine learning and neural networks. The results demonstrate that new no-reference metrics exhibit high correlation with subjective quality and approach the capability of top full-reference metrics.
@inproceedings{kulikovVideo2022,title={{Video compression dataset and benchmark of learning-based video-quality metrics}},author={Antsiferova, Anastasia and Lavrushkin, Sergey and Smirnov, Maksim and Gushchin, Aleksandr and Vatolin, Dmitriy and Kulikov, Dmitriy},year={2022},booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},}
2021
VISAPP
Erqa: Edge-restoration quality assessment for video super-resolution
Anastasia Kirillova, Eugene Lyapustin, Anastasia Antsiferova, and Dmitry Vatolin
In 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2021
Despite the growing popularity of video super-resolution (VSR), there is still no good way to assess the quality of the restored details in upscaled frames. Some SR methods may produce the wrong digit or an entirely different face. Whether a method’s results are trustworthy depends on how well it restores truthful details. Image super-resolution can use natural distributions to produce a high-resolution image that is only somewhat similar to the real one. VSR enables exploration of additional information in neighboring frames to restore details from the original scene. The ERQA metric, which we propose in this paper, aims to estimate a model’s ability to restore real details using VSR. On the assumption that edges are significant for detail and character recognition, we chose edge fidelity as the foundation for this metric. Experimental validation of our work is based on the MSU Video Super-Resolution Benchmark, which includes the most difficult patterns for detail restoration and verifies the fidelity of details from the original frame. Code for the proposed metric is publicly available at https://github.com/msu-video-group/ERQA.
@inproceedings{vatolinErqa2021,title={{Erqa: Edge-restoration quality assessment for video super-resolution}},author={Kirillova, Anastasia and Lyapustin, Eugene and Antsiferova, Anastasia and Vatolin, Dmitry},year={2021},booktitle={17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications},}
2019
GraphiCon
Barriers towards no-reference metrics application to compressed video quality analysis: on the example of no-reference metric NIQE
Anastasia Zvezdakova, Dmitriy Kulikov, Denis Kondranin, and Dmitriy Vatolin
In 29th International Conference on Computer Graphics and Vision, CEUR Workshop Proceedings, 2019
This paper analyses the application of no-reference metric NIQE to the task of video-codec comparison. A number of issues in the metric behaviour on videos was detected and described. The metric has outlying scores on black and solid-coloured frames. The proposed averaging technique for metric quality scores helped to improve the results in some cases. Also, NIQE has low-quality scores for videos with detailed textures and higher scores for videos of lower bitrates due to the blurring of these textures after compression. Although NIQE showed natural results for many tested videos, it is not universal and currently can not be used for video-codec comparisons.
@inproceedings{vatolinBarriers2019,title={{Barriers towards no-reference metrics application to compressed video quality analysis: on the example of no-reference metric NIQE}},author={Zvezdakova, Anastasia and Kulikov, Dmitriy and Kondranin, Denis and Vatolin, Dmitriy},year={2019},booktitle={29th International Conference on Computer Graphics and Vision, CEUR Workshop Proceedings},}