Analysis of how video preprocessing can artificially increase VMAF, the industry-standard video quality metric, and a stable modification of it.
VMAF is widely used to compare and tune video codecs, so its vulnerabilities directly affect industry decisions. We showed that simple color and contrast adjustments can artificially increase VMAF (Zvezdakova et al., 2019), and that both VMAF and its “no enhancement gain” version VMAF NEG are vulnerable to various preprocessing methods (Siniukov et al., 2021). We also developed a neural preprocessing method that increases VMAF via distillation (Solov’ev et al., 2023).
Building on this analysis, we proposed evolutionary and distillation-based adversarial attacks on VMAF and Stable VMAF, a modification that is substantially more robust to such manipulations (Lavrushkin et al., 2025).
References
2025
MMSJ
Stable VMAF: investigating VMAF’s vulnerabilities to adversarial attacks
Sergey Lavrushkin, Maksim Khrebtov, Anastasia Antsiferova, Georgii Bychkov, Alexey Soloviev, and Dmitriy Vatolin
In recent years, Video Multimethod Assessment Fusion (VMAF) has become a prominent metric thanks to its high correlation with subjective video-quality assessments, making it preferable for evaluating video codecs and video-processing algorithms. Like many machine-learning-based metrics, however, it is susceptible to adversarial attacks, which can manipulate scores while preserving or even degrading visual quality. This paper investigates VMAF’s vulnerabilities to such attacks and proposes a novel, stable modification to enhance its robustness. We propose two adversarial attacks: an evolutionary-based attack, which achieves an average VMAF gain of 9.27 with a processing speed of 21.116 FPS, and a distillation-based neural attack, yielding a 6.86 average VMAF gain at 7.016 FPS. Using these methods, we created a dataset for pseudoadversarial training of our stable-VMAF modification, which …
@article{vatolinStable2025,title={{Stable VMAF: investigating VMAF’s vulnerabilities to adversarial attacks}},author={Lavrushkin, Sergey and Khrebtov, Maksim and Antsiferova, Anastasia and Bychkov, Georgii and Soloviev, Alexey and Vatolin, Dmitriy},year={2025},journal={Multimedia Systems},}
2023
Preprint
Development of neural network-based video preprocessing method to increase the VMAF score relative to source video using distillation
Aleksei Valer’evich Solov’ev, Anastasiya Vsevolodovna Antsiferova, Dmitry Sergeevich Vatolin, and Vladimir Aleksandrovich Galaktionov
AV Solovev, AV Antsiferova, DS Vatolin, VA Galaktionov, “Development of neural network-based video preprocessing method to increase the VMAF score relative to source video using distillation”, Keldysh Institute preprints, 2023, 066, 11 pp. Preprints of the Keldysh Institute of Applied Mathematics RUS ENG JOURNALS PEOPLE ORGANISATIONS CONFERENCES SEMINARS VIDEO LIBRARY PACKAGE AMSBIB General information Latest issue Archive Search papers Search references RSS Latest issue Current issues Archive issues What is RSS Keldysh Institute preprints: Year: Volume: Issue: Page: Find Your organisation: Google bot Personal entry: Login: Password: Save password Enter Forgotten password? Register Powered by MathJax Preprints of the Keldysh Institute of Applied Mathematics, 2023, 066, 11 pp. DOI: https://doi.org/10.20948/prepr-2023-66 (Mi ipmp3198) Development of neural network-…
@misc{galaktionovDevelopment2023,title={{Development of neural network-based video preprocessing method to increase the VMAF score relative to source video using distillation}},author={Solov'ev, Aleksei Valer'evich and Antsiferova, Anastasiya Vsevolodovna and Vatolin, Dmitry Sergeevich and Galaktionov, Vladimir Aleksandrovich},year={2023},journal={Preprints of the Keldysh Institute of Applied Mathematics},}
2021
Hacking VMAF and VMAF NEG: vulnerability to different preprocessing methods
Maksim Siniukov, Anastasia Antsiferova, Dmitriy Kulikov, and Dmitriy Vatolin
Video quality measurement plays a critical role in the development of video processing applications. In this paper, we show how popular quality metrics VMAF and its tuning-resistant version VMAF NEG can be artificially increased by video preprocessing. We propose a pipeline for tuning parameters of processing algorithms which allows to increase VMAF by up to 218.8%. A subjective comparison of preprocessed videos showed that with the majority of methods visual quality drops down or stays unchanged. We show that VMAF NEG scores can also be increased by some preprocessing methods by up to 21.9%.
@article{vatolinHacking2021,title={{Hacking VMAF and VMAF NEG: vulnerability to different preprocessing methods}},author={Siniukov, Maksim and Antsiferova, Anastasia and Kulikov, Dmitriy and Vatolin, Dmitriy},year={2021},}
2019
GraphiCon
Hacking VMAF with video color and contrast distortion
Anastasia Zvezdakova, Sergey Zvezdakov, Dmitriy Kulikov, and Dmitriy Vatolin
In 29th International Conference on Computer Graphics and Vision, CEUR Workshop Proceedings, 2019
Video quality measurement takes an important role in many applications. Full-reference quality metrics which are usually used in video codecs comparisons are expected to reflect any changes in videos. In this article, we consider different color corrections of compressed videos which increase the values of full-reference metric VMAF and almost don’t decrease other widely-used metric SSIM. The proposed video contrast enhancement approach shows the metric inapplicability in some cases for video codecs comparisons, as it may be used for cheating in the comparisons via tuning to improve this metric values.
@inproceedings{vatolinHacking2019,title={{Hacking VMAF with video color and contrast distortion}},author={Zvezdakova, Anastasia and Zvezdakov, Sergey and Kulikov, Dmitriy and Vatolin, Dmitriy},year={2019},booktitle={29th International Conference on Computer Graphics and Vision, CEUR Workshop Proceedings},}