Adversarial Attacks on Image/Video Quality Metrics
New attacks and a large-scale benchmark revealing vulnerabilities of learning-based image and video quality metrics.
Role: Project Head
Period: Apr 2022 – Dec 2024
Learning-based image and video quality metrics are used to tune codecs and train enhancement models, which makes them a target: a model can learn to “hack” the metric instead of improving perceived quality. We developed a family of white-box and black-box attacks on no-reference metrics — universal perturbations (Shumitskaya et al., 2022), fast CNN-based attacks (Shumitskaya et al., 2023), the invisible one-iteration IOI attack (Shumitskaya et al., 2024), and transferable cross-modal image-to-video attacks (Gotin et al., 2025; Gotin et al., 2026).
Using these attacks, we built a benchmark of metrics robustness (Antsiferova et al., 2024), which found vulnerabilities in 15 no-reference metrics and the current full-reference SOTA method.
This paper proposes a novel method for transferable adversarial attacks from Image Quality Assessment (IQA) to Video Quality Assessment (VQA) models. Attacking modern VQA models is challenging due to their high complexity and the temporal nature of video content. Since IQA and VQA models share similar low-and mid-level feature representations, and IQA models are substantially cheaper and faster to run, we leverage them as surrogates to generate transferable adversarial perturbations. Our method, MaxT-I2VQA jointly Maximizes IQA scores and Targets IQA feature activations to improve transferability from IQA to VQA models. We first analyze the correlation between IQA and VQA internal features and use these insights to design a feature-targeting loss. We evaluate MaxT-I2VQA by transferring attacks from four state-of-the-art IQA models to four recent VQA models and compare against three competitive baselines. Compared to prior methods, MaxT-I2VQA increases the transferability of an attack success rate by 7.9% and reduces per-example attack runtime by 8 times. Our experiments confirm that IQA and VQA feature spaces are sufficiently aligned to enable effective cross-task transfer.
@article{antsiferovaImproving2026,title={{Improving Transferability of Adversarial Attacks via Maximization and Targeting from Image to Video Quality Assessment}},author={Gotin, Georgii and Shumitskaya, Ekaterina and Vatolin, Dmitriy and Antsiferova, Anastasia},year={2026},journal={Big Data and Cognitive Computing},}
2025
arXiv
Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics
Georgii Gotin, Ekaterina Shumitskaya, Anastasia Antsiferova, and Dmitriy Vatolin
Recent studies have revealed that modern image and video quality assessment (IQA/VQA) metrics are vulnerable to adversarial attacks. An attacker can manipulate a video through preprocessing to artificially increase its quality score according to a certain metric, despite no actual improvement in visual quality. Most of the attacks studied in the literature are white-box attacks, while black-box attacks in the context of VQA have received less attention. Moreover, some research indicates a lack of transferability of adversarial examples generated for one model to another when applied to VQA. In this paper, we propose a cross-modal attack method, IC2VQA, aimed at exploring the vulnerabilities of modern VQA models. This approach is motivated by the observation that the low-level feature spaces of images and videos are similar. We investigate the transferability of adversarial perturbations across different modalities; specifically, we analyze how adversarial perturbations generated on a white-box IQA model with an additional CLIP module can effectively target a VQA model. The addition of the CLIP module serves as a valuable aid in increasing transferability, as the CLIP model is known for its effective capture of low-level semantics. Extensive experiments demonstrate that IC2VQA achieves a high success rate in attacking three black-box VQA models. We compare our method with existing black-box attack strategies, highlighting its superiority in terms of attack success within the same number of iterations and levels of attack strength. We believe that the proposed method will contribute to the deeper analysis of robust VQA metrics.
@misc{vatolinCrossmodal2025,title={{Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics}},author={Gotin, Georgii and Shumitskaya, Ekaterina and Antsiferova, Anastasia and Vatolin, Dmitriy},year={2025},journal={arXiv preprint arXiv:2501.08415},}
2024
ICML
IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image-and Video-Quality Metrics
Ekaterina Shumitskaya, Anastasia Antsiferova, and Dmitriy Vatolin
In Proceedings of the 41st International Conference on Machine Learning, 2024
No-reference image- and video-quality metrics are widely used in video processing benchmarks. The robustness of learning-based metrics under video attacks has not been widely studied. In addition to having success, attacks on metrics that can be employed in video processing benchmarks must be fast and imperceptible. This paper introduces an Invisible One-Iteration (IOI) adversarial attack on no-reference image and video quality metrics. The proposed method uses two modules to ensure high visual quality and temporal stability of adversarial videos and runs for one iteration, which makes it fast. We compared our method alongside eight prior approaches using image and video datasets via objective and subjective tests. Our method exhibited superior visual quality across various attacked metric architectures while maintaining comparable attack success and speed. We made the code available on GitHub: https://github.com/katiashh/ioi-attack.
@inproceedings{vatolinIoi2024,title={{IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image-and Video-Quality Metrics}},author={Shumitskaya, Ekaterina and Antsiferova, Anastasia and Vatolin, Dmitriy},year={2024},booktitle={Proceedings of the 41st International Conference on Machine Learning},}
AAAI
Comparing the robustness of modern no-reference image-and video-quality metrics to adversarial attacks
Anastasia Antsiferova, Khaled Abud, Aleksandr Gushchin, Ekaterina Shumitskaya, Sergey Lavrushkin, and Dmitriy Vatolin
In Proceedings of the AAAI Conference on Artificial Intelligence, 2024
Nowadays neural-network-based image- and video-quality metrics show better performance compared to traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics’ scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. However, the adversarial robustness of image-quality metrics is also an area worth researching. In this paper, we analyse modern metrics’ robustness to different adversarial attacks. We adopted adversarial attacks from computer vision tasks and compared attacks’ efficiency against 15 no-reference image/video-quality metrics. Some metrics showed high resistance to adversarial attacks which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts new metrics submissions for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. Try our benchmark using pip install robustness-benchmark.
@inproceedings{vatolinComparing2024,title={{Comparing the robustness of modern no-reference image-and video-quality metrics to adversarial attacks}},author={Antsiferova, Anastasia and Abud, Khaled and Gushchin, Aleksandr and Shumitskaya, Ekaterina and Lavrushkin, Sergey and Vatolin, Dmitriy},year={2024},booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},}
2023
ICLR TP
Fast Adversarial CNN-based Perturbation Attack on No-Reference Image-and Video-Quality Metrics
Ekaterina Shumitskaya, Anastasia Antsiferova, and Dmitriy Vatolin
Modern neural-network-based no-reference image- and video-quality metrics exhibit performance as high as full-reference metrics. These metrics are widely used to improve visual quality in computer vision methods and compare video processing methods. However, these metrics are not stable to traditional adversarial attacks, which can cause incorrect results. Our goal is to investigate the boundaries of no-reference metrics applicability, and in this paper, we propose a fast adversarial perturbation attack on no-reference quality metrics. The proposed attack (FACPA) can be exploited as a preprocessing step in real-time video processing and compression algorithms. This research can yield insights to further aid in designing of stable neural-network-based no-reference quality metrics.
@article{vatolinFast2023,title={{Fast Adversarial CNN-based Perturbation Attack on No-Reference Image-and Video-Quality Metrics}},author={Shumitskaya, Ekaterina and Antsiferova, Anastasia and Vatolin, Dmitriy},year={2023},journal={Tiny Papers @ {ICLR} 2023},}
2022
BMVC
Universal perturbation attack on differentiable no-reference image-and video-quality metrics
Ekaterina Shumitskaya, Anastasia Antsiferova, and Dmitriy Vatolin
In The 33rd British Machine Vision Conference, 2022
Universal adversarial perturbation attacks are widely used to analyze image classifiers that employ convolutional neural networks. Nowadays, some attacks can deceive image- and video-quality metrics. So sustainability analysis of these metrics is important. Indeed, if an attack can confuse the metric, an attacker can easily increase quality scores. When developers of image- and video-algorithms can boost their scores through detached processing, algorithm comparisons are no longer fair. Inspired by the idea of universal adversarial perturbation for classifiers, we suggest a new method to attack differentiable no-reference quality metrics through universal perturbation. We applied this method to seven no-reference image- and video-quality metrics (PaQ-2-PiQ, Linearity, VSFA, MDTVSFA, KonCept512, Nima and SPAQ). For each one, we trained a universal perturbation that increases the respective scores. We also propose a method for assessing metric stability and identify the metrics that are the most vulnerable and the most resistant to our attack. The existence of successful universal perturbations appears to diminish the metric’s ability to provide reliable scores. We therefore recommend our proposed method as an additional verification of metric reliability to complement traditional subjective tests and benchmarks.
@inproceedings{vatolinUniversal2022,title={{Universal perturbation attack on differentiable no-reference image-and video-quality metrics}},author={Shumitskaya, Ekaterina and Antsiferova, Anastasia and Vatolin, Dmitriy},year={2022},booktitle={The 33rd British Machine Vision Conference},}