User 1
User 2
User 3
User 4
User 1
User 2
User 3
User 4
Abstract
Since the advent of social media platforms, image selection based on social preference is a challenging task that all users inherently undertake before sharing images with the public. In our user study for this problem, human choices of images based on perceived social preference are largely inaccurate (58.7% accuracy). The challenge of this task, also known as image popularity assessment, lies in its subjective nature caused by visual and non-visual factors. Especially in the social media setting, social feedback on a particular image largely differs depending on who uploads it. Therefore social preference model should be able to account for this user-specific image aspect of the task. To address this issue, we present a retrieval-augmented approach that leverages both image features and user-specific statistics for neural image popularity assessment. User-specific statistics are derived by retrieving past images with their statistics from a memory bank. By combining these statistics with image features, our approach achieves 79.5% accuracy, which significantly outperforms human and baseline models on the pairwise ranking of images from the Instagram Influencer Dataset.
The framwork pipeline. (1) Along with the Swin Transformer backbone, we have two auxiliary branches to extract features. One is to extract the retrieval-augmented scores, and the other is to extract the 3D-aware features using the monocular depth estimation model. M_Train stands for the memory bank. (2) The aggregation block contains a stack of transformer encoder blocks that aggregate the feature from the extractors.
Illustration of retrieval strategy and user-specific image feature extractor. Image retrieval and features are processed based on the user's previous post and corresponding likes.
Top 5 retrieved images given a query image. Similarity scores between the query and retrieved images are stated in the bottom left corner of each image. The weighted sum of similarity calculates the similarity in three aspects: pose, background, and caption.
Video Presentation
BibTeX
@inproceedings{ji2023neural,
author = {Liya Ji and
Chan Ho Park and
Zhefan Rao and
Qifeng Chen},
title = {Neural Image Popularity Assessment with Retrieval-augmented Transformer},
booktitle = {ACM Multimedia},
year = {2023},
}