Yichao Zhou (University of California, Los Angeles), Shaunak Mishra (Yahoo Research), Manisha Verma (Yahoo Research), Narayan Bhamidipati (Yahoo Research) and Wei Wang (University of California, Los Angeles)
There is a perennial need in the online advertising industry to refresh ad creatives, i.e., images and text used for enticing online users towards a brand. Such refreshes are required to reduce the likelihood of ad fatigue among online users, and to incorporate insights from other successful campaigns in related product categories. However, given a brand, to come up with themes for a new ad is a painstaking and time consuming process for creative strategists. Among other things, strategists typically draw inspiration from the images and text used for past ad campaigns, as well as world knowledge on the brands. To automatically infer ad themes via such multimodal sources of information in past ad campaigns, we propose a theme (keyphrase) recommender system for ad creative strategists. In particular, the theme recommender is based on aggregating results from a visual question answering (VQA) task, which ingests the following: (i) ad images, (ii) text associated with the ads as well as Wikipedia pages on the brands in the ads, and (iii) questions around the ad. To harness the multimodal nature of the above inputs, we leverage transformer based cross-modality encoders to train visual-linguistic representations for our VQA task. We study two formulations for the VQA task along the lines of classification and ranking; via experiments on a public dataset, we show that cross-modal representations lead to significantly better classification accuracy and ranking precision-recall metrics. Specifically, cross-modal representations show better performance compared to separate image and text representations. In addition, the use of multimodal information shows a significant lift over using only textual or visual information. Finally, we share creative strategy insights on selected product categories in the public dataset using our approach.