Student projects list

Part of ELLIS Summer School on Large-Scale AI for Research and Industry, 18–22 September 2023. This event has ended: the information below is kept for reference.

Efficient Foundation Model Training Using FLIP

Vision-language foundation models like CLIP [1] are usually trained using huge training datasets composed of images paired with their textual description. For instance, in CLIP, the visual encoder and the textual encoder represent an image and its textual description using two corresponding vectors, whose mutual cosine similarity is maximized using a contrastive loss. However, the training time of these models is very expensive, with thousands of GPU-days. In [2], the authors propose FLIP, in which training is largely sped up using a Masked Autoencoder (MAE) paradigm [3]. Specifically, similarly to [3], at training time, the visual encoder of FLIP represents only a random subset of visible image patches. Training is still based on a contrastive loss, where the (average pooled) representation of the visible patches is compared with the representation of the image textual description. However, representing only a subset of the image patches leads to a large training speedup and to a lower memory footprint, where the latter is exploited using a larger batch size, which is known to be particularly helpful with a contrastive loss.

The goal of this project is to propose and investigate possible variants of FLIP, including, but not limited to, different (additional) loss functions, different ways to reduce the computational burden or increase the batch size, etc., or to investigate the effects of the dataset size or the batch size in the pretraining phase.

In [4] the students can find an open-source implementation of FLIP, while in [5] they find the LAION-400M dataset used in [2] for pre-training (suggestion: use only a subset of LAION-400M in all your experiments). The encoders of the project should be based on (one of) the ViT architectures adopted in [2].

For testing, the students are encouraged to use one of the two main evaluation protocols adopted in [2], i.e., either Zero-shot accuracy on ImageNet-1K [6] classification or Linear probing on ImageNet-1K classification.

References

[1] A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision, ICML, 2021.
[2] Y. Li et al., Scaling Language-Image Pre-training via Masking, CVPR 2023.
[3] K. He et al., Masked autoencoders are scalable vision learners, CVPR 2022.
[4] FLIP code
[5] LAION dataset
[6] J. Deng et al., ImageNet: A large-scale hierarchical image database, CVPR 2009.