Topics

In this lecture, we will cover the most recent approaches and principles at the frontier of learning and applying of foundation models in language, vision (image generation, image segmentation, image understanding) as well as multimodal (=multimedia) data processing, including

1. learning vision foundation models from natural language supervision, with applications to open-vocabulary image classification and retrieval, object detection, segmentation, and multimodal understanding;
2. learning vision foundation models via masked image modeling, with its extensions to multimodal pre-training
3. vision foundation model architecture design with transformer and beyond.

Exercises

There will be exercises each week. The solutions are discussed in the exercise lecture in the following week. The exercises are not mandatory (but essential for passing the exam).

Turnus

The lecture is offered each summer term.