3D Camera Calibration of Sports Videos Using Physical Guidance

Project Overview

In recent years, technologies like goal-line systems have revolutionized sports, providing accurate and reliable solutions for refereeing, game analysis, and athlete performance evaluation. Many of these innovations depend on extracting 3D information from 2D images or videos, a process that relies heavily on precise camera calibration. This thesis project aims to optimize the camera calibration process, especially in the context of sports videos, by leveraging physical knowledge.

Motivation

A well-calibrated camera is essential for accurately interpreting 3D information from a scene. However, obtaining accurate camera calibration matrices can be challenging. Traditional approaches often rely on:

• Multi-view camera systems.

• Videos captured with a moving camera in a static environment.

• Known 3D positions of key points in the image.

While solving for camera matrices from known 2D-3D point pairs is straightforward, it can become numerically unstable when the detected 2D points are noisy. In this project, we aim to enhance camera calibration for sports videos by incorporating physical knowledge of ball dynamics. By analyzing the ball's trajectory in image space, we can refine the calibration matrices, yielding significantly more stable and reliable estimates.

What You Will Do

• Deep Learning: Implement and use neural networks for keypoint detection and semantic segmentation.

• Numerical Optimization: Improve calibration accuracy through advanced optimization techniques.

• Physics Integration: Utilize differential equations to model and analyze the ball’s motion.

• Real-World Application: Work with broadcast sports videos and develop an evaluation benchmark tailored to the proposed calibration method.

• Publication Potential: If the results are promising, there will be an opportunity to publish your findings.

Don’t worry—you will receive guidance and basic implementations to support your work.

Required Background and Skills

• Strong Foundations: Knowledge of camera matrices and coordinate transformations, as covered in the Grundlagen der Signalverarbeitung und des Maschinellen Lernens lecture, is highly recommended.

• Comfort with Math and Physics: While a background in physics is not mandatory, familiarity with relevant equations and numerical methods is beneficial.

Why Join This Project?

• Gain hands-on experience with cutting-edge technologies in computer vision and machine learning.

• Solve real-world challenges in sports video analysis.

• Work on a project with high potential for academic publication and practical impact.

Interested? Let’s refine sports technology together!

For more information, contact Daniel Kienzle

Last edited: 27.08.2024

Keywords: Scene Graph, Graph Database, LLM, Synthetic Data

Scene Graphs are used to represent a given image or video as a graph structure. This graph structure can then be queried to support further down stream tasks. Scene graphs can be queried by converting the graph to a graph database (e.g. Neo4j) and querying it using a graph query language (e.g. Cypher). Users could define their queries in such a query language, but in this thesis, the query should be derived from a text prompt using an LLM.

However, most Scene Graph datasets were not built with complex queries in mind and relations were selected by how easy they were to obtain and not how beneficial they are for a complex graph query. Therefore, you will create a set of dummy graphs (without any connection to existing images/videos) and explore relations and settings that allow for complex reasoning using graph queries. Next, you will use large language models (LLMs) to convert user text prompts into graph queries and evaluate the results.

Finally, with the insights gained, a set of useful relations and attributes for complex scene graph queries can be derived. These relations can then be generated and evaluated with our synthetic dataset generator.

Project outline:

1. Get to know Neo4j and Cypher
2. Create example graphs that can be queried
3. Derive graph queries from text prompts using an LLM
4. Define a set of relations that future scene graph datasets must contain
5. Generate and evaluate a synthetic dataset with our dataset generator

If you are interested in this topic or require more information, please contact Julian Lorenz.

Human Pose Estimation (HPE) is the task of detecting human keypoints in images or videos. 2D Human Pose Estimation means the localization of these keypoints in pixel coordinates in the image or video frame. 3D Human Pose Estimation is the task of estimating a three dimensional pose of the humans in the image or video. Mostly, this task is accomplished by uplifting estimated 2D poses to the third dimension, e.g., by leveraging the time context in videos.

Transformer architectures are currently most common in these taks. They have the benefit to have a global view instead of the local view that convolution operations have. Thesis topics in this field could include analyzing 3D HPE architectures, improving/adapting them, e.g., for different domains or target applications, analyzing different input or training modes like semi-supervised learning, etc. 

Semi-Supervised Learning is an active research field in computer vision with the goal to train neural networks with only a small labeled dataset and a lot of unlabeled data. For human pose estimation, this means that a large dataset with images from people is available, but only a small subset has annotated keypoints. Semi-supervised human pose estimation uses different techniques to train jointly on labeled and unlabeled images in order to improve the detection performance of the network. Popular methods are pseudo labels - the usage of network predictions as annotations - and teacher-student-approaches, where one network is enhanced by being trained by a second network.   

If you are interested and want more information, please contact Katja Ludwig

The computer vision task of Human Pose Estimation estimates keypoints of humans in either 2D or 3D. These keypoints can be connected such that a skeleton model of the human can be created. This skeleton model is sufficient for some tasks, but does not reflect the body shape of the person. Human Mesh Estimation overcomes this issue. It estimates not only keypoints, but a whole mesh representing the pose and the body shape of humans. This task is more challenging than pure 3D Human Pose Estimation, as a lot more parameters need to be estimated. In order to keep the amount of parameters relatively small, body models like SMPL and its successors are common in this field. Thesis topics could include the analysis of Human Mesh architectures, slight adaptations to the models or training routines, analyses or conversion of body models, etc.    

If you are interested and want more information, please contact  Katja Ludwig

The access to masks for objects in images is of great importance to many computer vision tasks. Manually annotating such object masks (for example with polygon drawings), however, takes an extensive amount of time. In addition to this, the annotation of finely jagged edges and delicate structures poses a considerable problem. Interactive segmentation systems try to drastically ease this task by using forms of user guidance that can be annotated cheaply in order to predict an object mask. Usually this guidance takes the form of right/left mouse clicks to annotate single background/foreground pixels.

Semantic segmentation constitutes the task of classfiying every single pixel into one of several predefined classes. In consequence interactive segmentation systems constitute a combination of the two tasks: The segmentation happens on the basis of user guidance while the goal is to circumvent a costly annotation process. Instead of annotating single objects, the goal is to divide the entire input image into several class surfaces.

Literature:

[1] : https://ceur-ws.org/Vol-2766/paper1.pdf

[2] : https://arxiv.org/abs/2003.14200

If case of interest, contact Robin Schön (robin.schoen@uni-a.de)