Defense Day

Time and Place

The next defense session will take place on March, 31 2023, Friday at 10:00 AM. You can join the Zoom meeting room using the information below. 

You can join the Zoom meeting room using the information below. 

Join Zoom Meeting
https://tum-conf.zoom.us/j/67107016249?pwd=cy8zam5Na3JQRGNwMHhWZWlQYkpOdz09

Meeting ID: 671 0701 6249
Passcode: 466589

Schedule

10:00 - 10:10 Gerhard Wardana (Initial Topic Presentation)

Title: Camera-LiDAR Deep Fusion Using Vehicle and Roadside Sensors   

Advisor: Walter Zimmer

Keywords: Camera-LiDAR Deep Fusion, Roadside Sensors, Autonomous Driving, V2I, 3D Deep Learning

Abstract: Most autonomous driving research today is focused on vehicle-based methods. Although their accuracy and robustness have improved over the years, among other things by utilizing multimodality, they are inherently limited by the weakness of vehicle-based methods: occlusion and hardware limitation. This thesis attempts to alleviate these weaknesses by fusing vehicle camera-LiDAR data with infrastructure camera-LiDAR data. We propose a two-step, deep fusion method to achieve this: deep camera-LiDAR fusion and deep vehicle-infrastructure fusion, with the infrastructure Roadside Unit (RSU) as its core.

10:10 - 10:20 Egemen Kopuz (Initial Topic Presentation)

Title: Multi-Modal 3D Object Detection in Long Range and Low-Resolution Conditions of Sensors

Advisor: Walter Zimmer

Keywords: 3D Deep Learning, Camera-LiDAR Deep Fusion, Roadside Sensors, Autonomous Driving

Abstract: The precise identification of objects within three-dimensional space is a critical requirement for the reliable and effective functioning of various applications, including autonomous driving, robotics, and surveillance systems. However, accurate 3D object detection presents a formidable challenge, particularly under difficult conditions such as those associated with low-resolution sensors and long distances. The primary objective of this thesis is to develop a technique capable of addressing these challenges, while also ensuring the technique's applicability across different systems.

10:20 - 10:30 Ahmed Alaa (Initial Topic Presentation)

Title: Generation of Synthetic Datasets of 3D Objects Using Semi-Supervised Learning

Advisor:  Walter Zimmer

Keywords: Datasets, Synthetic Data, Labeling, Semi-Supervised Learning, Autonomous Driving

Abstract: The need for extensive 3D labelled datasets becomes increasingly crucial for efficient scene perception in the field of autonomous driving. However, annotating 3D point clouds poses significant challenges and is a major bottleneck in the development of accurate 3D perception models. This thesis investigates semi-supervised learning methodologies to accelerate the 3D annotation process, while addressing the challenges of point cloud data complexity, class imbalance, and domain shift.

10:30 - 10:55 Donghao Song (Master's Thesis final)

Title: Privacy Risk Analysis for Synthetic Data    

Advisor: Shangding Gu

Keywords: AI Safety, Synthetic Data, Privacy Risk Analysis

Abstract: The adoption of data-driven decision-making approaches has highlighted the necessity for data sharing. However, the disclosure of sensitive data to third-party entities poses a significant risk to privacy. In order to mitigate such risk, the use of synthetic data has emerged as a promising alternative to direct exposure of original data. In this regard, the adoption of synthetic data, which approximates the statistical properties of the original data, has emerged as a viable practical replacement for the original sensitive data for various analytical tasks. Synthetic images, in particular, have garnered significant attention as a potential solution for privacy-preserving image sharing, given the advancements in image synthesis techniques.  To evaluate the privacy leakage risk, this work uses the de facto standard: Membership Inference Attack (MIA) that aims to predict membership information. A new MIA approach under the most practical threats model is proposed in this work. Experimental results demonstrate the great advantage of our new attack approach in terms of the metric: Area Under ROC-Curve (AUC).  All analysis experiments rely on a modular re-usable software, SIPA, which enables image owners to assess both the privacy risk and utility of it's synthetic images generated by varies generative models, and equally serves as a benchmark tool for researchers and practitioners.

10:55 - 11:15 Kathleen Baur (Bachelor's Thesis final)

Title: Multi-Agent Proximal Policy Optimization with PID Lagrangian Method    

Advisor:  Shangding Gu

Keywords: Reinforcement Learning, Multi-Agent Systems, Lagrangian Optimization

Abstract: The importance of safe multi-agent reinforcement learning (MARL) is evident due to the variety of related real-world applications. One of the principal challenges encountered is that cooperation requires agents to consider both their own and others' safety constraints and reward maximization. Therefore, existing safe MARL algorithms face a variety of issues such as high computational complexity. A promising algorithm is Multi-Agent Proximal Policy Optimization with the Lagrangian method (MAPPO-L), which unfortunately exhibits stability issues. Alongside unsafe behavior during learning, this is a common issue in Lagrange multiplier methods. PID-controlled Lagrangian updates have successfully speed up and stabilized dual variable dynamics of Constrained Policy Optimization algorithms. The PID method’s ability to stabilize MAPPO-L will be investigated in Safe Multi-Agent MuJoCo and Safe Multi-Agent Robosuite environments. Furthermore, alternative methods for stability improvements will be researched.    

11:15 - 11:40 Mhamed Jaafar (Master's Thesis final)

Title: Safe Reinforcement Learning with Safety Critic Policy Optimization             

Advisor:  Shangding Gu   

Keywords: Safe Reinforcement Learning, Robotics, Policy Optimization

Abstract: Incorporating safety is a critical requirement for extending the practical applications of reinforcement learning to real-world scenarios. Constrained Markov Decision Processes (CMDP) have been utilized to address this, introducing a separate cost function to represent safety violations. This approach eliminates the need for designing a reward function that considers safety. The Lagrangian relaxation technique has been used in prior algorithms to transform constrained optimization problems into unconstrained dual problems. However, predicting unsafe behavior in these algorithms can be inaccurate, leading to instability in learning the Lagrange multiplier. This thesis presents a novel safe reinforcement learning algorithm. We define the safety critic, which enables nullifying rewards gained by violating safety constraints. The algorithm automatically manages the trade-off between adhering to safety constraints and maximizing the return. The effectiveness of our algorithm is empirically demonstrated by benchmarking it against five other safe reinforcement learning techniques.

11:40 - 12:05 Xiangyu Bao (Master's Thesis final)

Title:  Convex Natural Evolutionary Strategies with Path Integrals for Robot Learning              

Advisor:  Yingbai Hu     

Keywords: Reinforcement Learning, Convex Natural Evolutionary Strategies, Path Integrals, Multi-layer Perceptron

Abstract: In robotic learning tasks, the properties that are requisite in the autonomous robot learning systems include fast learning capability, robustness to additional task goals and adaptability to multi-tasks. In this regard, we implement a path integral based learning framework under the guidance of convex natural gradient. The first step is to train a multi-layer perceptron model from the learning demonstrations. The second step is to refine the model using reinforcement learning with path integrals and convex natural evolutionary strategies. The framework is tested on via-points robot task and MuJoCo tasks in OpenAI gym suite and is verified to have faster convergence and safer convergence paths.

12:05 - 12:30 Xiongyu Xie (Master's Thesis final)

Title:  Continual State Representation Learning on Randomized Simulations for Sim2real Transfer in Robotic Manipulation

Advisor:  Josip Josifovski, Hossein Malmir     

Keywords: Reinforcement Learning, Continual Learning, Robotics, Simulation, Sim2Real, Domain  Randomization, Variational Autoencoder

Abstract: Currently, training the most successful reinforcement learning (RL) algorithms requires large amounts of data, which makes direct real-world training infeasible. To overcome this, one usually relies on training in simulation and using randomization methods to complete sim-to-real model transfer. However, this process is very time and resource consuming. To address this problem, in this thesis we explore how continual state representation learning can be combined with parameter randomization for vision-based reinforcement learning of robotic task. To this end, we use variational auto-encoder (VAE) to continually learn to reconstruct invariant image representation from sequentially randomized/augmented simulation images. Independently, an RL model is trained on the invariant image representation to solve a robotic manipulation task. Then, the VAE is used to translate randomized/augmented simulation images or real-world images to the invariant representation images on which the RL agent can operate. Experimental results demonstrate that the VAE is able to continually learn reconstruction to invariant images and it can bridge the sim-to-real gap by reconstructing correctly also real-world images. We then show that the RL agent using outputs from the VAE can accomplish the target reaching task with different image inputs in the simulator and has the potential of sim-to-real transfer..

12:30 - 13:00 Lunch Break

13:00 - 13:25 Zafercan Aygürler (Master's Thesis final))

Title: Camera-Based Object Detection in Poor-Visibility Conditions Using Adaptive Teacherr            

Advisor:  Christian Creß                 

Keywords: : Computer Vision, Object Detection, Poor-Visibility Object Detection, Adaptive Teacher

Abstract: Object detection is a critical technology that has improved rapidly in the last decade. Most of the improvements can be attributed to the focus on research in deep learning and advancements in GPUs. It is useful in many different areas such as facial recognition, object tracking, people counting, inventory management and autonomous driving. Autonomous driving systems should be reliable in all kinds of weather and daytime otherwise they would be posing threats to human lives on the road. However, this is not an easy task since even humans struggle in some poor-visibility conditions. Performance of current state-of-the-art object detectors is limited in scope to clear weather conditions because of the datasets they were trained with. Datasets in poor-visibility conditions are usually not available or not large enough especially because of high costs of annotating data. This thesis is focused on Object Detection in adverse weather and nighttime situations on a highway setting. A suitable Cross-domain object detection algorithm was selected and evaluated to perform 3 different domain adaptation experiments which are from clear weather domain to one of the poorvisibility domains. This algorithm only uses annotations of Clear Weather Dataset and does not require annotations in target domain. Evaluations were done on the A9 Dataset which was recorded by the cameras used in Providentia++ project. Unlike cameras of other driving datasets, Providentia++ cameras are fixed on a bridge overlooking the highway B471 near Munich, Germany. There has been significant performance improvements across all 3 experiments thanks to the domain adaptation while using Adaptive Teacher compared to the classical object detectors trained only on Clean weather. In Snow and Night adaptation cases, the performances of selected Adaptive Teacher model got close to the Oracle model which was trained by using annotations in target domain.    

13:25 - 13:50 Ruining Wang (Master's Thesis Final)

Title: Accurate Semantic Segmentation using Fused Multi-Scale Features and CCPL-based Data Augmentation

Advisor: Liguo Zhou        

Keywords: Semantic Segmentation, GAN, Autonomous Driving

Abstract: The accuracy of the semantic segmentation model has always been an important indicator of whether it can be applied to the field of autonomous driving. We solve this problem from two aspects of model and data. In terms of the model, we improve the CSPNet-based model by using a multi-scale feature fusion module to ensure that it can retain as many multi-dimensional features as possible during the encoding process. In terms of data, the main obstacle is that the manual labeling of image data is too costly to obtain in large quantities. Therefore, we use the simulated road image data obtained from the open-world 3D game GTA5, and utilize the style transfer model based on CCPL, to convert the simulated data into a dataset close to the urban scene dataset Cityscapes. In this way, the cost of labeled data acquisition will be greatly reduced. At the same time, this thesis also proposes a data matching method to optimize the transfer effect of the style transfer model, so that the source data and the target data are as close as possible and are usable for the model training. Combining the above points of work, this thesis proposes a semantic segmentation model framework for autonomous driving scenarios to improve the accuracy of the model. This paper does several sets of experiments on various semantic segmentation models and style transfer models and gives some further suggestions for improvements in the future..   

13:50 - 14:15 Ahmed Ewva (Master's Thesis Final)

Title: Tracking with deep learning instance segmentation  

Advisor: Soubarna Banik  

Keywords: Video Instance Segmentation, transformers, multi-object tracking

Abstract: Video instance segmentation is an important computer vision task that aims to simultaneously detect, classify, segment, and track objects in videos. Numerous online and offline methods based on powerful Transformer models have been proposed to tackle this task. In this context, this thesis proposes a new approach for online video instance segmentation that achieves efficient and accurate tracking through propagation, without relying on heuristics or handcrafted algorithms for the matching of detections. The proposed method leverages a gating mechanism based on conditional convolution to regulate the feature propagation across frames, while also introducing learnable position propagation and reinforced tracking through auxiliary training loss functions. To further improve efficiency, the proposed method also utilizes Encoder sparsification, where only salient locations of the frame in the Encoder are processed, relying on past information from previous frames to select the locations and make processing faster in subsequent frames. Experimental results show that the proposed method achieves a significant improvement over the baseline, demonstrating the effectiveness of the proposed approach in achieving efficient and accurate online video instance segmentation. Overall, this work contributes to advancing the field of computer vision and paves the way for further improvements in video instance segmentation task.   

14:15 - 14:35 Mengyu Li (Guided Research Presentation)

Title: Neuromorphic vision-based multi-object tracking for intelligent transportation systems

Advisor: Hu Cao   

Keywords: Multi-object tracking, Intelligent Transportation Systems, Event-based cameras, ByteTrack

Abstract: Multi-object tracking is a crucial task in Intelligent Transportation Systems (ITS), requiring real-time and accurate monitoring of multiple objects simultaneously. Recent advancements in neuromorphic vision sensors and online multi-object tracking algorithms have improved performance in promising multi-object tracking in complex traffic scenarios. This study compares two state-of-the-art algorithms for online multi-object tracking in ITS: Online Multi-object Tracking-by-Clustering and ByteTrack. Furthermore, we propose an Alternative approach by employing YOLOX as the pre-detection method, enhancing detection results through supervised learning. Data preprocessing steps are also described, including collecting data from the time camera, simplifying the processing, and converting marked data into a more general format. The hyperparameters of ByteTrack are tuned to improve mAP and accuracy. Our results show that the proposed approach outperforms Online Multi-object Tracking-by-Clustering in terms of detection accuracy and processing time.

14:35 - 14:55 Haichuan Li (Guided Research Presentation)

Title: Autonomous driving simulator based on Unity3D   

Advisor: Liguo Zhou  

Keywords: Autonomous Driving, Simulation, Unity

Abstract: Self-driving cars have the potential to revolutionize transportation, but they require extensive testing to ensure their safety and effectiveness. Real-world testing can be time-consuming and costly, and it also raises safety and regulatory concerns. Moreover, testing under extreme weather conditions and complex traffic scenarios is challenging to replicate in the real world. In response to these challenges, the industry has turned to automated driving simulation testing, which has become a widely adopted method for testing autonomous driving algorithms. Statistics show that the vast majority of autonomous driving algorithm tests, around 90%, are conducted through simulation platforms, while only 1% of tests are performed on actual roads. This trend underscores the importance of simulation testing as a vital tool for the development and optimization of self-driving cars.

14:55 - 15:15 Andrei Radu Zitti (Bachelor's Theis Final)

Title: Capturing Intra Object Relations via Graphs for Object Detection   

Advisor: Bare Luka Žagar  

Keywords: 3D Object Detection, Graph Theory, Loss Functions, Autonomous Driving, Deep Learning

Abstract: Just as the transition from horse-drawn carriages to automobiles marked a significant shift in transportation history, so are we now currently at the brink of yet another revolution in the automotive world. Autonomous driving will drastically lower traffic accident rates, reduce trafic congestions and also be accessible to everyone. However, one of the key  challenges to achieving full autonomy is accurate and reliable object detection. State-of-the-art algorithms don’t analyze the spacial correlation between objects, missing out on possible local and global patterns. This work aims to improve object detection algorithms by exploring the intra-connnectivity of the cars detected during training. To the end of exploring this avenue, we represent each car as a node in a graph and make use of graph isomorphism to compare ground-truth and prediction, and to the best of our knowledge are the first to do so. We are analyzing the graphs not only at a local-based level, exploring the connections between neighboring cars, but also analyze it as a whole, designing different losses for each use case. We design our algorithm as an plug-and-play module for any existing object detection program, and test it on Centerpoint with the nuScenes dataset for our experiments. We conduct extensive experiments and show a strong correlation between traffic patterns and object  detection, even obtaining improvements on the car mean Average precision as high as 4.1 %.

About Defense Day

The I6 defense day is held monthly, usually at the last Friday of each month. The common formats of talks are:

More information on the preparation of presentations can be found in the Thesis Submission Guidelines.