Abstract
Collecting data on a large scale is vital for the development of cutting-edge artificial intelligence (AI) technologies, especially those involving machine learning (ML) models, such as deep neural networks, which require training with relevant data. On one hand, the collection of real-world data, using devices such as cameras and microphones, would enable AI systems to better understand everyday life and ultimately behave or assist in a manner akin to human interaction. On the other hand, growing concerns about security and privacy make it increasingly difficult to collect such real-world data. As a result, the emergence of digital twins offers a promising direction for intelligent robots that employ deep learning models for tasks such as perception, planning, localization, and control.
This presentation will discuss a framework for data collection, training, and learning, utilizing the assistance of digital twins. This approach leverages a collection of AI models for self-navigating mobile robots. We will focus particularly on developing visual perception models that interpret the real world through a camera. These models play a pivotal role in a variety of AI-powered products and services, such as autonomous vehicles and smart cities. They are also a primary research focus at Elsa Lab. Visual perception models based on deep neural networks have achieved unprecedented accuracy in benchmark datasets. Implementing these models would enable edge AI systems to better perceive and understand their environments, and therefore acting more intelligently in the real world. However, they often experience drops in accuracy and lack sufficient effective real-world data samples. This can lead to unsatisfactory performance and safety concerns in practical deployments.
To address the aforementioned problems, we explore and incorporate the following key technologies into our framework: virtual-to-real transfer learning, foundation model-based domain adaptation, and the usage of mid- level representations. Virtual-to-real transfer learning enables ML models to train first in simulated environments and then migrate to real-world settings with ease. Foundation model-based domain adaptation facilitates this migration process, even under challenging scenarios where data collection in the real world involves labor-intensive manual preprocessing. Moreover, mid-level representations are employed to transmit various types of information from the perception module to the control module, forming the basis of modular frameworks in many learning-based systems. The primary scientific challenge of this research direction lies in integrating these elements into a unified solution, and improving the adaptation ability of AI models to real- world environments. We will present how these methodologies can be integrated with existing systems for intelligent perception, planning, localization, and control in deep learning-based intelligent robotics.
This presentation will discuss a framework for data collection, training, and learning, utilizing the assistance of digital twins. This approach leverages a collection of AI models for self-navigating mobile robots. We will focus particularly on developing visual perception models that interpret the real world through a camera. These models play a pivotal role in a variety of AI-powered products and services, such as autonomous vehicles and smart cities. They are also a primary research focus at Elsa Lab. Visual perception models based on deep neural networks have achieved unprecedented accuracy in benchmark datasets. Implementing these models would enable edge AI systems to better perceive and understand their environments, and therefore acting more intelligently in the real world. However, they often experience drops in accuracy and lack sufficient effective real-world data samples. This can lead to unsatisfactory performance and safety concerns in practical deployments.
To address the aforementioned problems, we explore and incorporate the following key technologies into our framework: virtual-to-real transfer learning, foundation model-based domain adaptation, and the usage of mid- level representations. Virtual-to-real transfer learning enables ML models to train first in simulated environments and then migrate to real-world settings with ease. Foundation model-based domain adaptation facilitates this migration process, even under challenging scenarios where data collection in the real world involves labor-intensive manual preprocessing. Moreover, mid-level representations are employed to transmit various types of information from the perception module to the control module, forming the basis of modular frameworks in many learning-based systems. The primary scientific challenge of this research direction lies in integrating these elements into a unified solution, and improving the adaptation ability of AI models to real- world environments. We will present how these methodologies can be integrated with existing systems for intelligent perception, planning, localization, and control in deep learning-based intelligent robotics.
Bio
Chun-Yi Lee is a Professor in the Department of Computer Science and Informa9on Engineering at Na9onal Taiwan University (NTU), Taipei, Taiwan, and the supervisor of Elsa Lab. He received his B.S. and M.S. degrees from Na9onal Taiwan University in 2003 and 2005, respec9vely, and his M.A. and Ph.D. degrees from Princeton University, Princeton, NJ, USA, in 2009 and 2013, respec9vely, all in Electrical Engineering. Prof. Lee joined the Department of Computer Science at NTHU as an Assistant Professor in 2015. He was promoted to Associate Professor in 2019 and to full Professor in 2023. In 2024, Prof. Lee joined the Department of Computer Science and Informa9on Engineering at Na9onal Taiwan University. Prior to 2015, he served as a senior engineer at Oracle America, Inc. in Santa Clara, CA, USA, from 2012 to 2015. Prof. Lee founded Elsa Lab at Na9onal Tsing Hua University in 2015. Under his leadership, Elsa Lab has garnered several pres9gious awards from global robo9cs and AI challenges. These include the first place at the NVIDIA Embedded Intelligent Robo9cs Challenge in 2016, the first place at the NVIDIA Jetson Robo9cs Challenge in 2018, the second place in the Person-In-Context (PIC) Challenge at ECCV 2018, the second place in the NVIDIA AI at the Edge Challenge in 2020, and the Best Solu9on Award (1st Place) in the Small Object Detec9on Challenge for Spo[@BackSlash]ng Birds at MVA 2023.
Prof. Lee's research primarily focuses on deep reinforcement learning (DRL), intelligent robo9cs, computer vision (CV), and parallel compu9ng systems. His contribu9ons include developing key deep learning methodologies for intelligent robo9cs, such as sim-to-real and real-to-sim training, transfer techniques for robo9c policies, digital twins, scene coordinate regression approaches, and domain adapta9on techniques for seman9c segmenta9on models. He has also advanced explora9on approaches for DRL agents, mul9- agent reinforcement learning (MARL) techniques, and genera9ve modeling methodologies, including super- resolu9on and score-based genera9ve models, as well as autonomous naviga9on strategies. His work has been published at major ar9ficial intelligence (AI) conferences such as NeurIPS, CVPR, IJCAI, AAMAS, ICLR, ICML, ECCV, BMVC, CoRL, ICRA, IROS, GTC, MVA, and others. Addi9onally, his research appears in top AI journals, including IEEE Transac9ons on Paaern Recogni9on and Machine Intelligence (TPAMI), IEEE Transac9ons on Neural Networks and Learning Systems (TNNLS), Journal of Machine Learning Research (JMLR), and ACM Transac9ons on Evolu9onary Learning and Op9miza9on (TELO). In the realm of parallel systems, Prof. Lee has introduced innova9ve graphics processing unit (GPU) enhancement methodologies to improve efficiency. His work in this area has been published in IEEE Transac9ons on Very Large Scale Integra9on Systems (TVLSI), IEEE Transac9ons on Computer-Aided Design of Integrated Circuits and Systems (TCAD), the Design Automa9on Conference (DAC), the Asia and South Pacific Design Automa9on Conference (ASP-DAC), and Interna9onal Conference on Computer Design (ICCD).
Prof. Lee's research is par9cularly impacdul in autonomous systems, decision-making systems, game engines, and vision-AI based robo9c applica9ons. In 2024, Prof. Lee was awarded the MARC Academia- Industry Collabora9on Excellent Research Award by MediaTek Inc., and received the Garmin Scholar Fellowship from Garmin Inc. He was honored with the Academia Sinica Early-Career Inves9gator Research Achievement Award in 2022 and the Ta-You Wu Memorial Award from the Ministry of Science and Technology (MOST) in 2020. These pres9gious awards in Taiwan recognize outstanding achievements in intelligence compu9ng for young researchers. Prof. Lee has garnered numerous accolades, including outstanding research awards, dis9nguished teaching awards, innova9on teaching awards, young scholar research awards, and contribu9on awards from ins9tu9ons such as NVIDIA Deep Learning Ins9tute (DLI), The Taiwan IC Design Society (TICD), the Founda9on for the Advancement of Outstanding Scholarship (FAOS), The Chinese Ins9tute of Electrical Engineering (CIEE), Taiwan Semiconductor Industry Associa9on (TSIA), Ins9tute of Informa9on & Compu9ng Machinery (IICM), and Na9onal Tsing Hua University (NTHU). In the academic community, Prof. Lee will serve as the Program Co-Chair at the Interna9onal Conference on Machine Vision Applica9ons (MVA 2025). He has served as the Area Chairs at NeurIPS 2023-2024, ICLR 2024, and ACML 2024, and as tutorial chair of MVA 2023. His roles have included commiaee member and reviewer at numerous interna9onal and domes9c conferences. He has been session chair and technical program commiaee member mul9ple 9mes at IROS, ASP-DAC, NoCs, ISVLSI, and MVA, and has held various chair roles at different interna9onal conferences. Prof. Lee has served as paper reviewer for NeurIPS, ICML, CVPR, ICLR, AAAI, ICCV, BMVC, ICRA, IROS, IEEE TPAMI, IEEE TVLSI, IEEE TCAD, IEEE ISSCC, and IEEE ASP-DAC
on mul9ple occasions. He was the main organizer of the 3rd, 4th, and 5th Augmented Intelligence and Interac9on (AII) Workshops from 2019 to 2023, and chaired the ACML Workshop on Machine Learning for Mobile Robot Vision and Control (MRVC) in 2021. Prof. Lee served as the director of the NVIDIA-NTHU Joint Innova9on Center from 2023 to 2024, and served as the co-director of the MOST Office for Interna9onal AI Research Collabora9on from 2018 to 2020. From 2024, Prof. Lee serves as the co-director of the NVIDIA- NTU Joint Innova9on Center. Prof. Lee is a professional member of IEEE and ACM.
Contact:
Prof. Chun-Yi Lee
Professor of Computer Science,
Department of Computer Science and Information Engineering, National Taiwan University,
Taipei, Taiwan
Email: cylee@csie.ntu.edu.tw
Prof. Lee's research primarily focuses on deep reinforcement learning (DRL), intelligent robo9cs, computer vision (CV), and parallel compu9ng systems. His contribu9ons include developing key deep learning methodologies for intelligent robo9cs, such as sim-to-real and real-to-sim training, transfer techniques for robo9c policies, digital twins, scene coordinate regression approaches, and domain adapta9on techniques for seman9c segmenta9on models. He has also advanced explora9on approaches for DRL agents, mul9- agent reinforcement learning (MARL) techniques, and genera9ve modeling methodologies, including super- resolu9on and score-based genera9ve models, as well as autonomous naviga9on strategies. His work has been published at major ar9ficial intelligence (AI) conferences such as NeurIPS, CVPR, IJCAI, AAMAS, ICLR, ICML, ECCV, BMVC, CoRL, ICRA, IROS, GTC, MVA, and others. Addi9onally, his research appears in top AI journals, including IEEE Transac9ons on Paaern Recogni9on and Machine Intelligence (TPAMI), IEEE Transac9ons on Neural Networks and Learning Systems (TNNLS), Journal of Machine Learning Research (JMLR), and ACM Transac9ons on Evolu9onary Learning and Op9miza9on (TELO). In the realm of parallel systems, Prof. Lee has introduced innova9ve graphics processing unit (GPU) enhancement methodologies to improve efficiency. His work in this area has been published in IEEE Transac9ons on Very Large Scale Integra9on Systems (TVLSI), IEEE Transac9ons on Computer-Aided Design of Integrated Circuits and Systems (TCAD), the Design Automa9on Conference (DAC), the Asia and South Pacific Design Automa9on Conference (ASP-DAC), and Interna9onal Conference on Computer Design (ICCD).
Prof. Lee's research is par9cularly impacdul in autonomous systems, decision-making systems, game engines, and vision-AI based robo9c applica9ons. In 2024, Prof. Lee was awarded the MARC Academia- Industry Collabora9on Excellent Research Award by MediaTek Inc., and received the Garmin Scholar Fellowship from Garmin Inc. He was honored with the Academia Sinica Early-Career Inves9gator Research Achievement Award in 2022 and the Ta-You Wu Memorial Award from the Ministry of Science and Technology (MOST) in 2020. These pres9gious awards in Taiwan recognize outstanding achievements in intelligence compu9ng for young researchers. Prof. Lee has garnered numerous accolades, including outstanding research awards, dis9nguished teaching awards, innova9on teaching awards, young scholar research awards, and contribu9on awards from ins9tu9ons such as NVIDIA Deep Learning Ins9tute (DLI), The Taiwan IC Design Society (TICD), the Founda9on for the Advancement of Outstanding Scholarship (FAOS), The Chinese Ins9tute of Electrical Engineering (CIEE), Taiwan Semiconductor Industry Associa9on (TSIA), Ins9tute of Informa9on & Compu9ng Machinery (IICM), and Na9onal Tsing Hua University (NTHU). In the academic community, Prof. Lee will serve as the Program Co-Chair at the Interna9onal Conference on Machine Vision Applica9ons (MVA 2025). He has served as the Area Chairs at NeurIPS 2023-2024, ICLR 2024, and ACML 2024, and as tutorial chair of MVA 2023. His roles have included commiaee member and reviewer at numerous interna9onal and domes9c conferences. He has been session chair and technical program commiaee member mul9ple 9mes at IROS, ASP-DAC, NoCs, ISVLSI, and MVA, and has held various chair roles at different interna9onal conferences. Prof. Lee has served as paper reviewer for NeurIPS, ICML, CVPR, ICLR, AAAI, ICCV, BMVC, ICRA, IROS, IEEE TPAMI, IEEE TVLSI, IEEE TCAD, IEEE ISSCC, and IEEE ASP-DAC
on mul9ple occasions. He was the main organizer of the 3rd, 4th, and 5th Augmented Intelligence and Interac9on (AII) Workshops from 2019 to 2023, and chaired the ACML Workshop on Machine Learning for Mobile Robot Vision and Control (MRVC) in 2021. Prof. Lee served as the director of the NVIDIA-NTHU Joint Innova9on Center from 2023 to 2024, and served as the co-director of the MOST Office for Interna9onal AI Research Collabora9on from 2018 to 2020. From 2024, Prof. Lee serves as the co-director of the NVIDIA- NTU Joint Innova9on Center. Prof. Lee is a professional member of IEEE and ACM.
Contact:
Prof. Chun-Yi Lee
Professor of Computer Science,
Department of Computer Science and Information Engineering, National Taiwan University,
Taipei, Taiwan
Email: cylee@csie.ntu.edu.tw