Intelligent systems are believed to have a significant role in addressing pressing issues such as industrial labor, personalized healthcare, and household chores in our daily lives. However, for intelligent systems to be widely accepted, they must be capable of perceiving and interpreting human movements while naturally interacting with users like a human would.

This PhD project has the objective of creating an AI system that can comprehend and model human states and nonverbal behavior to realize empathetic interaction in the wild. We propose to build such systems using Computer Vision for interpreting input signals from humans’ body pose, hand pose, eye gaze, facial expression, and tones as input from the user, and also consider contextual information such as objects and scenes. For this purpose, we must develop nonverbal behavior and multimodal social interaction models as a holistic representation of interaction dynamics. We will explore pre-trained large language models in a visual setting, and investigate training with a self-collected human behavior dataset to construct a Large Contextural and Behavior Model (LCBM). We plan to use Computer Vision and text encoders to perceive the input from the human user and surrounding information as input to the LCBM. The output from LCBM will be used to generate the reaction of the AI agent as the natural response to the human user. Finally, we will deploy the proposed AI system to a real robot to evaluate its practicality together with our industrial partner Honda Research Institute in Japan. We believe that our proposed system will largely increase the use case of AI agents in real-world applications.

We are looking for an enthusiastic, self-motivated person to come work with us on the development of the method for individual modality detection, a multi-camera experimental room, generation of social signals with multiple modalities. The main technical difficulty that the PhD candidate should address is the aggregation of multiple modalities to maximize their positive effect on downstream task prediction.

We offer a dynamic, stimulating, and diverse research environment. You will be advised by Dr. Jan van Gemert and Dr. Xucong Zhang and be a part of the Computer Vision Lab within the Department of Intelligent Systems, which includes researchers working on the methodology of machine learning, and computer vision

• A critical thinker with an open mind to draw upon ideas and research methodologies from different disciplines.
• A strong interest in understanding and developing statistical and machine learning methods applied to human behavior understanding.
• A Master's degree or equivalent, or about to graduate with one, in a relevant computational field such as computer science, computer vision/graphics, artificial intelligence, etc.
• An ability to program in Python
• Proficiency in spoken and written English

Doing a PhD at TU Delft requires English proficiency at a certain level to ensure that the candidate is able to communicate and interact well, participate in English-taught Doctoral Education courses, and write scientific articles and a final thesis. For more details please check the Graduate Schools Admission Requirements.

Fixed-term contract: 4 years.

Doctoral candidates will be offered a 4-year period of employment in principle, but in the form of 2 employment contracts. An initial 1,5 year contract with an official go/no go progress assessment within 15 months. Followed by an additional contract for the remaining 2,5 years assuming everything goes well and performance requirements are met.

Salary and benefits are in accordance with the Collective Labour Agreement for Dutch Universities, increasing from € 2770 per month in the first year to € 3539 in the fourth year. As a PhD candidate you will be enrolled in the TU Delft Graduate School. The TU Delft Graduate School provides an inspiring research environment with an excellent team of supervisors, academic staff and a mentor. The Doctoral Education Programme is aimed at developing your transferable, discipline-related and research skills.

The TU Delft offers a customisable compensation package, discounts on health insurance, and a monthly work costs contribution. Flexible work schedules can be arranged.

For international applicants, TU Delft has the Coming to Delft Service. This service provides information for new international employees to help you prepare the relocation and to settle in the Netherlands. The Coming to Delft Service offers a Dual Career Programme for partners and they organise events to expand your (social) network.

Delft University of Technology

Delft University of Technology is built on strong foundations. As creators of the world-famous Dutch waterworks and pioneers in biotech, TU Delft is a top international university combining science, engineering and design. It delivers world class results in education, research and innovation to address challenges in the areas of energy, climate, mobility, health and digital society. For generations, our engineers have proven to be entrepreneurial problem-solvers, both in business and in a social context.

At TU Delft we embrace diversity as one of our core values and we actively engage to be a university where you feel at home and can flourish. We value different perspectives and qualities. We believe this makes our work more innovative, the TU Delft community more vibrant and the world more just. Together, we imagine, invent and create solutions using technology to have a positive impact on a global scale. That is why we invite you to apply. Your application will receive fair consideration.

Challenge. Change. Impact!

Faculty Electrical Engineering, Mathematics and Computer Science

The Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS) brings together three scientific disciplines. Combined, they reinforce each other and are the driving force behind the technology we all use in our daily lives. Technology such as the electricity grid, which our faculty is helping to make completely sustainable and future-proof. At the same time, we are developing the chips and sensors of the future, whilst also setting the foundations for the software technologies to run on this new generation of equipment – which of course includes AI. Meanwhile we are pushing the limits of applied mathematics, for example mapping out disease processes using single cell data, and using mathematics to simulate gigantic ash plumes after a volcanic eruption. In other words: there is plenty of room at the faculty for ground-breaking research. We educate innovative engineers and have excellent labs and facilities that underline our strong international position. In total, more than 1000 employees and 4,000 students work and study in this innovative environment.

Click here to go to the website of the Faculty of Electrical Engineering, Mathematics and Computer Science.