Evaluating Information Retrieval and Recommendation systems, as well as other dataset-trained systems, has proved to be challenging. There are many factors known to affect the results we obtain from an experiment, such as the dataset, metric, data partitions, network initialization, statistics, etc. Many other factors are typically decided upon only once, and researchers just follow the same methodologies in their everyday research: data collection, dataset size, metric design, user models, annotators, statistical analysis, interpretation of results, missing ground truth, etc.

Many of these factors are known to have a large impact on our evaluation experiments, even to the point of questioning the trustworthiness of the results and the conclusions drawn thereof. Research over the past two decades has come up with ways to minimize this impact and allow us to have a more robust understanding of the field. However, how to best deal with others is an open research problem. This PhD project will continue current research to explore ways to increase the reliability of experiments in Information Retrieval and related fields.

You will have the opportunity to address some of the following topics:

• Test collection design.
• Handling of missing judgments.
• Ranking of retrieval systems.
• Statistical analysis of evaluation results.
• Simulation of test collections
• Development of evaluation tools

Research output is expected in the form of papers to be published in leading venues like SIGIR, RecSys, CIKM, ECIR, WSDM, WWW, TOIS or IRJ, software packages and datasets, with a very clear contribution to Open Science and reproducible research.

The successful PhD candidate has an MSc in Statistics, Computer Science, or a closely related field. Solid foundation in one or more of the following disciplines is very much looked for: information retrieval, recommender systems, machine learning, statistical modeling, data analytics. We require programming skills in modern languages, preferably in R. We require fluently spoken and written English. In addition, we are looking for creativity, critical thinking, rigor, independence, and good communication skills.

Fixed-term contract: 4 years.

 TU Delft offers a customisable compensation package, a discount for health insurance and sport memberships, and a monthly work costs contribution. Flexible work schedules can be arranged. An International Children’s Centre offers childcare and an international primary school. Dual Career Services offers support to accompanying partners. Salary and benefits are in accordance with the Collective Labour Agreement for Dutch Universities.
As a PhD candidate you will be enrolled in the TU Delft Graduate School. TU Delft Graduate School provides an inspiring research environment; an excellent team of supervisors, academic staff and a mentor; and a Doctoral Education Programme aimed at developing your transferable, discipline-related and research skills. Please visit www.tudelft.nl/phd for more information.

Technische Universiteit Delft

Delft University of Technology (TU Delft) is a multifaceted institution offering education and carrying out research in the technical sciences at an internationally recognised level. Education, research and design are strongly oriented towards applicability. TU Delft develops technologies for future generations, focusing on sustainability, safety and economic vitality. At TU Delft you will work in an environment where technical sciences and society converge. TU Delft comprises eight faculties, unique laboratories, research institutes and schools.

Faculty Electrical Engineering, Mathematics and Computer Science

The Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS) is known worldwide for its high academic quality and the social relevance of its research programs. The faculty’s excellent facilities accentuate its international position in teaching and research. Within this interdisciplinary and international setting the faculty employs more than 1100 employees, including about 400 graduate students and about 2100 students. Together they work on a broad range of technical innovations in the fields of sustainable energy, telecommunications, microelectronics, embedded systems, computer and software engineering, interactive multimedia and applied mathematics.

The mission of the Department of Intelligent Systems (INSY) is to enable humans and machines to deal with the increasing volume and complexity of data and communications. Within INSY, the Multimedia Computing section focuses on the development and evaluation of algorithms for enriching, accessing, and searching large quantities of data. The focus is on innovative systems that are oriented to the needs of users, enabling satisfying and personalized interactions with large collections of data. The group combines expertise in information retrieval, recommender systems, multimedia, signal processing, social network analysis, and quality of experience. The group is currently expanding as a result of successful funding acquisition, and provides a young and vibrant environment to work in.

The PhD candidate will be part of the Evaluation Lab and work under the supervision of Assistant Professor Julián Urbano (https://julian-urbano.info).