David Rau

About Me

I am a PhD candidate at the University of Amsterdam and the Institute of Logic, Language, and Computation (ILLC) under the supervision of Jaap Kamps. I am working on the intersection of NLP and Information Retrieval. Particularly, I am interested in developing methods for applying LLMs for search at scale. Previously, I obtained an M.Sc. in Artificial Intelligence from the University of Amsterdam, Netherlands, and a B.Sc. in Bioinformatics from the University of Tübingen, Germany. I worked at the Max Planck Institute, Tübingen as a Research Engineer and at the University of Amsterdam as a Machine Learning Engineer.

News

• Dec 2023: Our submission “Query Generation using Large Language Models” to ECIR’24 was accepted to the Reproducibility Track.

• Dec 2023: I’ll be joining Naver Labs Europe for a Research Internship where I will be working on Retrieval Augmented Machine Learning under the supervision of Stéphane Clinchant.

• Nov 2023: My open-source implementation of the Sparsely Gated Mixture of Experts Layer in PyTorch has reached 600 stars on Github.

• Nov 2023: Our submission to interactive Knowledge Acquisition Task (iKAT) @ TREC 2023 became 1st place.

• Nov 2023: Our paper “Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers”, together with Jaap Kamps and Mostafa Dehghani was accepted to TOIS’s Special Section on Efficiency in Neural Information Retrieval.

• Dec 2022: I was invited to give a talk about my work on the role of syntax in neural IR @ Search Engines Amsterdam

• Nov 2022: I joined the British Computer Society’s Information Retrieval Specialist Group (IRSG) as a committee member.

• Jun 2022: Our paper “The Role of Complex NLP in Transformers for Text Ranking?” together with Jaap Kamps was accepted to ICTIR’22 (pdf).

• Dec 2021: Our paper “How Different are Pre-trained Transformers for Text Ranking?” together with Jaap Kamps was accepted to ECIR’22 (pdf).

Publications

• Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers
  David Rau, Mostafa Dehghani, Jaap Kamps
  Special Section on Efficiency in Neural Information Retrieval. TOIS’23.
  
• The Role of Complex NLP in Transformers for Text Ranking?
  David Rau, Jaap Kamps
  ICTIR’22.
  
• How Different are Pre-trained Transformers for Text Ranking?
  David Rau, Jaap Kamps.
  ECIR’22.
  
• On the Realization of Compositionality in Neural Networks
  Baan, J.^, Leible, J.^, Nikolaus, Rau, David^, M.^, Ulmer, D.^, Baumgärtner, T., Hupkes, D., and Bruni, E., ^ Equal contribution.
  ACL BlackboxNLP’19.
  
• A completely reimplemented MPI bioinformatics toolkit with a new HHpred server at its core
  Zimmermann, L^, Stephens, A^, Nam, S. Z^, Rau, David^, Kübler, J^, Lozajic, M., Alva, V., ^Equal contribution.
  Journal of molecular biology’19.
  

Selected Talks

• The Role of Syntax in Transformers for Text Ranking
  Search Engines Amsterdam Meetup
  Amsterdam, Netherlands December 2022 Search Engines Amsterdam

Services

• Committee member of the British Computer Society’s Information Retrieval specialist group (IRSG)
• since 2023: Program Committee at SIGIR
• since 2023: Program Committee at ICTIR
• since 2022] Program Committee at EACL
• since 2022: Program Committee at ECIR
• since 2021: Program Committee ML Reproducibility Challenge (co-located with NeurIPS)

Open Source Contributions

• Tensor2Tensor Tensorflow, Google. Bug fix in the Tensor2Tensor library in the implementation of the Mixture-of-Expert implementation. Commit e7c23fc.
• Implemented a Sparsely-Gated Mixture-of-Expert neural model in PyTorch, 500+ stars (link).
• MPI Bioinformatics Toolkit. Substantial contribution (35k+ lines in Scala, Bash, Python, JavaScript) building the core framework of the open-access high-performance computing platform. (link)

Summer Schools

• Oxford Machine Learning Summer School, Oxford, England. Topics: Representation Learning, Statistical ML, NLP, AI in Healthcare, 2021
• CIFAR DLRL Summer School, MILA, Montreal, Canada. Topics: Deep Learning, Reinforcement Learning , 2020
• Eastern European Machine Learning Summer School, Krakow, Poland. Topics: Machine Learning, Learning Deep Learning, Reinforcement Learning, 2019
• ACM Europe Summer School on Data Science, Athens, Greece. Topics: XAI, Bias, Machine Learning, 2019