Measure Embedding Relevance: A New Approach to Retrieval Benchmarking

Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right.

We worked on a new metric which seems more proportional to document relevance, and built up a benchmarking dataset to measure embedding models.

Here is an article on the approach: https://medium.com/qonto-way/qontofaq-benchmarking-information-retrieval-acd89600ebe1

and the associated code: https://github.com/qonto/qonto-faq-benchmark