SBERT
SBERT, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations ( 65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.
Keywords for this software
References in zbMATH (referenced in 6 articles , 1 standard article )
Showing results 1 to 6 of 6.
Sorted by year (- Koto, Fajri; Baldwin, Timothy; Lau, Jey Han: FFCI: a framework for interpretable automatic evaluation of summarization (2022)
- Pakzad, Atefe; Analoui, Morteza: A word selection method for producing interpretable distributional semantic word vectors (2021)
- Tripathy, Jatin Karthik; Sethuraman, Sibi Chakkaravarthy; Cruz, Meenalosini Vimal; Namburu, Anupama; P., Mangalraj; R., Nandha Kumar; S., Sudhakar Ilango; Vijayakumar, Vaidehi: Comprehensive analysis of embeddings and pre-training in NLP (2021)
- Zhang, Meishan; Li, Zhenghua; Fu, Guohong; Zhang, Min: Dependency-based syntax-aware word representations (2021)
- Jialun Cao, Meiziniu Li, Yeting Li, Ming Wen, Shing-Chi Cheung: SemMT: A Semantic-based Testing Approach for Machine Translation Systems (2020) arXiv
- Nils Reimers, Iryna Gurevych: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) arXiv