In evaluations of tweet representation clustering, advanced language models do not consistently outperform simpler baseline methods. Large-scale pre-trained models from the BERT family (e.g., ALBERT, Sentence-BERT) do not vastly exceed the performance of methods like bag-of-words and TF-IDF. However, XLNet has been observed to perform best for capturing tweet representations, followed closely by the Universal Sentence Encoder (USE), though XLNet's performance remains highly volatile depending on the number of clusters ($$k$$). XLNet's advantage over BERT may be due to its use of permutation language modeling, which allows token prediction in a random order.

Performance of Text Representation Models on Tweet Clustering

Text representation methods convert text (words, tweets, or documents) into numerical vectors for downstream NLP tasks such as clustering or classification. Common methods include: bag-of-words, TF-IDF, LDA, word2vec, doc2vec, GloVe, fastText, Tweet2vec, Universal Sentence Encoder, ELMo, BERT, ALBERT, XLNet, and Sentence-BERT.

University of Michigan - Ann Arbor

Google

Done with the use of large scale pre-trained language models that use Transformer-based architecture 

Learn Before

Related

Learn After