In evaluations of tweet representation clustering, advanced language models do not consistently outperform simpler baseline methods. Large-scale pre-trained models from the BERT family (e.g., ALBERT, Sentence-BERT) do not vastly exceed the performance of methods like bag-of-words and TF-IDF. However, XLNet has been observed to perform best for capturing tweet representations, followed closely by the Universal Sentence Encoder (USE), though XLNet's performance remains highly volatile depending on the number of clusters ($k$). XLNet's advantage over BERT may be due to its use of permutation language modeling, which allows token prediction in a random order.