Dhingra, B., Shallue, C. J., Norouzi, M., Dai, A. M., & Dahl, G. E. (2018). Embedding text in hyperbolic spaces. arXiv preprint arXiv:1806.04313.

University of Michigan - Ann Arbor

To generate Poincaré embeddings from a natural language text corpus, the author converted the text to a list of directed edges in the format $u,v,weight$, where $v$ is a hypernym of $u$, as input to the Poincaré embeddings algorithm, using the method described by Dhingra et al. (2018).  They constructed a co-occurrence graph $G = \{(u, v)\}$ that includes all pairs of words that occur within a fixed window of each other. Each edge $(u, v)$ in the graph has a weight $w = f^c$, where $f$ is the frequency of the co-occurrence pairs in the corpus and $c$ is a downsampling constant. 

Improved Concept Embeddings for Learning Prerequisite Chains: Generating Poincaré embeddings from natural-language text corpora

Reference for generating Poincaré embeddings from natural-language text corpora

Co-occurrence is symmetric property of pairs of words, meaning if $(u,v,w)$ exists, there will also be an edge $(v,u,w')$, where $w'=w$.

Improved Concept Embeddings for Learning Prerequisite Chains: Co-occurrence

The co-occurrences were calculated using symmetric fixed window sizes of 10, 20, 50, and the entire document. The reason that the authors also considered the entire document is that the prerequisite relationship between two concepts far apart in a file might also exist because all topics in one file are somehow connected.

Improved Concept Embeddings for Learning Prerequisite Chains: Window sizes

Downsampling is a method to deal with unbalanced data in Machine Learning.
For the weights $w = f^c$ , the authors tested downsampling constants 0.3, 0.5, 0.7 and 1, and got the best results with $c = 0.5$.

Learn Before

Related