In the model unlike to the original transformer model masking is applied to all multi-head attention layers. As at this problem only information from previous interactions are needed -> $e_1, \cdots e_n$ and $l_1, \cdots, l_{n-1}$, that's why masking is used for all the layers.


San Diego State University

1. Input Representation
2. Model Description
3. Subsequent Masking

Propose Method (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

As an input we have following features (here $e_i$ stands for question metadata and $l_i$ for the response metadata and their embedding vectors look as
$$e_i = emb_e (id_i, c_i, st_i, p_i, sp_i)$$
$$l_i = emb_l(r_i,, et_i, st_i ,iot_i,d_i, p_i, sp_i)$$
):



Input Representation (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

The  proposed model has similar structure as the transformer, consists of n stacked encoders and decoders. Encoder consists of self-attention and feedforward layers, after each of these layer there is a residual connection and layer normalization. Similarly, decoders contain the same layers and additionally fully connected layer (for prediction). Encoder and decoders have distinctive inputs, for the encoder we have $e_i$s, question metadata features and for the decoders we have response features shifted by one -> $(S, l_1, \cdots, l_{n-1})$ and the output of encoder. In their model, the authors use MultiHeadAttention and upper triangular masking. 

Learn Before

Related