I would like to point out even with Logistic Regression, it yields acceptable resutls.

Johns Hopkins University

This manipulation, also known as face reenactment, consists of modifying the facial expression of the person while maintaining the identity of the person. 
Early methods solve this problem by first modeling the facial frames of the two faces, then adjust the expression parameters frame by frame for the two videos. The more popular techniques include Face2Face and NeuralTextures, which uses GAN to achieve this expression transformation.
We should also be aware of the ethics of using expression swap since it can be used to harm people.

Learn Before