Natural Language Processing Engineer    Pattern Recognition Center, Wechat AI, Tencent

Worked on two research topics. Firstly, we propose a new attention mechanism called Mixture of Attention Heads (MoA), which is a combination of Mixture of Experts and Multi-head Attention Mechanism. MoA permits easily scale up the model capacity while constraining the same level of computational cost. We achieved better performance on Machine Translation and Language Modeling tasks compared to SOTA baselines. Secondly, we work on the model editing task in order to efficiently correct the false prediction of a pre-trained language model. We propose a new task named sequential editing task and conceive a novel way to edit the stored knowledge in the pre-trained language model. We add a few new neurons in the last Feed Forward layer of Transformer architecture. The experimental result show that our method outperforms other strong baselines.