Paper on Transformer positional encodings presented at ICML 2021 as a long talk
By O. Cífka

A paper from the team has been presented at ICML 2021 as a long talk (among 3 % of all submissions):

(Liutkus et al., 2021): Liutkus, A., Cífka, O., Wu, S.-L., Şimşekli, U., Yang, Y.-H., & Richard, G. (2021). Relative Positional Encoding for Transformers with Linear Complexity. ICML 2021 - 38th International Conference on Machine Learning, Proceedings of Machine Learning Research(139), 7067–7079.

In this paper, we propose Stochastic Positional Encoding (SPE), which provably behaves like relative positional encoding while being compatible with linear-complexity Transformers. We do this by drawing a connection between positional encoding and cross-covariance structures of correlated Gaussian processes. We demonstrate the performance of SPE on the Long-Range Arena benchmark and on two music generation tasks.

Additional resources for the paper are available on its companion website and GitHub repository. The recorded talk is included below.