Paper on Transformer positional encodings presented at ICML 2021 as a long talk
By O. Cífka

A paper from the team has been presented at ICML 2021 as a long talk (among 3 % of all submissions):

(Liutkus et al., 2021): Liutkus, A., Cífka, O., Wu, S.-L., Şimşekli, U., Yang, Y.-H., & Richard, G. (2021). Relative Positional Encoding for Transformers with Linear Complexity. ICML 2021 - 38th International Conference on Machine Learning, Proceedings of Machine Learning Research(139), 7067–7079. https://telecom-paris.hal.science/hal-03256451

In this paper, we propose Stochastic Positional Encoding (SPE), which provably behaves like relative positional encoding while being compatible with linear-complexity Transformers. We do this by drawing a connection between positional encoding and cross-covariance structures of correlated Gaussian processes. We demonstrate the performance of SPE on the Long-Range Arena benchmark and on two music generation tasks.

Additional resources for the paper are available on its companion website and GitHub repository. The recorded talk is included below.