Paper on Transformer positional encodings presented at ICML 2021 as a long talk
By O. Cífka
A paper from the team has been presented at ICML 2021 as a long talk (among 3 % of all submissions):
(Liutkus et al., 2021): Liutkus, A., Cífka, O., Wu, S.-L., Şimşekli, U., yang, yi-H., & Richard, G. (2021, July). Relative Positional Encoding for Transformers with Linear Complexity. ICML 2021 - 38th International Conference on Machine Learning. https://hal.telecom-paris.fr/hal-03256451
In this paper, we propose Stochastic Positional Encoding (SPE), which provably behaves like relative positional encoding while being compatible with linear-complexity Transformers. We do this by drawing a connection between positional encoding and cross-covariance structures of correlated Gaussian processes. We demonstrate the performance of SPE on the Long-Range Arena benchmark and on two music generation tasks.