Resources for the paper: Neuro-steered music source separation with EEG-based auditory attention decoding and contrastive-NMF
By G. Cantisani

In our paper (Cantisani et al., 2021), we present a novel neuro-steered music source separation framework. In particular, we propose an unsupervised nonnegative matrix factorisation (NMF) variant, named Contrastive-NMF (C-NMF), that separates a target instrument, guided by the user’s selective auditory attention to that instrument, which is tracked in his/her electroencephalographic (EEG) response to music.

The paper

Cantisani, G., Essid, S., & Richard, G. (2021, June). NEURO-STEERED MUSIC SOURCE SEPARATION WITH EEG-BASED AUDITORY ATTENTION DECODING AND CONTRASTIVE-NMF. ICASSP 2021 - 46th International Conference on Acoustics, Speech, and Signal Processing. https://hal.telecom-paris.fr/hal-02978978

Demo

The experiments are designed to evaluate whether the EEG information helps the separation process (C-NMF-e). However, to verify that the improvement is due to the EEG and not to the model’s discriminative capacity, in addiction to the blind NMF (NMF), we built a second baseline which consists of the C-NMF to which meaningless side information is given (C-NMF-r). As the models are entirely unsupervised, the factorized components need to be assigned to each source before applying the Wiener filter. In the two baselines, the components are clustered according to their MFCC similarity. In the case of the C-NMF-e we do not need this as the EEG information automatically identifies and gathers the target instrument components.

c-nmf

In Figure, one can compare the Signal-to-Distortion Ratio (SDR) obtained for different methods and instruments in the dataset. For all the instruments except for the Bass, our model performs significantly better than both the blind NMF and C-NMF-r. The high variance experienced when separating the Bass and the Drums is due to the high variance experienced across different subjects.

The audio examples are taken from the test set which is described in the paper. For the proposed model C-NMF-e, we propose several examples which are related to different subjects and different spatial renderings.


Vocals

Mix
NMF
C-NMF-r
C-NMF-e/mono/sbj 9
C-NMF-e/stereo/sbj 9
C-NMF-e/mono/sbj 5
C-NMF-e/stereo/sbj 5

Bass

Mix
NMF
C-NMF-r
C-NMF-e/mono/sbj 1
C-NMF-e/stereo/sbj 1
C-NMF-e/mono/sbj 4
C-NMF-e/stereo/sbj 4

Drums

Mix
NMF
C-NMF-r
C-NMF-e/mono/sbj 1
C-NMF-e/stereo/sbj 1
C-NMF-e/mono/sbj 4
C-NMF-e/stereo/sbj 4

Guitar

Mix
NMF
C-NMF-r
C-NMF-e/mono/sbj 4
C-NMF-e/stereo/sbj 4
C-NMF-e/mono/sbj 8
C-NMF-e/stereo/sbj 8

loghi