Internet Archive Music Dataset (IAMD): captioned music segments from Creative Commons audio
By P. Stamatiadis, B. V. Miranda, C. Berger, G. Richard, M. Fontaine, S. Essid
The Internet Archive Music Dataset (IAMD) is a large-scale collection of captioned music segments derived from audio uploads to the Internet Archive. With over 34,000 hours of audio spanning 4.1M captioned 30-second segments extracted from 548k files, it is, to the best of our knowledge, the largest music-caption dataset released to date — more than double the size of the previous largest, JamendoMaxCaps. IAMD is intended to support research on music understanding, music-language modeling, and generative audio systems by providing openly accessible audio together with automatically generated natural language descriptions.
Large music-caption datasets are typically built on private, licensed catalogs, which makes it difficult to reproduce or fairly compare results across studies. IAMD addresses this gap by relying exclusively on content declared to be distributed under Creative Commons licenses. To mitigate the risk of mislabeled licenses, a subset of the data is further cross-referenced against the MusicBrainz database, allowing files linked to entries with registered ISWCs to be flagged as likely copyrighted.
Captions are produced by a two-stage automatic annotation pipeline: a lightweight audio-language model (TinyMU) first generates a base caption for each segment, which an aggregator LLM then refines using Internet Archive metadata and tags predicted by MATPAC++ audio classifiers. This lets the pipeline correct and enrich the initial caption with genre, mood, and instrumentation cues without relying on costly large-scale human annotation. Both objective metrics (CAF-Score) and a subjective listening test show that caption quality holds up at scale, remaining competitive with JamendoMaxCaps and not far behind the human-annotated MusicCaps benchmark.
The dataset is available on HuggingFace, and the code for data collection, license filtering, MusicBrainz cross-referencing, and captioning is released on GitHub to support reproducibility and future extensions.
Companion page
The companion page provides an interactive overview of the dataset composition and of the captioning pipeline, including statistics on the collected audio, caption samples, automatically generated tags, a description of available Internet Archive metadata, and the cross-referencing strategy with MusicBrainz.