Internet Archive Music Dataset
Demo Page
The Internet Archive Music Dataset (IAMD) is a large-scale collection of captioned music segments derived from audio uploads to the Internet Archive. The dataset is intended to support research on music understanding, music-language modeling, and generative audio systems by providing openly accessible audio together with automatically generated natural language descriptions.
IAMD is constructed exclusively from content declared to be distributed under Creative Commons licenses, with additional cross-referencing against the MusicBrainz database to improve license reliability. Captions are generated using a two-stage automatic annotation pipeline combining audio-language models, Internet Archive metadata, and audio tagging models. We will soon release the full scraping, filtering, license verification, and caption generation pipeline to support reproducible research and future extensions of the dataset.
This companion page provides an overview of the dataset and of the captioning pipeline. It includes statistics on the collected audio, caption samples, statistics on automatically generated tags, a description of available Internet Archive metadata, and further detail on the cross-referencing strategy with MusicBrainz. The bottom of the page contains sample tables with the final structure of the dataset, to be provided soon.
Overview
Click to expand — Details (Scale & Structure, distributions, audio metrics, caption length)
Audio Examples with Captions
Methodology
Captions in IAMD are generated with a two-stage automatic annotation pipeline. First, a base caption is generated from audio using TinyMU, a lightweight audio-language model. These base captions are then enriched using metadata extracted from Internet Archive uploads and segment-level audio-derived tags produced by MATPAC++ linear-probe classifiers. Internet Archive metadata provides coarse-grained contextual information, while MATPAC++ tags provide fine-grained signals on genre, mood, and instrumentation. The base captions and auxiliary information are combined using the Qwen3.5-9B large language model, which refines and aggregates them into the final caption. The prompts used for both base caption generation with TinyMU and final caption aggregation with Qwen3.5-9B are provided below.
Copyright Analysis
Internet Archive Metadata
Native metadata fields provided by Internet Archive uploaders, with field-level descriptions and a concrete example of what the raw metadata looks like for a single item.
Example: Raw Internet Archive Metadata for a Single Item
Illustrative sample of the raw metadata returned by the Internet Archive API for one item
(here, a 2005 French electropop album distributed under CC BY-NC-SA 2.5). Free-form
fields such as description, subject, and notes are
the primary source of human-provided musical context fed to the captioning pipeline.
Automatic Tags (Segment-Level)
Audio Properties
Statistics on the audio files, including codec of the downloaded files and bitrate (when applicable), sample rate, number of channels, and duration. It should be noted that all segments are saved in the same channel configuration as the source file, encoded as mp3 (320 kbps) at 44100 Hz after being extracted from the files.
Sample Tables
The dataset will be made available using three tables: ITEM, FILE, and SEGMENT. The ITEM table will hold item-level information, including Internet Archive original metadata. The FILE table will hold audio information, like sample rate and original format. Finally, the SEGMENT table will hold start and end of each segment in seconds, as well as base caption, tags and confidences, and final captions. An excerpt of the three tables is provided below.