AI & ML
ProCraft: Instruction-Driven Music Production from Free-Form Text
arXiv cs.SD0 views
arXiv:2611.07423v1 Announce Type: new Abstract: Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoising and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.
Source:https://arxiv.org/abs/2611.07423PreviousAdaptive and High-Fidelity Matching Algorithm for Retrieval-Based Accent ConversionNextRoute-DMD: Distribution Matching Distillation with Task-Routed Mechanism for Speech Foundation Models
Related posts
No Man's Sky『Forsaken』が照らす、僕ら開発者の「見えない闇」とAmblyotubeの二眼レンダリング戦略
0dev.to Programming

I Built an ICU Shift-Handoff Agent with Hindsight That Caught a Fatal Dose
0dev.to Programming
Moving from Conductor: the Conductor alternatives, and why Moontors is the one to pick
0dev.to Git
Your Anthropic bill doesn't fit FOCUS. Here's how to convert it (and the cached-token trap)
0dev.to JavaScript
Export rekordbox playlists to a USB stick without opening rekordbox
0dev.to Rust
7 macOS details behind an AI dictation app written in Rust
0dev.to Rust