Background

Automatic tuning for virtual singers is one of the applied deep learning directions I have been working on. In order to improve the model’s performance, several approaches were tested during development. One of the more troublesome problems was song audio segmentation: cutting a vocal track so that each sung syllable or character corresponds to its own audio segment.

The goal sounds straightforward: use the waveform information of a song to locate the boundaries of every sung word or character, then split the audio accordingly. In practice, sung audio is much harder to handle than ordinary spoken dialogue. In speech, pauses between words are often clearer and more regular. In singing, the boundaries can be blurred by pitch changes, sustained notes, legato phrasing, accompaniment, breath noise, and expressive timing. Even a human listener may not be able to identify every single character boundary with confidence.

After several rounds of reproduction and experimentation over multiple days, the following segmentation strategies were tested or investigated.

Splitting a Song by Silence

The most direct method is to cut the audio according to silent intervals. A loudness threshold is set; any segment quieter than that threshold is treated as silence. This is also one of the most common approaches for processing spoken audio, mainly because it is fast, simple, and usually reliable in dialogue scenarios.

The Python implementation uses the pydub module:

from pydub import AudioSegment
from pydub.silence import split_on_silence
chunks = split_on_silence(sound,
    # must be silent for at least half a second
    min_silence_len=500,
    # consider it silent if quieter than -16 dBFS
    silence_thresh=-16)

For singing, however, silence-based splitting is often not enough. Many syllables are connected without a clean silent gap, while some pauses do not correspond neatly to word boundaries. The method is still useful as a baseline, but it cannot solve the full segmentation problem by itself.

Using K-Means for Audio Segmentation

Another idea is to use clustering to divide the audio into different segments. In this approach, features extracted from the audio are grouped, and the resulting clusters are used to guide segmentation.

A relatively complete implementation can be found in the py_speech_seg project, which includes Python functions that combine K-means with the VAD method discussed below.

Using K-means alone does not produce reasonable segmentation for this task. It often cuts through the middle of a character or syllable, and it cannot naturally adapt to the expected number of characters in the lyrics. This also makes it difficult to decide the number of clusters, K, in a reliable way.

When VAD is used as a preprocessing step before K-means, the process becomes faster and the results improve. In that setup, the segmentation can often be pushed toward pause regions rather than arbitrary points in the audio. Even so, the number of clusters K still has to be chosen manually based on experimental output. Another unresolved issue is text adhesion: one extracted segment frequently contains several characters instead of just one.

Detecting Intervals with VAD

VAD, or Voice Activity Detection, is designed to determine whether a given audio signal contains human speech. It distinguishes voice segments from background noise or non-speech audio, allowing the two kinds of signals to be processed differently.

In the context of song segmentation, VAD can help identify where vocal activity begins and ends. It is especially useful as a preprocessing step for other methods, because it can remove or mark regions that are unlikely to contain sung voice. But VAD itself does not solve character-level cutting. It detects voice activity, not linguistic boundaries. A continuous sung phrase may still contain many characters with no obvious inactive region between them.

Segmentation with BIC

Another tested method is segmentation based on BIC, the Bayesian Information Criterion. Relevant references include Python implementations of BIC-based speech dialogue segmentation, as well as work on speaker, environment, and channel change detection and clustering via BIC.

Compared with K-means, BIC-based segmentation is slower, but the resulting segments tend to be more accurate. The problem of multiple characters sticking together in the same segment is improved to some extent. Still, the method does not fully meet the requirement of reliably cutting every character or syllable in sung audio.

A Possible BiLSTM-Based Method

The py_speech_seg project also mentions an unfinished machine learning method based on BiLSTM. Since that part is not complete, its feasibility for this specific task remains uncertain.

In theory, a sequence model such as BiLSTM could be a promising direction, because audio segmentation is naturally a temporal problem. But without a complete implementation and validated results, it can only be treated as a possible path for later exploration rather than a current solution.

Current Takeaway

Each method has its own strengths. Silence-based segmentation is simple and fast. K-means can be combined with preprocessing but depends heavily on the choice of K. VAD is useful for detecting vocal activity but cannot locate fine-grained lyric boundaries on its own. BIC gives better segment quality than K-means in some cases, though at a slower speed. The BiLSTM approach may be worth watching, but its practicality is still unknown.

None of these methods, at least in the tested form, satisfies the need for stable character-level segmentation in sung audio. Because of that, the automatic tuning project for virtual singers has to slow down while the audio cutting problem is studied further.

References

• Python pydub implementation for splitting speech by pauses
• py_speech_seg, a relatively complete but unfinished audio segmentation library
• Introductory material on Voice Activity Detection
• Python implementation of BIC-based speech dialogue segmentation
• Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion