Extract sound event labels from the queried class (Sound Event Detection/SED) using SAM Audio

[konsgken]

Hi,

I would like to ask if SAM Audio can also extract the sound event labels of the queried class from the original sound and if so, how?

Kind regards.

[chevalierNoir]

@konsgken could you elaborate on that? If your query is "dog barking", what label do you want to extract?

[konsgken]

Hi @chevalierNoir, thanks for your answer. If the query is "dog barking" (separated sound query), I want for each timestep to predict/visualize the dog barking class in the original sound. You can see an example of what I mean in the image below (reference):

From my understanding if the model is capable of separating/isolating sounds, it should also be capable to provide the class label of the isolated sound from the original sound.

Thank you for your time and your support.

[jjmlovesgit]

The model can segment a named sound... It will not identify all sounds present with a single prompt... If you gave it a list of sounds to diarize and ask it to label each.... It can do that by segmenting each "Prompt" one at a time. It cannot be prompted to segment and label all sounds in a track at once ... You can chain the output to be your next input by using the non segmented residual as the next input to remove and diarize a series of sound one by one until no sound is left.

[konsgken]

@jjmlovesgit, thank you for your answer. Is it possible to let me know how I can extract the segmented labels for the single prompt you are mentioning ?

[jjmlovesgit]

The Prompt is the Sound label in this strategy. If i take a movie trailer and i process the .mp4 file with a series of prompts: male voice, female voice, car, gunshots, etc. The result is a set of individual files with the segmented prompt/label audio that can be analyzed to produce a table of start and end times for the given sound label across the track. You do this for each prompt/label result file to label all sounds in each track composing the original. It does require you pick the sounds to analyze and aggregate them.

[chevalierNoir]

@konsgken you can check PE-A-Frame for timespan extraction

[jjmlovesgit]

Quick example:

See the temporal table in the UI representing event_ID segments for the target prompt/label sound:

TABLE.mp4

You could comma separate the prompts and do a loop on each of the prompts and produce a master table. You would suffix the event IDs with prompt label's and organize them temporally in the table to get a time series classified view of the audio segment.

[jjmlovesgit]

To self-label:

To operate classification without any manual prompt input from you:

• Discovery Phase: The engine scans the 10-second segment for any energy bursts that cross a minimal threshold.

• Modulo Analysis: For each burst, it extracts a "Clone" and runs the classify_with_confidence prompt test.

• Classification: It scores the audio against an internal list (Inherent list due to model training) for sound label aka. Speech, Guitar, Gunshot, Siren, Engine, etc. and picks the most likely match.

• Table Generation: It populates your Temporal Activity Map with the discovered labels (e.g., EVT-001 [ALARM]).