How do I construct the input for the “Speech-to-Singing Style Transfer” task?

[JepsonWong]

Hi, Thank you for your great work! I am very interested in this work!

From the inference/style_transfer.py, i need to construct the following input for "Zero-Shot Style Transfer" task.

• prompt audio path
• prompt ph, note, note_dur, note_type
• target ph, note, note_dur, note_type

And I think the “Speech-to-Singing Style Transfer” task only requires the following input, is my understanding correct? I think if the input is speech.WAV, there should be no elements such as ph and note? How should I modify the inference/style_transfer.py for “Speech-to-Singing Style Transfer” task?

• prompt audio path
• target ph, note, note_dur, note_type

"Zero-Shot Style Transfer" and “Speech-to-Singing Style Transfer” task in https://tcsinger.github.io/#parallel-style-control.

[AaronZ345]

Hi, to maintain consistency across inputs, my current speech-to-singing inference procedure also requires the following information. Phoneme sequences (ph) are essential, while musical notes (note) are included for standardized input. You can utilize tools like Whisper, pypinyin, and Rosvot to automatically generate this data. I'll be refining the code to streamline the speech processing workflow in the future.

[JepsonWong]

I still don't understand. For the “Speech-to-Singing Style Transfer” task, prompt input is a person's speech audio.
How do I get the information such as prompt ph and notes? Extract it from the speech audio using tools such as MFA and Rosvot?

[JepsonWong]

In other words, for the "Speech-to-Singing Style Transfer" task, the prompt input include not only a person's speech audio, but also a person's singing audio (I can extract prompt ph, note, note_dur, note_type from singing audio)?

[AaronZ345]

In this task, we perform a simplified implementation. Specifically, we replace the singing voice with speech as the prompt input. Consequently, the accompanying auxiliary input information for the speech (e.g., phonemes and notes) must also be provided. You can use ASR to recognize the words in the speech and then leverage language-specific tools (e.g., pypinyin for Chinese) to extract the required input phonemes. Additionally, you can use ROSVOT to directly extract MIDI data from the speech—this is solely for input consistency. There’s no need to provide both singing and speech inputs simultaneously.

If you still wish to apply special processing to speech, such as using paired speech from GTSinger for dedicated STS training, you can exclude the note input from the prompt during training.

[JepsonWong]

ok, thanks.

[JepsonWong]

@AaronZ345 hello，i'm sorry I have a new question.

For “Speech-to-Singing Style Transfer” task, i want to use the inference/style_transfer.py to test the results with the GTSinger dataset.
I changed inp['ref_audio']=item['wav_fn'] to inp['ref_audio']=item['speech_fn']. Is this change correct?
Because I found that in metadata.json, there are only have ph, note, note_dur, note_type for Singing wav file, but not ph, note, note_dur, note_type for Speech wav file.

    inp = {
        # 'text_gen': ,
        # 'note_gen': ,
        # 'note_dur_gen' : ,
        # 'note_type_gen':,
        # 'text_in': ,
        # 'note_in': ,
        # 'note_dur_in' :,
        # 'note_type_in':,
        # 'ref_audio': ,
        # 'ph_durs':
        'ref_name':"English#EN-Alto-2#Mixed_Voice_and_Falsetto#A Thousand Years#Control_Group#0002",
        'gen_name': "Chinese#ZH-Alto-1#Mixed_Voice_and_Falsetto#一次就好#Mixed_Voice_Group#0001",
    }
    
    # use info in metadata.json
    if 'ref_name' in inp:
        items_list = json.load(open(f"{hparams['processed_data_dir']}/metadata.json"))
        for item in items_list:
            if inp['ref_name'] in item['item_name']:
                inp['text_in']=item['ph']
                inp['note_in']=item['ep_pitches']
                inp['note_dur_in'] =item['ep_notedurs']
                inp['note_type_in']=item['ep_types']
                inp['ref_audio']=item['speech_fn'] # inp['ref_audio']=item['wav_fn']
                inp['ph_durs']=item['ph_durs']
                break
        for item in items_list:        
            if inp['gen_name'] in item['item_name']:
                inp['text_gen']=item['ph']
                inp['note_gen']=item['ep_pitches']
                inp['note_dur_gen'] =item['ep_notedurs']
                inp['note_type_gen']=item['ep_types']    
                break

[JepsonWong]

By the way, i can find the note info in the Chinese/ZH-Alto-1/Breathy/不再见/Control_Group/0000.json, but I can't find the relevant information about note in the Chinese/ZH-Alto-1/Breathy/不再见/Paired_Speech_Group/0000.json.

[AaronZ345]

In GTSinger, speech data does not have note info but should have ph info. For note info, you can extract by ROSVOT.

[JasonBourne1998]

By the way, i can find the note info in the Chinese/ZH-Alto-1/Breathy/不再见/Control_Group/0000.json, but I can't find the relevant information about note in the Chinese/ZH-Alto-1/Breathy/不再见/Paired_Speech_Group/0000.json.

Hi @JepsonWong I also tried this and the result is not good, have you figuried out how to edit the code ?