Reproduce Problem

[JianWang21]

Hi, thanks for this repo!

An error occurred while reproducing the code. Could you please help resolve it? It appears that torchaudio.load() is receiving an MP4 file as input.

Traceback (most recent call last):
File "../src/run_ft.py", line 82, in
train(cavmae_ft, train_loader, val_loader, args)
File "/home/ubuntu/code/mm_detectors/AVFF_wj/src/traintest_ft.py", line 85, in train
for i, (a_input, v_input, labels) in enumerate(train_loader):
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 631, in next
data = self._next_data()
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/ubuntu/code/mm_detectors/AVFF_wj/src/dataloader.py", line 266, in getitem
fbank, frames, label = self._augment_replace(index)
File "/home/ubuntu/code/mm_detectors/AVFF_wj/src/dataloader.py", line 237, in _augment_replace
fbank = self._wav2fbank(video_name_1)
File "/home/ubuntu/code/mm_detectors/AVFF_wj/src/dataloader.py", line 118, in _wav2fbank
waveform, sr = torchaudio.load(filename)
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torchaudio/_backend/utils.py", line 204, in load
backend = dispatcher(uri, format, backend)
File "/home/ubuntu/anaconda3/envs/AVFF/lib/python3.8/site-packages/torchaudio/_backend/utils.py", line 116, in dispatcher
raise RuntimeError(f"Couldn't find appropriate backend to handle uri {uri} and format {format}.")
RuntimeError: Couldn't find appropriate backend to handle uri /root/autodl-tmp/data/FakeAV/video_face/FakeVideo-FakeAudio/Asian_South/men/id03344/00114_id04599_Q60DPi5Nhmk_id07179_wavtolip.mp4 and format None.

[vladhondru25]

I have the sane error. Did you manage to find a solution? I can obviously extract the audio for each video, but I was wondering if the author thought of something else when writing the code.

LE: I managed to solve the issue by setting the backend:
waveform, sr = torchaudio.load(filename, backend="ffmpeg")
Make sure you have ffmpeg installed and soundfile package.