CUDA out of memory in decoding

[Lzhang-hub]

Hi, I am newer to learn icefall,I finished the training of tdnn_lstm_ctc, when run the decode steps, I meet the following error, I change the --max-duration, there are still errors:

we set --max-duration=100 and use Tesla V100-SXM, the GPU info follow:

would you give me some advice？thanks

[CSerV]

The 100 is still big for max-duration. Maybe you can reduce the max-duration to 50, 30 or even less.

[Lzhang-hub]

The 100 is still big for max-duration. Maybe you can reduce the max-duration to 50, 30 or even less.

I have reduce max-duration to 1 ,but the error still exist.

[Lzhang-hub]

@csukuangfj We have use you advices (1) and (3) ,but the problem is not solved. If you can give some other advices, thank you very much!

[danpovey]

You could also mess with the decoding parameters, e.g. reduce the max-active and/or the beam.

…

On Fri, Oct 8, 2021 at 6:41 PM Lzhang-hub ***@***.***> wrote: @csukuangfj <https://github.com/csukuangfj> We have use you advices (1) and (3) ,but the problem is not solved. If you can give some other advices, thank you very much! [image: 企业微信截图_16336894139504] <https://user-images.githubusercontent.com/57925599/136542593-f8c0f1e1-2bc4-44a7-9fa8-bdeb4077d9ea.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO6XVD6EXTWOT4X2TSDUF3DDZANCNFSM5FTIMTGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[csukuangfj]

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states.

---

By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning?

[Lzhang-hub]

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states.

By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning?

Thanks! I will attempt to decode with your advices.

CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool.

[danpovey]

What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding.

…

On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub ***@***.***> wrote: https://github.com/k2-fsa/icefall/blob/adb068eb8242fe79dafce5a100c3fdfad934c7a5/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py#L132-L135 You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdxie]

What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding.
…
On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.***> wrote:

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@danpovey @csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%.
The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs：
##############
2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started
2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2}
2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt
2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda
2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt
2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/epoch-19.pt']
2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception:
CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch)

2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527
2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145
2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18
2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch)

2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753
2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129
2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch)
#########################

we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the https://github.com/kaldi-asr/kaldi/pull/4594 to prune your G.
We now can't pinpoint the cause of the error, so we need help,thanks

[danpovey]

Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK.

…

On Sat, Oct 9, 2021 at 11:21 AM cdxie ***@***.***> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub *@*.***> wrote: https://github.com/k2-fsa/icefall/blob/adb068eb8242fe79dafce5a100c3fdfad934c7a5/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py#L132-L135 You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey <https://github.com/danpovey> @csukuangfj <https://github.com/csukuangfj> ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <http://url> to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdxie]

Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK.
…
On Sat, Oct 9, 2021 at 11:21 AM cdxie @.> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.> wrote:

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey https://github.com/danpovey @csukuangfj https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@danpovey OK, this is the training log of tdnn_lstm_ctc :
tdnn-lstm-ctc-log-train.txt

[danpovey]

Your model did not converge; loss should be something like 0.005, not 0.5. I believe when we ran it, we used --bucketing-sampler=True, that could possibly be the reason. Also we used several GPUs, but that should not really affect convergence I think. (Normally this script converges OK).

…

On Sat, Oct 9, 2021 at 2:17 PM cdxie ***@***.***> wrote: Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK. … <#m_3115122914114341433_> On Sat, Oct 9, 2021 at 11:21 AM cdxie *@*.*> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.*> wrote: https://github.com/k2-fsa/icefall/blob/adb068eb8242fe79dafce5a100c3fdfad934c7a5/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py#L132-L135 You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey <https://github.com/danpovey> https://github.com/danpovey @csukuangfj <https://github.com/csukuangfj> https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594> http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . OK, this is the training log of tdnn_lstm_ctc : tdnn-lstm-ctc-log-train.txt <https://github.com/k2-fsa/icefall/files/7315071/tdnn-lstm-ctc-log-train.txt> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO5Y56FEMGRNZ4QATXDUF7M73ANCNFSM5FTIMTGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdxie]

Your model did not converge; loss should be something like 0.005, not 0.5. I believe when we ran it, we used --bucketing-sampler=True, that could possibly be the reason. Also we used several GPUs, but that should not really affect convergence I think. (Normally this script converges OK).
…
On Sat, Oct 9, 2021 at 2:17 PM cdxie @.*> wrote: Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK. … <#m_3115122914114341433_> On Sat, Oct 9, 2021 at 11:21 AM cdxie @.> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.> wrote:

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey https://github.com/danpovey https://github.com/danpovey @csukuangfj https://github.com/csukuangfj https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594> http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . OK, this is the training log of tdnn_lstm_ctc : tdnn-lstm-ctc-log-train.txt https://github.com/k2-fsa/icefall/files/7315071/tdnn-lstm-ctc-log-train.txt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5Y56FEMGRNZ4QATXDUF7M73ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Thanks, I will modify the parameters and run again.

The GPU device I used is a100 NVIDIA GPU-40G, and single-GPU single-machine. The parameters of the script "./tdnn_lstm_ctc/train.py" are not modified

[danpovey]

And please show us some sample decoding output, it is written to somewhere (aligned output vs. the ref text). I want to see how the model failed. To get 59% WER is unusual; would normally be either 100% or close to 0, I'd expect.

…

On Sat, Oct 9, 2021 at 4:36 PM cdxie ***@***.***> wrote: Your model did not converge; loss should be something like 0.005, not 0.5. I believe when we ran it, we used --bucketing-sampler=True, that could possibly be the reason. Also we used several GPUs, but that should not really affect convergence I think. (Normally this script converges OK). … <#m_7389038400197059205_> On Sat, Oct 9, 2021 at 2:17 PM cdxie *@*.**> wrote: Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK. … <#m_3115122914114341433_> On Sat, Oct 9, 2021 at 11:21 AM cdxie @.> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.*> wrote: https://github.com/k2-fsa/icefall/blob/adb068eb8242fe79dafce5a100c3fdfad934c7a5/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py#L132-L135 You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 <#70> <#70 <#70>> (comment) <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey <https://github.com/danpovey> https://github.com/danpovey https://github.com/danpovey @csukuangfj <https://github.com/csukuangfj> https://github.com/csukuangfj https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594> < kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594>> http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . OK, this is the training log of tdnn_lstm_ctc : tdnn-lstm-ctc-log-train.txt https://github.com/k2-fsa/icefall/files/7315071/tdnn-lstm-ctc-log-train.txt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5Y56FEMGRNZ4QATXDUF7M73ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . Thanks, I will modify the parameters and run again. The GPU device I used is a100 NVIDIA GPU-40G, and single-GPU single-machine. The parameters of the script "./tdnn_lstm_ctc/train.py" are not modified — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO7PDH2BU52NMFTXSL3UF75JNANCNFSM5FTIMTGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cdxie]

And please show us some sample decoding output, it is written to somewhere (aligned output vs. the ref text). I want to see how the model failed. To get 59% WER is unusual; would normally be either 100% or close to 0, I'd expect.
…
On Sat, Oct 9, 2021 at 4:36 PM cdxie @.> wrote: Your model did not converge; loss should be something like 0.005, not 0.5. I believe when we ran it, we used --bucketing-sampler=True, that could possibly be the reason. Also we used several GPUs, but that should not really affect convergence I think. (Normally this script converges OK). … <#m_7389038400197059205_> On Sat, Oct 9, 2021 at 2:17 PM cdxie @.> wrote: Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK. … <#m_3115122914114341433_> On Sat, Oct 9, 2021 at 11:21 AM cdxie @.> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.> wrote:

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 <#70> <#70 <#70>> (comment) <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey https://github.com/danpovey https://github.com/danpovey https://github.com/danpovey @csukuangfj https://github.com/csukuangfj https://github.com/csukuangfj https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594> < kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594>> http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . OK, this is the training log of tdnn_lstm_ctc : tdnn-lstm-ctc-log-train.txt https://github.com/k2-fsa/icefall/files/7315071/tdnn-lstm-ctc-log-train.txt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5Y56FEMGRNZ4QATXDUF7M73ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . Thanks, I will modify the parameters and run again. The GPU device I used is a100 NVIDIA GPU-40G, and single-GPU single-machine. The parameters of the script "./tdnn_lstm_ctc/train.py" are not modified — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO7PDH2BU52NMFTXSL3UF75JNANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

OK, I choose the best results file(lm_scale_0.7) of tdnn-lstm-ctc model, the decoding parameters { 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000 } :

errs-test-clean-lm_scale_0.7.txt
errs-test-other-lm_scale_0.7.txt
recogs-test-clean-lm_scale_0.7.txt
recogs-test-other-lm_scale_0.7.txt
wer-summary-test-clean.txt
wer-summary-test-other.txt

[cdxie]

And please show us some sample decoding output, it is written to somewhere (aligned output vs. the ref text). I want to see how the model failed. To get 59% WER is unusual; would normally be either 100% or close to 0, I'd expect.
…
On Sat, Oct 9, 2021 at 4:36 PM cdxie @.> wrote: Your model did not converge; loss should be something like 0.005, not 0.5. I believe when we ran it, we used --bucketing-sampler=True, that could possibly be the reason. Also we used several GPUs, but that should not really affect convergence I think. (Normally this script converges OK). … <#m_7389038400197059205_> On Sat, Oct 9, 2021 at 2:17 PM cdxie @.> wrote: Hm, can you show the last part of one of the training logs or point to the tensorboard log (tensorfboard dev upload --logdir blah/log)? I wonder whether the model is OK. … <#m_3115122914114341433_> On Sat, Oct 9, 2021 at 11:21 AM cdxie @.> wrote: What do you mean by very poor? Is this your own data, or Librispeech? The model quality and data quality can affect the memory used in decoding. … <#m_4937422915890188941_> On Sat, Oct 9, 2021 at 9:39 AM Lzhang-hub @.> wrote:

icefall/egs/librispeech/ASR/tdnn_lstm_ctc/decode.py

Lines 132 to 135 in adb068e

	"search_beam": 20,
	"output_beam": 5,
	"min_active_states": 30,
	"max_active_states": 10000,

You can reduce search_beam, output_beam, or max_active_states. By the way, does CUDA out of memory abort your decoding process? Does it continue to decode after pruning? Thanks! I will attempt to decode with your advices. CUDA out of memory do not abort my decoding process, the decode can be done, but the results are very pool. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 <#70> <#70 <#70>> (comment) <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO66KEUIGGBQEQCGRCLUF6MN3ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . @danpovey https://github.com/danpovey https://github.com/danpovey https://github.com/danpovey @csukuangfj https://github.com/csukuangfj https://github.com/csukuangfj https://github.com/csukuangfj ,Thanks for your reply, we are newer to learn icefall, we just run the recipes of Librispeech, we finished the training steps, the above errors occured in the decoding steps. The decoding process can finised, but the wer of test-other is 59.41%. The device we used is V100 NVIDIA GPU-32G, and we follow the csukuangfj advices (1) and (3), the above errors still occurs： ############## 2021-10-09 10:38:49,103 INFO [decode.py:387] Decoding started 2021-10-09 10:38:49,241 INFO [decode.py:388] {'exp_dir': PosixPath('tdnn_lstm_ctc/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 80, 'subsampling_factor': 3, 'search_beam': 15, 'output_beam': 5, 'min_active_states': 30, 'max_active_states': 7000, 'use_double_scores': True, 'epoch': 19, 'avg': 5, 'method': 'whole-lattice-rescoring', 'num_paths': 100, 'nbest_scale': 0.5, 'export': False, 'full_libri': True, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 150, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2} 2021-10-09 10:38:50,467 INFO [lexicon.py:113] Loading pre-compiled data/lang_phone/Linv.pt 2021-10-09 10:38:52,312 INFO [decode.py:397] device: cuda 2021-10-09 10:40:48,429 INFO [decode.py:428] Loading pre-compiled G_4_gram.pt 2021-10-09 10:43:25,546 INFO [decode.py:458] averaging ['tdnn_lstm_ctc/exp/ epoch-15.pt', 'tdnn_lstm_ctc/exp/epoch-16.pt', 'tdnn_lstm_ctc/exp/ epoch-17.pt', 'tdnn_lstm_ctc/exp/epoch-18.pt', 'tdnn_lstm_ctc/exp/ epoch-19.pt'] 2021-10-09 10:44:14,941 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 4.38 GiB (GPU 0; 31.75 GiB total capacity; 27.41 GiB already allocated; 365.75 MiB free; 30.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:14,942 INFO [decode.py:732] num_arcs before pruning: 2061527 2021-10-09 10:44:14,977 INFO [decode.py:739] num_arcs after pruning: 113145 2021-10-09 10:44:16,184 INFO [decode.py:336] batch 0/?, cuts processed until now is 18 2021-10-09 10:44:16,944 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.89 GiB already allocated; 4.36 GiB free; 26.23 GiB reserved in total by PyTorch) 2021-10-09 10:44:16,944 INFO [decode.py:732] num_arcs before pruning: 2814753 2021-10-09 10:44:16,982 INFO [decode.py:739] num_arcs after pruning: 120129 2021-10-09 10:44:18,624 INFO [decode.py:731] Caught exception: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 21.80 GiB already allocated; 1.54 GiB free; 29.05 GiB reserved in total by PyTorch) ######################### we reduce search_beam(20->15), max_active_states(10000->7000) a moment ago, the error is same. We suspect the error could be casued by processing G, and we may follow the kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594> < kaldi-asr/kaldi#4594 <kaldi-asr/kaldi#4594>> http://url to prune your G. We now can't pinpoint the cause of the error, so we need help,thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 <#70> (comment) <#70 (comment) <#70 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO74EHY6MLXZINC73RDUF6YLBANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . OK, this is the training log of tdnn_lstm_ctc : tdnn-lstm-ctc-log-train.txt https://github.com/k2-fsa/icefall/files/7315071/tdnn-lstm-ctc-log-train.txt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment) <#70 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5Y56FEMGRNZ4QATXDUF7M73ANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . Thanks, I will modify the parameters and run again. The GPU device I used is a100 NVIDIA GPU-40G, and single-GPU single-machine. The parameters of the script "./tdnn_lstm_ctc/train.py" are not modified — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO7PDH2BU52NMFTXSL3UF75JNANCNFSM5FTIMTGQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@danpovey , I also run the recipes of Librispeech of Confromer CTC, using a100 NVIDIA GPU-40G, and single-GPU single-machine, and no parameters are modified during the training, my model may be not converge according your opinions. Now what should need to change that can make the loss converged?

the last part of the training logs:
###########
2021-10-02 04:08:09,315 INFO [train.py:506] Epoch 34, batch 53make the loss converge
the last part of the Confromer CTC, training logs:
###########
2021-10-02 04:08:09,315 INFO [train.py:506] Epoch 34, batch 53900, batch avg ctc loss 0.0350, batch avg att loss 0.0249, batch avg loss 0.0279, total avg ctc loss: 0.0481, total avg att loss: 0.03900, batch avg ctc loss 0.0350, batch avg att loss 0.0249, batch avg loss 0.0279, total avg ctc loss: 0.0481, total avg att loss: 0.0344, total avg loss: 0.0385, batch size: 15
2021-10-02 04:08:14,123 INFO [train.py:506] Epoch 34, batch 53910, batch avg ctc loss 0.0332, batch avg att loss 0.0225, batch avg loss 0.0257, total avg ctc loss: 0.0476, total avg att loss: 0.0339, total avg loss: 0.0380, batch size: 17
2021-10-02 04:08:19,109 INFO [train.py:506] Epoch 34, batch 53920, batch avg ctc loss 0.0688, batch avg att loss 0.0451, batch avg loss 0.0522, total avg ctc loss: 0.0472, total avg att loss: 0.0334, total avg loss: 0.0375, batch size: 18
2021-10-02 04:08:24,021 INFO [train.py:506] Epoch 34, batch 53930, batch avg ctc loss 0.0530, batch avg att loss 0.0363, batch avg loss 0.0413, total avg ctc loss: 0.0476, total avg att loss: 0.0335, total avg loss: 0.0377, batch size: 15
2021-10-02 04:08:29,207 INFO [train.py:506] Epoch 34, batch 53940, batch avg ctc loss 0.0420, batch avg att loss 0.0304, batch avg loss 0.0339, total avg ctc loss: 0.0473, total avg att loss: 0.0339, total avg loss: 0.0379, batch size: 16
2021-10-02 04:08:34,475 INFO [train.py:506] Epoch 34, batch 53950, batch avg ctc loss 0.0518, batch avg att loss 0.0357, batch avg loss 0.0405, total avg ctc loss: 0.0469, total avg att loss: 0.0335, total avg loss: 0.0376, batch size: 16
2021-10-02 04:08:39,350 INFO [train.py:506] Epoch 34, batch 53960, batch avg ctc loss 0.0602, batch avg att loss 0.0414, batch avg loss 0.0471, total avg ctc loss: 0.0465, total avg att loss: 0.0333, total avg loss: 0.0373, batch size: 13
2021-10-02 04:08:44,708 INFO [train.py:506] Epoch 34, batch 53970, batch avg ctc loss 0.0495, batch avg att loss 0.0328, batch avg loss 0.0378, total avg ctc loss: 0.0462, total avg att loss: 0.0330, total avg loss: 0.0370, batch size: 16
2021-10-02 04:08:49,894 INFO [train.py:506] Epoch 34, batch 53980, batch avg ctc loss 0.0661, batch avg att loss 0.0431, batch avg loss 0.0500, total avg ctc loss: 0.0465, total avg att loss: 0.0331, total avg loss: 0.0371, batch size: 15
2021-10-02 04:08:54,981 INFO [train.py:506] Epoch 34, batch 53990, batch avg ctc loss 0.0351, batch avg att loss 0.0310, batch avg loss 0.0323, total avg ctc loss: 0.0469, total avg att loss: 0.0336, total avg loss: 0.0376, batch size: 17
2021-10-02 04:09:01,103 INFO [train.py:506] Epoch 34, batch 54000, batch avg ctc loss 0.0616, batch avg att loss 0.0432, batch avg loss 0.0487, total avg ctc loss: 0.0466, total avg att loss: 0.0334, total avg loss: 0.0374, batch size: 16
2021-10-02 04:10:01,514 INFO [train.py:565] Epoch 34, valid ctc loss 0.0642,valid att loss 0.0416,valid loss 0.0483, best valid loss: 0.0445 best valid epoch: 22
2021-10-02 04:10:06,173 INFO [train.py:506] Epoch 34, batch 54010, batch avg ctc loss 0.0651, batch avg att loss 0.0448, batch avg loss 0.0509, total avg ctc loss: 0.0551, total avg att loss: 0.0359, total avg loss: 0.0416, batch size: 15
2021-10-02 04:10:12,536 INFO [train.py:506] Epoch 34, batch 54020, batch avg ctc loss 0.0393, batch avg att loss 0.0274, batch avg loss 0.0310, total avg ctc loss: 0.0516, total avg att loss: 0.0342, total avg loss: 0.0394, batch size: 20
2021-10-02 04:10:17,708 INFO [train.py:506] Epoch 34, batch 54030, batch avg ctc loss 0.0668, batch avg att loss 0.0434, batch avg loss 0.0504, total avg ctc loss: 0.0497, total avg att loss: 0.0325, total avg loss: 0.0377, batch size: 15
2021-10-02 04:10:23,342 INFO [train.py:506] Epoch 34, batch 54040, batch avg ctc loss 0.0456, batch avg att loss 0.0283, batch avg loss 0.0335, total avg ctc loss: 0.0484, total avg att loss: 0.0340, total avg loss: 0.0383, batch size: 17
2021-10-02 04:10:28,181 INFO [train.py:506] Epoch 34, batch 54050, batch avg ctc loss 0.0455, batch avg att loss 0.0313, batch avg loss 0.0356, total avg ctc loss: 0.0488, total avg att loss: 0.0339, total avg loss: 0.0384, batch size: 14
2021-10-02 04:10:33,627 INFO [train.py:506] Epoch 34, batch 54060, batch avg ctc loss 0.0549, batch avg att loss 0.1210, batch avg loss 0.1011, total avg ctc loss: 0.0488, total avg att loss: 0.0351, total avg loss: 0.0392, batch size: 18
2021-10-02 04:10:38,395 INFO [train.py:506] Epoch 34, batch 54070, batch avg ctc loss 0.0647, batch avg att loss 0.0357, batch avg loss 0.0444, total avg ctc loss: 0.0486, total avg att loss: 0.0346, total avg loss: 0.0388, batch size: 16
2021-10-02 04:10:43,016 INFO [train.py:506] Epoch 34, batch 54080, batch avg ctc loss 0.0360, batch avg att loss 0.0266, batch avg loss 0.0294, total avg ctc loss: 0.0477, total avg att loss: 0.0338, total avg loss: 0.0380, batch size: 14
2021-10-02 04:10:47,858 INFO [train.py:506] Epoch 34, batch 54090, batch avg ctc loss 0.0496, batch avg att loss 0.0290, batch avg loss 0.0352, total avg ctc loss: 0.0474, total avg att loss: 0.0332, total avg loss: 0.0375, batch size: 15
2021-10-02 04:10:52,855 INFO [train.py:506] Epoch 34, batch 54100, batch avg ctc loss 0.0421, batch avg att loss 0.0288, batch avg loss 0.0328, total avg ctc loss: 0.0477, total avg att loss: 0.0342, total avg loss: 0.0382, batch size: 14
2021-10-02 04:10:53,829 INFO [checkpoint.py:62] Saving checkpoint to conformer_ctc/exp/epoch-34.pt
2021-10-02 04:11:36,298 INFO [train.py:708] Done!
#############