Repository navigation
Training strategy for Zipformer using fp16 ?? #1461
Replies: 4 comments 2 replies
|
Are you using single GPU and max-duration=300? The gradient noise might be large with such a small batch size. You could try a smaller base-lr, like 0.025, and keep lr_batch/lr_epoch unchanged. Usually you don't need to tune the Balancer and Whitener configurations. |
|
I have the same issue here. Have you fixed it yet? |
|
Another option is to use the accum_grad option, see icefall/egs/librispeech/SSL/zipformer/pretrain.py Lines 580 to 586 in 1c388ad It simulates a larger batch size and allows you to use a bigger base_lr (while taking longer to train). You might need to implement that option in the librispeech ASR recipe. If you can make a PR for that and we will be very happy to review it. |
|
Your code is quite complicated and I suspect the combination of the grad accumulation with grad-scaling may have bugs in it but I dont have time to check. if you run with the --inf-check=True option it should give more information about where the infinity is coming from. look for the first time the word "finite" appears in the log (there will be a lot of additional warnings after the first one for each "bad frame" because the infinities propagate back into earlier layers of the network). |

Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
I am a student attempting to reproduce the results of Zipformer on Librispeech 100h, facing limitations in hardware resources that prevent me from using the recommended configuration. Due to these constraints, I have reduced the batch size (max_duration) to 300, as opposed to the recommended 1000. However, I am struggling to find the appropriate configuration for Eden.
Following the training strategy that suggests decreasing the learning rate by √k times when the batch size decreases by k times, I initially set the base_lr to 0.03, keeping other configurations at their default values. But the training process diverges. Despite attempts to adjust lr_batches, lr_epochs (3.5-6), and base_lr (0.3-0.45), it's still not working. Notably, the training process encounters divergence when the batch_count is around 700-900, leading to 'parameter domination' issues in the embed_conv and some attention modules. I attach some log information below.



In an effort to address this, I attempted to reduce the gradient scale of the layers experiencing 'parameter domination,' but this proved ineffective."
I have few questions:
Thank you.
All reactions