Take too much time twhen loading checkpoint shards

[maxin-cn]

Describe the bug

I tried to use from_pretrained function in Latte to automatically download related models from hugginface, but it took a long time to load the T5 model:

https://github.com/Vchitect/Latte/blob/main/sample/sample_t2v.py#L36

I also tried to download the model manually using wget and trained it with the from_pretrained function. It loaded in about 2s.

Reproduction

git clone https://github.com/Vchitect/Latte
cd Latte

conda env create -f environment.yml
conda activate latte

bash sample/t2v.sh

Logs

No response

System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• diffusers version: 0.24.0
• Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.11.8
• PyTorch version (GPU?): 2.2.1 (True)
• Huggingface_hub version: 0.22.2
• Transformers version: 4.39.3
• Accelerate version: 0.29.2
• xFormers version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[sayakpaul]

It's not a diffusers related issue. I think it's probably on the Hub side, so I am gonna tag @Wauplin here and move this to Discussions.

[sayakpaul]

And also, you should probably open this issue on the official repository of Latte.

[Wauplin]

Hi @maxin-cn, what is the exact problem here? Is it that wget took 2s to download the files while diffusers/huggingface_hub took 15s to download them? Or is it that once the files were downloaded it took 2s to load the weights in one case compared to 15s on the other case? If you re-run your code, can you confirm the difference still exists and that it is not a transient error caused by something else on your machine? Thanks in advance for the details!

[Wauplin]

Just noticed you downloaded 20GB of data so I assume the 2s/15s diff is not for downloading them.

[maxin-cn]

Hi @maxin-cn, what is the exact problem here? Is it that wget took 2s to download the files while diffusers/huggingface_hub took 15s

Hi, @Wauplin, thanks for your reply. This was a phenomenon that I later identified as the inability to load a huggingface T5 cache file with the from_pretrained function on the cluster I was using.

The cache file is:

And the code is:
https://github.com/Vchitect/Latte/blob/5f0fbed8bfa112cdc979450dded03243faee025f/sample/sample_t2x.py#L36

The T5 file is:
https://huggingface.co/maxin-cn/LatteT2V/tree/main/text_encoder

[Wauplin]

When you say "identified", does it mean it also solved? When you opened the issue it was about a timing issue, not a "not able to load checkpoints" issue. If the initial problem is solved, we can close this discussion. If you are struggling with another issue, please open a new issue/discussion in the corresponding repo.

[maxin-cn]

When you say "identified", does it mean it also solved? When you opened the issue it was about a timing issue, not a "not able to load checkpoints" issue. If the initial problem is solved, we can close this discussion. If you are struggling with another issue, please open a new issue/discussion in the corresponding repo.

Thanks for your reply.