Specify more frequent checkpoints

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Is there a setting to cause more frequent checkpoints (than one per epoch)? I have several million examples in my manifest and would like to be able to preserve training and resume before end-of-epoch.

3

Which network did you run?

4

The pre-trained model (-m) used was the speechtotext_english_quartznet.tlt

I’m not sure if that is what you mean by ‘network’.

5

Thanks.So you are running tlt speech_to_text .

6

Ah, yes. tlt speech_to_text finetune

7

Evaluation on ckpt file is not supported yet.
Internal team will check it.

8

On the the tlt speech_to_text finetune (task and subtask), I’d like to know if more frequent checkpoints can be specified (than once per epoch)?

9

Do you mean you want to get more checkpoints files?

10

Yes, is there a setting for intra-epoch checkpoints? For example, setting something like ckpt_interval_minutes so that checkpoints could be written every N minutes.

11

From looking at the pytorch lightning documentation there is an option to specify checkpoints every n steps.
And, looking at the logging about the pre-trained model’s Experiment configuration: there is a section where this perhaps can be modified via the spec yaml, for example:

exp_manager:
  checkpoint_callback_params:
    save_top_k: 5
    every_n_train_steps: 7200

I’ll test once I have some free GPUs.

12

Currently, it is not supported yet.

13

I gave it a try anyhow this is the error:

Error merging 'exp_finetune.yaml' with schema                                 
Key 'every_n_train_steps' not in 'CallbackParams'                             
        full_key: exp_manager.checkpoint_callback_params.every_n_train_steps  
        reference_type=Optional[CallbackParams]                               
        object_type=CallbackParams
15

Next TLT release will support saving tlt model every N training steps.

1 Like