Error merging 'cfg.yaml' with schema: Cannot merge DictConfig with ListConfig

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Network: grounding_dino
Tag: nvcr.io/nvidia/tao/tao-toolkit:5.5.0-pyt
Config (could not upload the file):

dataset:
  batch_size: 4
  max_labels: 50
  test_data_sources: null
  train_data_sources:
    image_dir: /root/workspace/german-plates-roboflow/train/
    json_file: /root/workspace/german-plates-roboflow/annotations/odvg/train_only_license_cont_id_odvg.jsonl  # odvg format
    label_map:  /root/workspace/german-plates-roboflow/annotations/odvg/train_only_license_cont_id_odvg_labelmap.json
  val_data_sources:
    image_dir: /root/workspace/german-plates-roboflow/val/
    json_file: /root/workspace/german-plates-roboflow/annotations/contiguous_ids/val_only_license_cont_id.json  # category ids need to be contiguous
  workers: 4
model:
  dec_layers: 6
  dropout_ratio: 0
  enc_layers: 6
  num_feature_levels: 4
  num_queries: 900
  num_select: 300
  pretrained_backbone_path: null
  use_dn: true
results_dir: /root/workspace/experiments/exp_0/
train:
  activation_checkpoint: true
  distributed_strategy: ddp
  # is_dry_run: true
  num_epochs: 10
  num_gpus: 1
  num_nodes: 1
  optim:
    lr: 0.0002
    lr_backbone: 2.0e-05
    lr_steps:
    - 10
  precision: fp32
  validation_interval: 1
  val_interval: 1
  # verbose: true

Running the training from this container directly, no apt updates, pulled unmodified.
(Hydra) gives the following error:

UserWarning: 
'jupyter_cfg.yaml' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
  _run_hydra(
Error merging 'jupyter_cfg.yaml' with schema
Cannot merge DictConfig with ListConfig
    full_key: 
    object_type=ExperimentConfig
2

Can you share the command line?

3

Within the container,

grounding_dino train -e cfg.yaml
4

Could you please set absolute path for cfg.yaml and retry?
# grounding_dino train -e /absolute/path/to/cfg.yaml

5

Error persists with a full path, or when I put the config to a default config dir:

grounding_dino train -e /root/workspace/jupyter_cfg.yaml

Error merging 'jupyter_cfg.yaml' with schema
Cannot merge DictConfig with ListConfig
    full_key: 
    object_type=ExperimentConfig

grounding_dino train -e /usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/grounding_dino/experiment_specs/jupyter_cfg.yaml

Error merging 'jupyter_cfg.yaml' with schema
Cannot merge DictConfig with ListConfig
    full_key: 
    object_type=ExperimentConfig

@Morganh

6

Please make sure the yaml file is available.
You can use -v /localfolder:/dockerpath when you run docker run.

7

It is available. It is physically in the container
@Morganh

8

Please try to follow tao_tutorials/notebooks/tao_launcher_starter_kit/grounding_dino/specs/train.yaml at main · NVIDIA/tao_tutorials · GitHub.
You can delete results_dir.
Then follow the training command in tao_tutorials/notebooks/tao_launcher_starter_kit/grounding_dino/finetune_grounding_dino.ipynb at main · NVIDIA/tao_tutorials · GitHub.

More, please try to put to another places instead of /root/.

9

Error persists:

grounding_dino train -e $SPECS_DIR/jupyter_cfg.yaml results_dir=$RESULTS_DIR

Then, after su taotoolkituser:

drwxrwxr-x 3 taotoolkituser taotoolkituser 4.0K Sep 11 09:06 experiments
-rw-r--r-- 1 taotoolkituser taotoolkituser 1.2K Sep 11 09:06 ttku_cfg.yaml
-rw-r--r-- 1 taotoolkituser taotoolkituser  45M Sep 11 09:01 german-plates-roboflow.tgz
-rw-r--r-- 1 taotoolkituser taotoolkituser 1.2K Sep 11 09:01 jupyter_cfg.yaml
drwxr-xr-x 8 taotoolkituser taotoolkituser 4.0K Sep  4 12:30 german-plates-roboflow

and

grounding_dino train -e /home/taotoolkituser/ttku_cfg.yaml results_dir=/home/taotoolkituser/experiments/e0/

Error merging 'ttku_cfg.yaml' with schema
Cannot merge DictConfig with ListConfig
    full_key: 
    object_type=ExperimentConfig

Error persists @Morganh

10

Please try to follow tao_tutorials/notebooks/tao_launcher_starter_kit/grounding_dino/specs/train.yaml at main · NVIDIA/tao_tutorials · GitHub to add -.

The training dataset sources is LIST_FIELD format. Refer to tao_pytorch_backend/nvidia_tao_pytorch/cv/grounding_dino/config/dataset.py at dc07b02eb78c2eb868315107892b466496e55a0f · NVIDIA/tao_pytorch_backend · GitHub.

11

Thanks, that resolved the issue. Did not notice the typo :/

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.