I am trying to setup the hpc environment using HPC SDK on cluster in an attempt to use cuFFTMp but upon compiling c2c sample after following the installation steps for HPC SDK I ran into the following compilation command as
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by /home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64/libcufftMp.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libopen-rte.so.40, needed by /home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libopen-pal.so.40, needed by /home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_delete'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_strerror'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_process_info'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_getcwd'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_pdata_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_first_key_uint32'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64/libcufftMp.so: undefined reference to `nvmlShutdown'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_show_help'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_allocator_component_lookup'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_backtrace_buffer'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_predefined_elem_desc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_compute_relative_locality'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_mpool_base_framework'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_ptr'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_thread_get_self'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_process_name_print'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_cpubind'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_session_dir_cleanup'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_pack'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_single_cpu'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_path_nfs'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_finalize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_errmgr'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_namelist_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_close_components'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_alloc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_load'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_clear_bit'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_unsetenv'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_free'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_local_set'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_find_by_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_create_desc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_send'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create_flag'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_event_poll_rate'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_show_orte_version'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint32'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_test_and_set_item'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_grow_st'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_iszero'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_mutex_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_set_element_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_commit'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_finalize_util'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_yield_when_idle'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_collect_all_data'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append_unique_nosize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_init_util'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_local_get'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_name_wildcard'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_register'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_init_epoch'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `ompi_sync_wait_mt'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_uint32'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_compute_remote_size'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_reset'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_base_exchange'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_ptr'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_close_components'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_built_with_cuda_support'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_in_parallel_debugger'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_initialized'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_tree_print'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_components_close'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_raw'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_set_bit'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cr_reg_coord_callback'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_delay_abort'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_rand'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_unregister'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_free'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_out'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_print_name_args'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_stop'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64/libcufftMp.so: undefined reference to `nvmlInit_v2'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_arch_set_fortran_logical_size'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_memory_base_malloc_init_hook'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_close'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_topology'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_stamp'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_allocator_base_framework'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_set_verbosity'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_is_open'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_read_value'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_register'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_install_dirs'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_thread_self_compare'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_local_arch'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_proc_is_bound'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_compare'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_dup_mpistandard'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_register_framework_params'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_base_async_modex'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_event_users_increment'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_value_enum'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_timer_base_get_cycles'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_isincluded'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convert_process_name_to_string'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_close'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_backtrace_print'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_set_value_enum'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint64'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convert_string_to_process_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_event_flag'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_desc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_convert_process_name_to_string'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cuda_support'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_stream_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_dump'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_obj_by_depth'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_resize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_split'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscriber_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_srand'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_compute_ptypes'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_flags'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscribe_change_info'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_item_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_nthkey'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_find_by_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_select'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_btl_base_framework'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_ptr'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_event_users_decrement'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_setenv'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_is_set_bit'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint64'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_compare_name_fields'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_to_string'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscribe_subscribe'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_item_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_set_value'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_show_help'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_for_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_next_key_uint32'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_write_value'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_proc_applied_binding'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_finalize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_join'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_condition_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_components_close'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get_value'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_register_synonym'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_sort'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_repository_release'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_add'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_verbose_output'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_type_depth'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_finalize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_contain_basic_datatypes'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_abort_print_stack'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_var_register'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_compare_proc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_register_project_frameworks'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_close'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_bool'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_get_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_copy_content_same_ddt'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_create'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_open'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_verbose'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_open'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_session_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `MPIR_being_debugged'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_odls'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_make_version_str'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append_nosize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_set'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_alloc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint32'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64/libcufftMp.so: undefined reference to `nvmlDeviceGetHandleByPciBusId_v2'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_get'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_free'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_unload'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_alloc'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_register_framework_params'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_session_dir_finalize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_initialize'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_convert_string_to_process_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_clone'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_list_item_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_register'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_find'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_init'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_add'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2mapstr'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_get_element_count'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_clone'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_str_to_bool'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_show_opal_version'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_find_by_name'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_valuelen'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_buffer_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_crs_base_state_str'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_recv'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_components_open'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_find_and_set_first_unset_bit'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_start'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cr_output'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_set_item'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_dss'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_timer_base_get_freq'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64/libcufftMp.so: undefined reference to `nvmlDeviceGetP2PStatus'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_dup'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_free'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_rcache_base_framework'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_process_info'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_set_max_size'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_object_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_standalone_operation'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_uses_threads'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_app_t_class'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_ess'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2str'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_data_server_uri'
/home/phd/23/metanir/installedpackages/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_unpack'
collect2: error: ld returned 1 exit status
make: *** [cufftmp_c2c] Error 1
So any directions in this regard will truely be helpful .
While this doesn’t directly answer your question, I’d recommend simplifying things by using the MPI driver, mpicxx, directly. This way you don’t need to worry about getting the paths and libraries correct.
I’ve set my PATH to use the mpicxx from the 23.5 NVHPC SDK’s HPCX MPI, but the OpenMPI4 version should work as well. Also, set your LD_LIBRARY_PATH to the NCCL library. The binary will work without it, but give a warning that NCCL isn’t found.
The solution that you suggested seems to compile but the issue I am facing is that
nvc++-Error-Switch -cudalib with unknown keyword cufftmp
-cudalib[=cublas|cufft:{callback}|cufftw|curand|cusolver|cusparse|cutensor|nvblas|nccl|nvshmem|nvlamath|nvlamath_ilp64]
Add appropriate versions of the CUDA-optimized libraries
For this I have loaded a globally installed version of nvidia hpc SDK 21.9 on the cluster so it seems that cufftmp has been recently added to nvidia hpc SDK .
Ohh I see . But after installing nvhpc 23.3 on cluster and upon compilation I am facing the following error message as
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by /home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libopen-rte.so.40, needed by /home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libopen-pal.so.40, needed by /home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_delete'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_strerror'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_process_info'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_getcwd'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_pdata_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_first_key_uint32'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlShutdown'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_show_help'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_allocator_component_lookup'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_backtrace_buffer'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_predefined_elem_desc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_compute_relative_locality'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_mpool_base_framework'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_ptr'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_thread_get_self'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_process_name_print'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_cpubind'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_session_dir_cleanup'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_pack'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_single_cpu'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_path_nfs'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_finalize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_errmgr'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_namelist_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_close_components'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_alloc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_load'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_clear_bit'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_unsetenv'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_free'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_local_set'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_find_by_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_create_desc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_send'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create_flag'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_event_poll_rate'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_show_orte_version'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint32'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_test_and_set_item'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_grow_st'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_iszero'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_mutex_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_set_element_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_commit'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_finalize_util'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_yield_when_idle'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_collect_all_data'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append_unique_nosize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_init_util'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_local_get'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_name_wildcard'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_register'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_init_epoch'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `ompi_sync_wait_mt'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_uint32'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_compute_remote_size'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_reset'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_base_exchange'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_ptr'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_close_components'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_built_with_cuda_support'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_in_parallel_debugger'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_initialized'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_mpool_base_tree_print'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_components_close'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_raw'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_set_bit'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cr_reg_coord_callback'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_delay_abort'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_rand'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_unregister'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_free'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_out'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_print_name_args'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_stop'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlInit_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_arch_set_fortran_logical_size'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_memory_base_malloc_init_hook'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_close'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_topology'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_stamp'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_allocator_base_framework'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_set_verbosity'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_is_open'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_read_value'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_register'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_install_dirs'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_thread_self_compare'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_local_arch'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_proc_is_bound'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_compare'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_dup_mpistandard'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_register_framework_params'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_base_async_modex'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_event_users_increment'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_value_enum'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_timer_base_get_cycles'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_isincluded'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convert_process_name_to_string'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_close'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_backtrace_print'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_set_value_enum'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint64'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convert_string_to_process_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_set_event_flag'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_desc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_convert_process_name_to_string'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cuda_support'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_stream_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_dump'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_obj_by_depth'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_resize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_split'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscriber_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_srand'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_compute_ptypes'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_flags'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscribe_change_info'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_item_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_nthkey'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_find_by_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_select'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_btl_base_framework'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_ptr'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_event_users_decrement'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_setenv'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_is_set_bit'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint64'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_compare_name_fields'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_to_string'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_infosubscribe_subscribe'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_free_list_item_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_set_value'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_show_help'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_proc_for_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_next_key_uint32'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_write_value'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_proc_applied_binding'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_finalize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_join'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_condition_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_components_close'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_get_value'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_register_synonym'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_sort'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_repository_release'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_add'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_verbose_output'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_get_type_depth'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_finalize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_contain_basic_datatypes'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_abort_print_stack'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_var_register'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_compare_proc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_register_project_frameworks'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_close'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_bool'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_get_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_copy_content_same_ddt'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_create'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_open'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_verbose'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_output_open'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_session_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `MPIR_being_debugged'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_odls'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_make_version_str'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_argv_append_nosize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_set'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_alloc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint32'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetHandleByPciBusId_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_get'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_free'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_value_unload'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_alloc'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_info_register_framework_params'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_session_dir_finalize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_class_initialize'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_util_convert_string_to_process_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hash_table_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_clone'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_component_list_item_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_progress_register'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_find'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_init'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_add'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2mapstr'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_get_element_count'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_clone'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_str_to_bool'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_show_opal_version'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_var_group_find_by_name'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_get_valuelen'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_buffer_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_crs_base_state_str'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_recv'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_framework_components_open'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_datatype_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_find_and_set_first_unset_bit'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_start'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_cr_output'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pointer_array_set_item'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_dss'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_timer_base_get_freq'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetP2PStatus'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_info_dup'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc201_hwloc_bitmap_free'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_rcache_base_framework'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_process_info'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_bitmap_set_max_size'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_object_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_list_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_standalone_operation'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_uses_threads'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_pmix_app_t_class'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_ess'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2str'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `orte_data_server_uri'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib/libmpi.so: undefined reference to `opal_convertor_unpack'
collect2: error: ld returned 1 exit status
make: *** [cufftmp_c2c] Error 1
These symbols are in the libopen-rte.so and libopen.pal.so which for some reason the linker can’t find. They should be located in the same directory as HPCX’s libmpi.so. so double check that they are there and you have read permissions.
How are you linking? Are you still using nvcc w/ -lmpi or switched to using mpicxx as I recommended?
So as you suggested I tried compiling with mpicxx but this time I am getting compilation error as
nvc++-Warning-CUDA_HOME has been deprecated. Please, use NVHPC_CUDA_HOME instead.
"cufftmp_c2c.cu", line 7: catastrophic error: cannot open source file "cufft.h"
#include <cufft.h>
^
1 catastrophic error detected in the compilation of "cufftmp_c2c.cu".
Compilation terminated.
In case I use the already provided make script then a compilation error come up as
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by /home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so, not found (try using -rpath or -rpath-link)
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlShutdown'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlInit_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetHandleByPciBusId_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetP2PStatus'
collect2: error: ld returned 1 exit status
make: *** [cufftmp_c2c] Error 1
Any headers to resolve this issue will certainly help out.
Showing the command line you’re using would be helpful.
The cufftmp_c2c.cu example I point to above doesn’t include “cufft.h” but rather “cufftMp.h”, but I can only replace your error if I don’t add “-cudalib=cufftmp” and hence the path to the cuFFT include files is not added. Did you include this flag or alternately explicitly add the include path: “-I<path/to/cudfft/include>”
For example:
% mpicxx cufftmp_c2c.cu
"cufftmp_c2c.cu", line 1: catastrophic error: cannot open source file "cufftMp.h"
#include <cufftMp.h>
^
1 catastrophic error detected in the compilation of "cufftmp_c2c.cu".
Compilation terminated.
% mpicxx cufftmp_c2c.cu -cudalib=cufftmp
%
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by ....Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so, not found
libnvidia-ml.so gets installed as part of the CUDA driver and not something shipped with the NVHPC SDK.
Why it can’t find it, I’m not sure. I’d first double check that it is indeed installed in /usr/lib64,
Doing a web search for “libnvidia-ml”, I see lots of posts where running “nvidia-smi” from a container gives similar errors. If you run “nvidia-smi”, are you able to see information about your GPU?
Several of the post seem to suggest that it’s a permission issue, but this is out of my area of expertise so I don’t have any recommendation.
Firstly thanks for probing into the issue and as you suggested I have tried with the -cudalib=cufftmp flag but still I am getting the following error
[metanir@login10 c2c]$ ${MPI_HOME}/bin/mpicxx cufftmp_c2c.cu -cudalib=cufftmp
nvc++-Warning-CUDA_HOME has been deprecated. Please, use NVHPC_CUDA_HOME instead.
"cufftmp_c2c.cu", line 7: catastrophic error: cannot open source file "cufft.h"
#include <cufft.h>
^
1 catastrophic error detected in the compilation of "cufftmp_c2c.cu".
Compilation terminated.
Secondly, I couldn’t access /usr/lib64 since I am using a login node into my cluster as well as nvidia-smi doesn’t work for me which suggests a similar issue as you suspected to be that of “permissions” for which I plan to compile the code on compute node to get the permissions issue resolved although I haven’t attempted that but thanks once again for pointing out the relevant direction into which I can look into to resolve this issue.
Thanks for suggesting that but it seems the issue still persists and here is the output after running the make command
[metanir@login07 c2c]$ make
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/cuda/bin/nvcc cufftmp_c2c.cu -o cufftmp_c2c -std=c++17 --generate-code arch=compute_70,code=sm_70 --generate-code arch=compute_80,code=sm_80 -I/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/math_libs/include/cufftmp -I/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/include -I/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/include -lcuda -L/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/math_libs/lib64 -L/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib -lcufftMp -lnvshmem_device -lnvshmem_host -L/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib -lmpi -L/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/cuda/11.8/lib64/stubs
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by /home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so, not found (try using -rpath or -rpath-link)
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlShutdown'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlInit_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetHandleByPciBusId_v2'
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/nvshmem/lib/libnvshmem_host.so: undefined reference to `nvmlDeviceGetP2PStatus'
collect2: error: ld returned 1 exit status
make: *** [cufftmp_c2c] Error 1
Also I added the suggested path to LD_LIBRARY_PATH and upon querying I get the output as
[metanir@login07 c2c]$ echo ${LD_LIBRARY_PATH}
/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/compilers/lib:/home-ext/apps/client_apps/gcc/gcc1120/lib:/home-ext/apps/client_apps/gcc/gcc1120/lib64:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/hpcx/latest/ompi/lib:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/cuda/lib64:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/comm_libs/12.0/nccl/lib:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-11.2.0/nvhpc-23.3-h6ipjj3yh2p7ed4j3l7poukpjx5wmrpz/Linux_x86_64/23.3/cuda/lib64/stubs
So any headers regarding what is going wrong will certainly be helpful.
Thanks in Advance
Pushkar
Thanks mfatica I am able to successfully compile the code but getting a runtime error as
cufftmp_c2c: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
cufftmp_c2c: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[34366,1],1]
Exit code: 127
--------------------------------------------------------------------------
make: *** [run] Error 127
Thanks @MatColgrove and @mfatica for helping out with this installation which I believe is successfully working on cluster and as @MatColgrove suggested I also realized that I need to run the c2c application on compute node instead of login node as the presently compiled application with stubs library simply delays the need for libcuda.so.1 and libnvidia-ml.so.1 until runtime and loader is once again supposed to look for these libraries in order to resolve all the symbols for which I need to have access to NVIDIA driver which can only be accessed on compute node.