/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/controller.py:8: UserWarning: horovod was not imported. This will make multi-gpu runs impossible warnings.warn("horovod was not imported. This will make multi-gpu runs impossible") WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/optimizer.py:353: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/optimizer.py:361: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/arch.py:36: The name tf.make_template is deprecated. Please use tf.compat.v1.make_template instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:224: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:236: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2022-03-09 15:22:30.098162: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2022-03-09 15:22:30.101418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: NVIDIA A100-PCIE-40GB major: 8 minor: 0 memoryClockRate(GHz): 1.41 pciBusID: 0000:c3:00.0 2022-03-09 15:22:30.102122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-03-09 15:22:30.104492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-03-09 15:22:30.106738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2022-03-09 15:22:30.107501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2022-03-09 15:22:30.110329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2022-03-09 15:22:30.112511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2022-03-09 15:22:30.118136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2022-03-09 15:22:30.123519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2022-03-09 15:22:30.123957: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2022-03-09 15:22:30.135777: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2350080000 Hz 2022-03-09 15:22:30.135963: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637bbba9c70 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2022-03-09 15:22:30.135994: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2022-03-09 15:22:30.265191: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637bbbb6320 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2022-03-09 15:22:30.265321: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA A100-PCIE-40GB, Compute Capability 8.0 2022-03-09 15:22:30.268081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: NVIDIA A100-PCIE-40GB major: 8 minor: 0 memoryClockRate(GHz): 1.41 pciBusID: 0000:c3:00.0 2022-03-09 15:22:30.268171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-03-09 15:22:30.268203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-03-09 15:22:30.268232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2022-03-09 15:22:30.268260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2022-03-09 15:22:30.268296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2022-03-09 15:22:30.268325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2022-03-09 15:22:30.268353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2022-03-09 15:22:30.273547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2022-03-09 15:22:30.273605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2022-03-09 15:22:30.276812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-03-09 15:22:30.276844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2022-03-09 15:22:30.276869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2022-03-09 15:22:30.282434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 38116 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-PCIE-40GB, pci bus id: 0000:c3:00.0, compute capability: 8.0) WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:175: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.get_or_create_global_step WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/variables.py:241: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/tf_utils/layers.py:34: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/tf_utils/layers.py:34: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/tf_utils/layers.py:307: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/variables.py:218: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/learning_rate.py:65: The name tf.train.exponential_decay is deprecated. Please use tf.compat.v1.train.exponential_decay instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/ops/math_grad.py:1375: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:480: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:241: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead. 2022-03-09 15:27:22.107945: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:97] Unknown compute capability (8, 0) .Defaulting to telling LLVM that we're compiling for sm_35 2022-03-09 15:27:22.255355: I tensorflow/compiler/jit/xla_compilation_cache.cc:238] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:262: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:262: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py:520: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead. 2022-03-09 15:27:23.727203: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:97] Unknown compute capability (8, 0) .Defaulting to telling LLVM that we're compiling for sm_35 2022-03-09 15:27:24.301341: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2022-03-09 15:27:31.730372: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.730520: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.730544: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.730563: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.730582: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.730601: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.730620: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.730639: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.730658: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.730677: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.730694: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.731659: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.731698: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.731733: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.731754: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.731773: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.731792: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.731811: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.731831: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.731849: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.731868: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.731885: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.732791: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.732829: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.732850: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.732869: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.732888: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.732907: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.732926: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.732945: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.732964: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.732983: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.733000: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.733912: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.733949: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.733971: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.733990: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.734010: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.734029: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.734048: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.734067: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.734085: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.734104: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.734132: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.735046: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.735083: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.735104: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.735124: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.735143: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.735162: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.735181: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.735200: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.735219: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.735238: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.735255: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.736180: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.736218: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.736238: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.736257: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.736277: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.736305: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.736327: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.736349: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.736368: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.736386: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.736403: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.737434: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.737472: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.737493: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.737513: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.737532: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.737551: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.737570: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.737600: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.737620: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.737640: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.737657: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.738595: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.738633: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.738654: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.738674: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.738694: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.738713: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.738732: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.738751: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.738770: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.738789: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.738806: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.739726: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1003.53 vs 0.815206 2022-03-09 15:27:31.739763: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.626 vs 0.731407 2022-03-09 15:27:31.739785: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1007.64 vs 0.59356 2022-03-09 15:27:31.739804: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1011.25 vs 0.781374 2022-03-09 15:27:31.739823: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.13 vs 0.0353111 2022-03-09 15:27:31.739842: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1012.78 vs 0.367286 2022-03-09 15:27:31.739861: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 993.421 vs 0.0553332 2022-03-09 15:27:31.739880: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1017.78 vs 0.744868 2022-03-09 15:27:31.739899: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 999.735 vs 0.324988 2022-03-09 15:27:31.739918: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.33 vs 0.892267 2022-03-09 15:27:31.739935: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.743443: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1004.34 vs 0.814724 2022-03-09 15:27:31.743486: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 990.761 vs 0.135477 2022-03-09 15:27:31.743508: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1008.54 vs 0.905792 2022-03-09 15:27:31.743527: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1012.09 vs 0.835009 2022-03-09 15:27:31.743558: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1000.26 vs 0.126987 2022-03-09 15:27:31.743579: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1013.75 vs 0.968868 2022-03-09 15:27:31.743598: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 994.334 vs 0.913376 2022-03-09 15:27:31.743617: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1018 vs 0.221034 2022-03-09 15:27:31.743636: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1000.37 vs 0.632359 2022-03-09 15:27:31.743655: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1007.64 vs 0.308167 2022-03-09 15:27:31.743672: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.852614: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.852660: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.852682: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.852702: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.852722: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.852741: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.852760: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.852779: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.852798: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.852817: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.852834: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.853516: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.853555: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.853575: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.853595: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.853614: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.853633: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.853652: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.853671: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.853690: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.853709: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.853726: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.854364: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.854413: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.854435: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.854455: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.854474: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.854493: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.854512: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.854531: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.854550: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.854569: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.854586: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.855209: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.855246: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.855267: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.855297: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.855318: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.855337: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.855356: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.855375: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.855394: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.855413: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.855430: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.856055: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.856092: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.856114: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.856133: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.856152: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.856171: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.856190: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.856209: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.856227: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.856258: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.856278: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.856917: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.856955: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.856975: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.856995: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.857014: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.857033: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.857052: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.857071: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.857090: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.857109: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.857126: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.857866: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.857904: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.857925: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.857944: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.857964: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.857983: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.858002: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.858021: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.858040: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.858058: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.858076: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.858714: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.858752: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.858773: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.858794: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.858813: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.858832: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.858862: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.858883: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.858902: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.858921: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.858938: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.859588: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.859626: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.859647: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.859666: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.859685: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.859704: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.859723: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.859742: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.859761: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.859780: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.859797: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.861880: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.94 vs 0.814724 2022-03-09 15:27:31.861923: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.15 vs 0.135477 2022-03-09 15:27:31.861944: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1014.13 vs 0.905792 2022-03-09 15:27:31.861964: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.85 vs 0.835009 2022-03-09 15:27:31.861983: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.83 vs 0.126987 2022-03-09 15:27:31.862002: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1008.18 vs 0.968868 2022-03-09 15:27:31.862020: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1011.17 vs 0.913376 2022-03-09 15:27:31.862039: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.48 vs 0.221034 2022-03-09 15:27:31.862058: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1029.18 vs 0.632359 2022-03-09 15:27:31.862076: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.72 vs 0.308167 2022-03-09 15:27:31.862094: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.870638: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.870681: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.870703: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.870734: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.870755: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.870775: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.870794: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.870813: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.870832: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.870851: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.870869: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.871553: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.871591: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.871612: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.871632: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.871651: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.871669: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.871688: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.871707: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.871726: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.871744: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.871761: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.872387: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.872426: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.872447: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.872466: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.872485: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.872504: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.872523: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.872542: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.872561: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.872580: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.872597: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.873231: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.873269: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.873299: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.873319: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.873339: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.873358: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.873377: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.873396: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.873414: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.873433: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.873450: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.874073: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.874110: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.874131: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.874150: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.874169: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.874188: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.874208: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.874226: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.874245: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.874264: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.874282: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.874927: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.874964: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.874985: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.875004: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.875023: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.875042: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.875061: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.875080: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.875109: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.875130: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.875147: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.875900: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.875939: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.875959: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.875979: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.875998: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.876017: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.876036: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.876055: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.876073: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.876092: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.876109: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.876757: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.876795: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.876816: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.876836: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.876855: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.876875: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.876894: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.876913: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.876931: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.876950: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.876968: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:31.877619: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 0: 1011.12 vs 0.764788 2022-03-09 15:27:31.877656: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 1: 1011.02 vs 0.203264 2022-03-09 15:27:31.877677: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 2: 1013.22 vs 0.421069 2022-03-09 15:27:31.877697: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 3: 1021.01 vs 0.804443 2022-03-09 15:27:31.877716: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 4: 1008.71 vs 0.0568132 2022-03-09 15:27:31.877746: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 5: 1007.21 vs 0.0707338 2022-03-09 15:27:31.877766: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 6: 1010.25 vs 0.585747 2022-03-09 15:27:31.877786: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 7: 1001.26 vs 0.883165 2022-03-09 15:27:31.877804: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 8: 1028.54 vs 0.174155 2022-03-09 15:27:31.877823: E tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:432] Difference at 9: 1014.42 vs 0.22722 2022-03-09 15:27:31.877840: E tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:139] Results mismatch between different GEMM algorithms. This is likely a bug/unexpected loss of precision in cuBLAS. 2022-03-09 15:27:32.287487: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:97] Unknown compute capability (8, 0) .Defaulting to telling LLVM that we're compiling for sm_35 2022-03-09 15:27:36.413728: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED CONFIGS: FullyConnectedArch, /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/architecture/fully_connected.py activation_fn: swish layer_size: 256 nr_layers: 6 skip_connections: False weight_norm: True adaptive_activations: False CONFIGS: ExponentialDecayLR, /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/learning_rate.py start_lr: 0.001 end_lr: 0.0 decay_steps: 200 decay_rate: 0.95 CONFIGS: AdamOptimizer, /scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/optimizer.py beta1: 0.9 beta2: 0.999 epsilon: 1e-08 amp: False CONFIGS: HemholtzSolver, helmholtz.py network_dir: ./network_checkpoint_hemholtz initialize_network_dir: added_config_dir: rec_results: True rec_results_cpu: False rec_results_freq: 1000 max_steps: 20000 save_filetypes: vtk,np xla: True inner_norm: 2 outer_norm: 2 save_network_freq: 1000 print_stats_freq: 100 tf_summary_freq: 500 optimizer_params_index: None initialize_network_params: None seq_train_domain: [] config: {'config': ModulusConfig(activation_fn='swish', adaptive_activations=False, added_config_dir='', amp=False, beta1=0.9, beta2=0.999, decay_rate=0.95, decay_steps=200, end_lr=0.0, epsilon=1e-08, initialize_network_dir='', inner_norm=2, layer_size=256, max_steps=20000, network_dir='./network_checkpoint_hemholtz', nr_layers=6, outer_norm=2, rec_results=True, rec_results_cpu=False, rec_results_freq=1000, run_mode='solve', save_filetypes='vtk,np', skip_connections=False, start_lr=0.001, weight_norm=True, xla=True)} arch: lr: optimizer: equations: [] nets: [] diff_nodes: [] Using XLA for optimized graph execution UNROLLING GRAPH: Wall Interior grad calls: 2 calculated: [u__x, u__y] grad calls: 2 calculated: [u__x, u__x__x, u__y, u__y__y] UNROLLING GRAPH: Val Solving for Domain iteration 0 Traceback (most recent call last): File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Unable to launch cuBLAS gemm on stream 0x5637bbc2a4a0 [[{{node cluster_1_1/xla_run}}]] (1) Internal: Unable to launch cuBLAS gemm on stream 0x5637bbc2a4a0 [[{{node cluster_1_1/xla_run}}]] [[cluster_1_1/merge_oidx_20/_1]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "helmholtz.py", line 75, in ctr.run() File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/controller.py", line 91, in run self.solver.solve() File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/solver.py", line 527, in solve train_stats = seq_train_step[domain_index](train_np_var) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/modulus-21.6-py3.7.egg/modulus/variables.py", line 510, in np_function np_outvar_list = sess.run(outvar_placeholders, feed_dict) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/scratch/s.e.j.bennett/anaconda_envs/nvidia_modulus2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Unable to launch cuBLAS gemm on stream 0x5637bbc2a4a0 [[{{node cluster_1_1/xla_run}}]] (1) Internal: Unable to launch cuBLAS gemm on stream 0x5637bbc2a4a0 [[{{node cluster_1_1/xla_run}}]] [[cluster_1_1/merge_oidx_20/_1]] 0 successful operations. 0 derived errors ignored.