Related to Model/Framework(s)
Mask RCNN
Describe the bug
I was trying to adapt my personal fork of Matterport’s Mask RCNN framework to NVIDIA’s example which is updated for tf2.
This is a fatal error, stating: “No such file or directory” in regards to a missing header file numpy/arrayobject.h.
My instincts tell me that this stems from an incompatibility between the versions of pycocotools and numpy that are trying to be compiled and installed as instructed of the Dockerfile included in the MaskRCNN directory.
Sending build context to Docker daemon 650.2kB
Step 1/12 : ARG FROM_IMAGE_NAME=nvcr.io/nvidia/tensorflow:21.02-tf2-py3
Step 2/12 : FROM ${FROM_IMAGE_NAME}
---> 9959a06c488c
Step 3/12 : LABEL model="MaskRCNN"
---> Using cache
---> 5978ab503868
Step 4/12 : ENV DEBIAN_FRONTEND=noninteractive
---> Using cache
---> 168a0b32d30d
Step 5/12 : RUN apt-get update && apt-get install -y libsm6 libxext6 libxrender-dev python3-tk cmake && apt-get clean && rm -rf /var/lib/apt/lists/*
---> Using cache
---> af1b72cfab97
Step 6/12 : RUN pip --no-cache-dir --no-cache install Cython pytest && git clone -b v2.5.0 https://github.com/pybind/pybind11 /opt/pybind11 && cd /opt/pybind11 && cmake . && make install && pip install .
---> Using cache
---> 9aef8974efd0
Step 7/12 : RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && unzip -u protoc-3.3.0-linux-x86_64.zip -d protoc3 && mv protoc3/bin/* /usr/local/bin/ && mv protoc3/include/* /usr/local/include/
---> Using cache
---> e3304b2de323
Step 8/12 : ARG WORKSPACE=/workspace/mrcnn_tf2
---> Using cache
---> dac2c9cfde9f
Step 9/12 : WORKDIR ${WORKSPACE}
---> Using cache
---> 3f33b48d5193
Step 10/12 : ADD requirements.txt .
---> Using cache
---> d11f28bc78e3
Step 11/12 : RUN pip install -r requirements.txt
---> Running in 3c69f70391fe
Collecting pycocotools
Cloning https://github.com/NVIDIA/cocoapi to /tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6
Running command git clone -q https://github.com/NVIDIA/cocoapi /tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6
Collecting git+https://github.com/NVIDIA/dllogger (from -r requirements.txt (line 3))
Cloning https://github.com/NVIDIA/dllogger to /tmp/pip-req-build-isyyso1x
Running command git clone -q https://github.com/NVIDIA/dllogger /tmp/pip-req-build-isyyso1x
Collecting opencv-python-headless
Downloading opencv_python_headless-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (48.3 MB)
Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.8/dist-packages (from pycocotools->-r requirements.txt (line 2)) (52.0.0)
Requirement already satisfied: cython>=0.27.3 in /usr/local/lib/python3.8/dist-packages (from pycocotools->-r requirements.txt (line 2)) (0.29.32)
Collecting matplotlib>=2.1.0
Downloading matplotlib-3.6.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB)
Requirement already satisfied: numpy>=1.19 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=2.1.0->pycocotools->-r requirements.txt (line 2)) (1.19.4)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=2.1.0->pycocotools->-r requirements.txt (line 2)) (2.8.1)
Collecting cycler>=0.10
Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting contourpy>=1.0.1
Downloading contourpy-1.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
Collecting fonttools>=4.22.0
Downloading fonttools-4.37.4-py3-none-any.whl (960 kB)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=2.1.0->pycocotools->-r requirements.txt (line 2)) (20.8)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=2.1.0->pycocotools->-r requirements.txt (line 2)) (2.4.7)
Collecting pillow>=6.2.0
Downloading Pillow-9.2.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.2 MB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7->matplotlib>=2.1.0->pycocotools->-r requirements.txt (line 2)) (1.15.0)
Building wheels for collected packages: DLLogger, pycocotools
Building wheel for DLLogger (setup.py): started
Building wheel for DLLogger (setup.py): finished with status 'done'
Created wheel for DLLogger: filename=DLLogger-1.0.0-py3-none-any.whl size=5657 sha256=b9da1710dc77e821057030043d854388f71c1f75fd0aacd4559dc181e5886291
Stored in directory: /tmp/pip-ephem-wheel-cache-hk7l5mrw/wheels/ad/94/cf/8f3396cb8d62d532695ec557e193fada55cd366e14fd9a02be
Building wheel for pycocotools (setup.py): started
Building wheel for pycocotools (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"'; __file__='"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-bmha15_a
cwd: /tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI
Complete output (68 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-3.8/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-3.8/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-3.8/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-3.8/pycocotools
running build_ext
cythoning pycocotools/_mask.pyx to pycocotools/_mask.c
/usr/local/lib/python3.8/dist-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/pycocotools/_mask.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'pycocotools._mask' extension
creating build/common
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/pycocotools
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.8/dist-packages/numpy/core/include -I../common -I/usr/include/python3.8 -c ../common/maskApi.c -o build/temp.linux-x86_64-3.8/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
../common/maskApi.c: In function ‘rleDecode’:
../common/maskApi.c:92:7: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
92 | for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
| ^~~
../common/maskApi.c:92:49: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
92 | for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
| ^
../common/maskApi.c: In function ‘rleFrPoly’:
../common/maskApi.c:212:3: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
212 | for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
| ^~~
../common/maskApi.c:212:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
212 | for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
| ^
../common/maskApi.c:213:3: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
213 | for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
| ^~~
../common/maskApi.c:213:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
213 | for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
| ^
../common/maskApi.c: In function ‘rleToString’:
../common/maskApi.c:258:7: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
258 | if(more) c |= 0x20; c+=48; s[p++]=c;
| ^~
../common/maskApi.c:258:27: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
258 | if(more) c |= 0x20; c+=48; s[p++]=c;
| ^
../common/maskApi.c: In function ‘rleFrString’:
../common/maskApi.c:266:3: warning: this ‘while’ clause does not guard... [-Wmisleading-indentation]
266 | while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
| ^~~~~
../common/maskApi.c:266:22: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘while’
266 | while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
| ^~~~
../common/maskApi.c:274:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
274 | if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
| ^~
../common/maskApi.c:274:34: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
274 | if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
| ^~~~
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.8/dist-packages/numpy/core/include -I../common -I/usr/include/python3.8 -c pycocotools/_mask.c -o build/temp.linux-x86_64-3.8/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/../common/maskApi.o build/temp.linux-x86_64-3.8/pycocotools/_mask.o -o build/lib.linux-x86_64-3.8/pycocotools/_mask.cpython-38-x86_64-linux-gnu.so
building 'ext' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c pycocotools/ext.cpp -o build/temp.linux-x86_64-3.8/pycocotools/ext.o -O3 -Wall -shared -fopenmp -std=c++17 -fPIC -I/opt/conda/lib/python3.8/site-packages/numpy/core/include -Ipycocotools/
pycocotools/ext.cpp:6:10: fatal error: numpy/arrayobject.h: No such file or directory
6 | #include "numpy/arrayobject.h"
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pycocotools
Running setup.py clean for pycocotools
Successfully built DLLogger
Failed to build pycocotools
Installing collected packages: pillow, kiwisolver, fonttools, cycler, contourpy, matplotlib, pycocotools, opencv-python-headless, DLLogger
Running setup.py install for pycocotools: started
Running setup.py install for pycocotools: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"'; __file__='"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9ht5e0aq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/pycocotools
cwd: /tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI
Complete output (12 lines):
running install
running build
running build_py
running build_ext
skipping 'pycocotools/_mask.c' Cython extension (up-to-date)
building 'ext' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c pycocotools/ext.cpp -o build/temp.linux-x86_64-3.8/pycocotools/ext.o -O3 -Wall -shared -fopenmp -std=c++17 -fPIC -I/opt/conda/lib/python3.8/site-packages/numpy/core/include -Ipycocotools/
pycocotools/ext.cpp:6:10: fatal error: numpy/arrayobject.h: No such file or directory
6 | #include "numpy/arrayobject.h"
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"'; __file__='"'"'/tmp/pip-install-demuscay/pycocotools_ec61179af437406f855ad77b973c25d6/PythonAPI/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9ht5e0aq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/pycocotools Check the logs for full command output.
WARNING: You are using pip version 21.0; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
To Reproduce
Steps to reproduce the behavior:
- Pull the NVIDIA/DeepLearningExamples repo from github.
- Navigate to
./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN - Execute
docker build .
Expected behavior
Expected the Mask RCNN image to build successfully and instantiate an associated container.
Environment
Please provide at least:
- Container version: nvcr.io/nvidia/tensorflow:21.02-tf2-py3
- GPUs in the system: 3080 Laptop (8GB)
- CUDA driver version: 515.76