[b]Hi.
I am running Python code on TX2 that has been checked for normal operation on TX1.
However, TX2 seems to have a difference in the way the kernel is upgraded and the memory is used.
The code below is my code written in python (pycuda).
I have confirmed the normal operation in TX1 with the code to generate the Fresnel hologram.
This is not the only error. I felt that the overall memory management mechanism was different in TX2, unlike TX1.
Anyone have a way to solve this?
Codes here.
You do not have to worry about other parts.
The size of fx, fy, Gbp, and Gfp in this function is (5184x3888). The specific error point is cuda.memcpy_dtoh( ).[/b]
def cuda_fd_hologram(fx, fy, Nx, Ny, k, dz, lamb):
"""
The Fresnel Diffraction hologram making function.
"""
gbp = np.zeros((Ny, Nx), dtype=np.complex64)
gfp = np.zeros((Ny, Nx), dtype=np.complex64)
y, x = fx.shape
gy, gx = gbp.shape
blkdim = 32
grid_x = gx/blkdim+1
grid_y = gy/blkdim+1
fx = np.reshape(fx, y*x)
fy = np.reshape(fy, y*x)
gbp = np.reshape(gbp, gy*gx)
gfp = np.reshape(gfp, gy*gx)
_fx = cuda.mem_alloc(fx.size * fx.dtype.itemsize)
_fy = cuda.mem_alloc(fy.size * fy.dtype.itemsize)
_gbp = cuda.mem_alloc(gbp.size * gbp.dtype.itemsize)
_gfp = cuda.mem_alloc(gfp.size * gfp.dtype.itemsize)
cuda.memcpy_htod(_fx, np.float32(fx))
cuda.memcpy_htod(_fy, np.float32(fy))
cuda.memcpy_htod(_gbp, np.complex64(gbp))
cuda.memcpy_htod(_gfp, np.complex64(gfp))
func = mod.get_function("kernel_fd")
func(_fx, _fy, _gbp, _gfp, np.float32(k), np.float32(dz), np.float32(lamb), \
np.int32(x), np.int32(y), block=(blkdim,blkdim,1), grid=(grid_x,grid_y))
cuda.memcpy_dtoh(gbp, _gbp)
cuda.memcpy_dtoh(gfp, _gfp)
gbp = np.reshape(gbp, (y, x))
gfp = np.reshape(gfp, (y, x))
del _fx, _fy, _gbp, _gfp, fx, fy
return gbp, gfp
Errors here
Traceback (most recent call last):
File "test.py", line 806, in <module>
running()
File "test.py", line 210, in running
test(data, ref)
File "test.py", line 483, in reconstruction
Gbp, Gfp = cuda_fd_hologram(fx, fy, Nx, Ny, k, dz, lamb)
File "test.py", line 254, in cuda_fd_hologram
cuda.memcpy_dtoh(gbp, _gbp)
pycuda._driver.LaunchError: cuMemcpyDtoH failed: unspecified launch failure
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: unspecified launch failure
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: unspecified launch failure
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: unspecified launch failure
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: unspecified launch failure
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuModuleUnload failed: unspecified launch failure
Finally, the output of dmesg.
[ 1753.520306] arm-smmu 12000000.iommu: Unhandled context fault: iova=0x2e3c6000, fsynr=0x20013, cb=19, sid=16(0x10 - GPU), pgd=267e63003, pud=267e63003, pmd=d5ae5003, pte=0
[ 1753.535534] gk20a 17000000.gp10b: gk20a_fifo_handle_pbdma_intr: pbdma_intr_0(0):0x00004000 PBH: 20040718 SHADOW: 200381c5 M0: 8000001d 8004041d 80040715 00000000
[ 1753.549953] gk20a 17000000.gp10b: gk20a_set_error_notifier: error notifier set to 32 for ch 505
[ 1753.559334] gk20a 17000000.gp10b: gk20a_fifo_set_ctx_mmu_error_tsg: TSG 4 generated a mmu fault
[ 1753.568052] ---- mlocks ----
[ 1753.568079] ---- syncpts ----
[ 1753.568091] id 4 (disp_d) min 8 max 8 refs 1 (previous client : )
[ 1753.568095] id 5 (disp_e) min 1 max 1 refs 1 (previous client : )
[ 1753.568101] id 6 (disp_f) min 1 max 1 refs 1 (previous client : )
[ 1753.568105] id 7 (vblank1) min 19065 max 0 refs 1 (previous client : )
[ 1753.568119] id 18 (17000000.gp10b_507) min 7134 max 7134 refs 1 (previous client : )
[ 1753.568123] id 19 (17000000.gp10b_506) min 4 max 4 refs 1 (previous client : )
[ 1753.568130] id 24 (17000000.gp10b_504) min 12 max 12 refs 1 (previous client : 17000000.gp10b_501)
[ 1753.568538] ---- channels ----
[ 1753.568548]
channel 0 - 15810000.se
[ 1753.568551] NvHost basic channel registers:
[ 1753.568555] CMDFIFO_STAT_0: 00002040
[ 1753.568576] CMDFIFO_RDATA_0: 02c08a00
[ 1753.568586] CMDP_OFFSET_0: 00000000
[ 1753.568593] CMDP_CLASS_0: 00000000
[ 1753.568598] CHANNELSTAT_0: 00000000
[ 1753.568600] The CDMA sync queue is empty.
[ 1753.568606]
channel 1 - 15820000.se
[ 1753.568610] NvHost basic channel registers:
[ 1753.568613] CMDFIFO_STAT_0: 00002040
[ 1753.568616] CMDFIFO_RDATA_0: 28200e11
[ 1753.568618] CMDP_OFFSET_0: 00000000
[ 1753.568621] CMDP_CLASS_0: 00000000
[ 1753.568623] CHANNELSTAT_0: 00000000
[ 1753.568625] The CDMA sync queue is empty.
[ 1753.568631]
channel 2 - 15830000.se
[ 1753.568634] NvHost basic channel registers:
[ 1753.568636] CMDFIFO_STAT_0: 00002040
[ 1753.568639] CMDFIFO_RDATA_0: 11a20210
[ 1753.568641] CMDP_OFFSET_0: 00000000
[ 1753.568644] CMDP_CLASS_0: 00000000
[ 1753.568646] CHANNELSTAT_0: 00000000
[ 1753.568648] The CDMA sync queue is empty.
[ 1753.568665]
channel 3 - 15840000.se
[ 1753.568674] NvHost basic channel registers:
[ 1753.568682] CMDFIFO_STAT_0: 00002040
[ 1753.568687] CMDFIFO_RDATA_0: 20470cc4
[ 1753.568689] CMDP_OFFSET_0: 00000000
[ 1753.568692] CMDP_CLASS_0: 00000000
[ 1753.568695] CHANNELSTAT_0: 00000000
[ 1753.568697] The CDMA sync queue is empty.
[ 1753.568703]
---- host general irq ----
[ 1753.568707] sync_intc0mask = 0x00000001
[ 1753.568709] sync_intmask = 0x50000003
[ 1753.568712]
---- host syncpt irq mask ----
[ 1753.568715]
---- host syncpt irq status ----
[ 1753.568720] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1753.568722] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1753.568725] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1753.568728] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1753.568732] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1753.568734] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1753.568737] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 1753.568740] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 1753.568742] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 1753.568745] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 1753.568748] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 1753.568752] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 1753.568760] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 1753.568769] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 1753.568777] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 1753.568784] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 1753.568788] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 1753.568792] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 1753.568797] 17000000.gp10b pbdma 0:
[ 1753.572191] id: 4 (tsg), next_id: 4 (tsg) status: valid
[ 1753.577619] PUT: 000000010180e1f4 GET: 000000010180e190 FETCH: 000001d3 HEADER: 20040718
[ 1753.587199] 17000000.gp10b eng 0:
[ 1753.590439] id: 2 (tsg), next_id: 4 (tsg), ctx: invalid
[ 1753.597257] 17000000.gp10b eng 1:
[ 1753.600477] id: 4 (tsg), next_id: 4 (tsg), ctx: invalid
[ 1753.608950] 500-17000000.gp10b, pid 1647, refs: 2:
[ 1753.613660] in use idle not busy
[ 1753.617156] TOP: 800000010047f4fc PUT: 000000010047f4fc GET: 000000010047f4fc FETCH: 000000010047f4fc
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000001 0000fb90 00000004 00001004
[ 1753.630365] arm-smmu 12000000.iommu: Unhandled context fault: iova=0x249c6000, fsynr=0x20013, cb=19, sid=16(0x10 - GPU), pgd=267e63003, pud=267e63003, pmd=21ad53003, pte=0
[ 1753.655209] 501-17000000.gp10b, pid 1647, refs: 2:
[ 1753.659924] in use idle not busy
[ 1753.663425] TOP: 8000000100800294 PUT: 0000000100800294 GET: 0000000100800294 FETCH: 0000000100800294
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.686176] 502-17000000.gp10b, pid 1647, refs: 2:
[ 1753.690879] not in use idle not busy
[ 1753.694632] TOP: 8000000100c00294 PUT: 0000000100c00294 GET: 0000000100c00294 FETCH: 0000000100c00294
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.717395] 503-17000000.gp10b, pid 1647, refs: 2:
[ 1753.722094] not in use idle not busy
[ 1753.725847] TOP: 8000000101000294 PUT: 0000000101000294 GET: 0000000101000294 FETCH: 0000000101000294
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.748603] 504-17000000.gp10b, pid 1647, refs: 2:
[ 1753.753302] in use idle not busy
[ 1753.756795] TOP: 8000002000600018 PUT: 0000002000600018 GET: 0000002000600018 FETCH: 0000002000600018
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001801 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.779542] 505-17000000.gp10b, pid 1647, refs: 2:
[ 1753.784241] in use on_pbdma busy
[ 1753.787737] TOP: 800000010180e190 PUT: 000000010180e1f4 GET: 000000010180e190 FETCH: 0000fc010180e1f4
HEADER: 20040718 COUNT: 00110002
SYNCPOINT 00000000 00000000 SEMAPHORE 00000001 02207ffc 00000002 00081004
[ 1753.810481] 506-17000000.gp10b, pid 903, refs: 2:
[ 1753.815097] in use idle not busy
[ 1753.818590] TOP: 8000002000140030 PUT: 0000002000140030 GET: 0000002000140030 FETCH: 0000002000140030
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001301 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.841343] 507-17000000.gp10b, pid 903, refs: 2:
[ 1753.845958] in use idle not busy
[ 1753.849450] TOP: 800000200005bde0 PUT: 000000200005bde0 GET: 000000200005bde0 FETCH: 000000200005bde0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001201 SEMAPHORE 00000003 fc000000 3dc0b704 01100002
[ 1753.872195] 508-17000000.gp10b, pid 764, refs: 2:
[ 1753.876806] in use idle not busy
[ 1753.880295] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.903042] 509-17000000.gp10b, pid 764, refs: 2:
[ 1753.907655] in use idle not busy
[ 1753.911150] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.933909] 510-17000000.gp10b, pid 764, refs: 2:
[ 1753.938521] in use idle not busy
[ 1753.942017] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.964775] 511-17000000.gp10b, pid 764, refs: 2:
[ 1753.969387] in use idle not busy
[ 1753.972883] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1753.996327] gk20a 17000000.gp10b: gk20a_fifo_handle_pbdma_intr: pbdma_intr_0(0):0x00004000 PBH: 20040718 SHADOW: 200381c5 M0: 8000001d 8004041d 80040715 00000000
[ 1754.010747] gk20a 17000000.gp10b: gk20a_set_error_notifier: error notifier set to 32 for ch 505
[ 1754.020123] gk20a 17000000.gp10b: gk20a_fifo_set_ctx_mmu_error_tsg: TSG 4 generated a mmu fault
[ 1754.028822] ---- mlocks ----
[ 1754.028844] ---- syncpts ----
[ 1754.028853] id 4 (disp_d) min 8 max 8 refs 1 (previous client : )
[ 1754.028856] id 5 (disp_e) min 1 max 1 refs 1 (previous client : )
[ 1754.028860] id 6 (disp_f) min 1 max 1 refs 1 (previous client : )
[ 1754.028863] id 7 (vblank1) min 19065 max 0 refs 1 (previous client : )
[ 1754.028874] id 18 (17000000.gp10b_507) min 7134 max 7134 refs 1 (previous client : )
[ 1754.028878] id 19 (17000000.gp10b_506) min 4 max 4 refs 1 (previous client : )
[ 1754.028885] id 24 (17000000.gp10b_504) min 12 max 12 refs 1 (previous client : 17000000.gp10b_501)
[ 1754.029288] ---- channels ----
[ 1754.029298]
channel 0 - 15810000.se
[ 1754.029301] NvHost basic channel registers:
[ 1754.029304] CMDFIFO_STAT_0: 00002040
[ 1754.029307] CMDFIFO_RDATA_0: 02c08a00
[ 1754.029310] CMDP_OFFSET_0: 00000000
[ 1754.029312] CMDP_CLASS_0: 00000000
[ 1754.029315] CHANNELSTAT_0: 00000000
[ 1754.029317] The CDMA sync queue is empty.
[ 1754.029322]
channel 1 - 15820000.se
[ 1754.029325] NvHost basic channel registers:
[ 1754.029327] CMDFIFO_STAT_0: 00002040
[ 1754.029330] CMDFIFO_RDATA_0: 28200e11
[ 1754.029332] CMDP_OFFSET_0: 00000000
[ 1754.029335] CMDP_CLASS_0: 00000000
[ 1754.029337] CHANNELSTAT_0: 00000000
[ 1754.029339] The CDMA sync queue is empty.
[ 1754.029344]
channel 2 - 15830000.se
[ 1754.029347] NvHost basic channel registers:
[ 1754.029350] CMDFIFO_STAT_0: 00002040
[ 1754.029352] CMDFIFO_RDATA_0: 11a20210
[ 1754.029355] CMDP_OFFSET_0: 00000000
[ 1754.029358] CMDP_CLASS_0: 00000000
[ 1754.029360] CHANNELSTAT_0: 00000000
[ 1754.029362] The CDMA sync queue is empty.
[ 1754.029367]
channel 3 - 15840000.se
[ 1754.029370] NvHost basic channel registers:
[ 1754.029372] CMDFIFO_STAT_0: 00002040
[ 1754.029375] CMDFIFO_RDATA_0: 20470cc4
[ 1754.029377] CMDP_OFFSET_0: 00000000
[ 1754.029380] CMDP_CLASS_0: 00000000
[ 1754.029382] CHANNELSTAT_0: 00000000
[ 1754.029384] The CDMA sync queue is empty.
[ 1754.029391]
---- host general irq ----
[ 1754.029395] sync_intc0mask = 0x00000001
[ 1754.029397] sync_intmask = 0x50000003
[ 1754.029399]
---- host syncpt irq mask ----
[ 1754.029402]
---- host syncpt irq status ----
[ 1754.029406] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1754.029409] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1754.029412] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1754.029414] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1754.029417] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1754.029420] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1754.029422] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 1754.029425] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 1754.029427] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 1754.029430] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 1754.029433] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 1754.029435] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 1754.029438] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 1754.029440] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 1754.029443] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 1754.029446] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 1754.029448] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 1754.029451] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 1754.029456] 17000000.gp10b pbdma 0:
[ 1754.032864] id: 4 (tsg), next_id: 4 (tsg) status: valid
[ 1754.038273] PUT: 000000010180e1f4 GET: 000000010180e190 FETCH: 000001d3 HEADER: 20040718
[ 1754.047842] 17000000.gp10b eng 0:
[ 1754.051070] id: 2 (tsg), next_id: 2 (tsg), ctx: invalid
[ 1754.057881] 17000000.gp10b eng 1:
[ 1754.061110] id: 4 (tsg), next_id: 4 (tsg), ctx: invalid
[ 1754.069540] 503-17000000.gp10b, pid 1647, refs: 2:
[ 1754.074243] not in use idle not busy
[ 1754.077996] TOP: 8000000101000294 PUT: 0000000101000294 GET: 0000000101000294 FETCH: 0000000101000294
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.100753] 504-17000000.gp10b, pid 1647, refs: 2:
[ 1754.105453] in use idle not busy
[ 1754.108947] TOP: 8000002000600018 PUT: 0000002000600018 GET: 0000002000600018 FETCH: 0000002000600018
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001801 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.131692] 505-17000000.gp10b, pid 1647, refs: 2:
[ 1754.136395] not in use on_pbdma busy
[ 1754.140148] TOP: 800000010180e190 PUT: 000000010180e1f4 GET: 000000010180e190 FETCH: 0000fc010180e1f4
HEADER: 20040718 COUNT: 00110002
SYNCPOINT 00000000 00000000 SEMAPHORE 00000001 02207ffc 00000002 00081004
[ 1754.162895] 506-17000000.gp10b, pid 903, refs: 2:
[ 1754.167509] in use idle not busy
[ 1754.171005] TOP: 8000002000140030 PUT: 0000002000140030 GET: 0000002000140030 FETCH: 0000002000140030
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001301 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.193758] 507-17000000.gp10b, pid 903, refs: 2:
[ 1754.198373] in use idle not busy
[ 1754.201866] TOP: 800000200005bde0 PUT: 000000200005bde0 GET: 000000200005bde0 FETCH: 000000200005bde0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001201 SEMAPHORE 00000003 fc000000 3dc0b704 01100002
[ 1754.224623] 508-17000000.gp10b, pid 764, refs: 2:
[ 1754.229234] in use idle not busy
[ 1754.232728] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.255477] 509-17000000.gp10b, pid 764, refs: 2:
[ 1754.260092] in use idle not busy
[ 1754.263585] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.286330] 510-17000000.gp10b, pid 764, refs: 2:
[ 1754.290944] in use idle not busy
[ 1754.294434] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
[ 1754.317188] 511-17000000.gp10b, pid 764, refs: 2:
[ 1754.321803] in use idle not busy
[ 1754.325296] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
PS. I’ve run this code on TX1 very well…plz help. ToT