Still cant get Nano 2gb to process Epoch, python3 memory crash

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

This is all with the GUI shut down also.

[ 7648.154671] Out of memory: Kill process 14442 (python3) score 87 or sacrifice child
[ 7648.190034] Killed process 14442 (python3) total-vm:9207728kB, anon-rss:0kB, file-rss:364656kB, shmem-rss:0kB
[ 7649.231135] oom_reaper: reaped process 14442 (python3), now anon-rss:0kB, file-rss:365388kB, shmem-rss:0kB

Dec 16 19:30:23 nano kernel: [ 7648.050005] systemd-timesyn invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Dec 16 19:30:23 nano kernel: [ 7648.153807] systemd-timesyn cpuset=/ mems_allowed=0
Dec 16 19:30:23 nano kernel: [ 7648.153823] CPU: 2 PID: 3589 Comm: systemd-timesyn Tainted: G        W       4.9.140-tegra #1
Dec 16 19:30:23 nano kernel: [ 7648.153827] Hardware name: NVIDIA Jetson Nano 2GB Developer Kit (DT)
Dec 16 19:30:23 nano kernel: [ 7648.153830] Call trace:
Dec 16 19:30:23 nano kernel: [ 7648.153840] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198
Dec 16 19:30:23 nano kernel: [ 7648.153846] [<ffffff800808c37c>] show_stack+0x24/0x30
Dec 16 19:30:23 nano kernel: [ 7648.153852] [<ffffff800845c7a0>] dump_stack+0x98/0xc0
Dec 16 19:30:23 nano kernel: [ 7648.153858] [<ffffff8008258178>] dump_header+0x6c/0x1b8
Dec 16 19:30:23 nano kernel: [ 7648.153865] [<ffffff80081c8e28>] oom_kill_process+0x268/0x498
Dec 16 19:30:23 nano kernel: [ 7648.153869] [<ffffff80081c9504>] out_of_memory+0x1e4/0x308
Dec 16 19:30:23 nano kernel: [ 7648.153874] [<ffffff80081cf02c>] __alloc_pages_nodemask+0x844/0xd38
Dec 16 19:30:23 nano kernel: [ 7648.153879] [<ffffff800821a798>] __read_swap_cache_async+0x168/0x2a0
Dec 16 19:30:23 nano kernel: [ 7648.153883] [<ffffff800821a910>] read_swap_cache_async+0x40/0x68
Dec 16 19:30:23 nano kernel: [ 7648.153887] [<ffffff800821aaf4>] swapin_readahead+0x1bc/0x248
Dec 16 19:30:23 nano kernel: [ 7648.153893] [<ffffff800820191c>] do_swap_page+0x2c4/0x838
Dec 16 19:30:23 nano kernel: [ 7648.153897] [<ffffff8008203e0c>] handle_mm_fault+0x60c/0xa68
Dec 16 19:30:23 nano kernel: [ 7648.153902] [<ffffff80080a36b0>] do_page_fault+0x308/0x518
Dec 16 19:30:23 nano kernel: [ 7648.153906] [<ffffff80080a392c>] do_translation_fault+0x6c/0x80
Dec 16 19:30:23 nano kernel: [ 7648.153910] [<ffffff8008080954>] do_mem_abort+0x54/0xb0
Dec 16 19:30:23 nano kernel: [ 7648.153914] [<ffffff8008083408>] el0_da+0x20/0x24
Dec 16 19:30:23 nano kernel: [ 7648.153916] Mem-Info:
Dec 16 19:30:23 nano kernel: [ 7648.153928] active_anon:15 inactive_anon:31 isolated_anon:0
Dec 16 19:30:23 nano kernel: [ 7648.153928]  active_file:55 inactive_file:661 isolated_file:0
Dec 16 19:30:23 nano kernel: [ 7648.153928]  unevictable:18016 dirty:0 writeback:0 unstable:0
Dec 16 19:30:23 nano kernel: [ 7648.153928]  slab_reclaimable:12743 slab_unreclaimable:20299
Dec 16 19:30:23 nano kernel: [ 7648.153928]  mapped:14470 shmem:0 pagetables:3811 bounce:0
Dec 16 19:30:23 nano kernel: [ 7648.153928]  free:5625 free_pcp:31 free_cma:204
Dec 16 19:30:23 nano kernel: [ 7648.153937] Node 0 active_anon:60kB inactive_anon:124kB active_file:220kB inactive_file:2644kB unevictable:72064kB isolated(anon):0kB isolated(file):0kB mapped:57880kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:13 all_unreclaimable? no
Dec 16 19:30:23 nano kernel: [ 7648.153947] DMA free:22500kB min:22528kB low:28160kB high:33792kB active_anon:60kB inactive_anon:124kB active_file:220kB inactive_file:2644kB unevictable:72064kB writepending:0kB present:2091008kB managed:2027368kB mlocked:60036kB slab_reclaimable:50972kB slab_unreclaimable:81196kB kernel_stack:7696kB pagetables:15244kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:816kB
Dec 16 19:30:23 nano kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 19:30:23 nano kernel: [ 7648.153961] DMA: 437*4kB (UMEC) 130*8kB (UMEC) 310*16kB (UMEH) 418*32kB (UME) 9*64kB (UM) 2*128kB (H) 1*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 22724kB
Dec 16 19:30:23 nano kernel: [ 7648.154005] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 16 19:30:23 nano kernel: [ 7648.154007] 18139 total pagecache pages
Dec 16 19:30:23 nano kernel: [ 7648.154011] 20 pages in swap cache
Dec 16 19:30:23 nano kernel: [ 7648.154014] Swap cache stats: add 3168123, delete 3168369, find 40650/1156404
Dec 16 19:30:23 nano kernel: [ 7648.154017] Free swap  = 18415600kB
Dec 16 19:30:23 nano kernel: [ 7648.154019] Total swap = 20578284kB
Dec 16 19:30:23 nano kernel: [ 7648.154021] 522752 pages RAM
Dec 16 19:30:23 nano kernel: [ 7648.154023] 0 pages HighMem/MovableOnly
Dec 16 19:30:23 nano kernel: [ 7648.154025] 15910 pages reserved
Dec 16 19:30:23 nano kernel: [ 7648.154028] 16384 pages cma reserved
Dec 16 19:30:23 nano kernel: [ 7648.154030] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 16 19:30:23 nano kernel: [ 7648.154056] [ 2019]     0  2019     8646      245      13       4      239             0 systemd-journal
Dec 16 19:30:23 nano kernel: [ 7648.154063] [ 2524]     0  2524     3955      215       7       3      437         -1000 systemd-udevd
Dec 16 19:30:23 nano kernel: [ 7648.154069] [ 3562]   102  3562     2709      147       9       3      245             0 systemd-resolve
Dec 16 19:30:23 nano kernel: [ 7648.154075] [ 3571]     0  3571     1602        0       7       4      137             0 rpcbind
Dec 16 19:30:23 nano kernel: [ 7648.154080] [ 3587]     0  3587     1920        2       7       3      783             0 haveged
Dec 16 19:30:23 nano kernel: [ 7648.154085] [ 3589] 62583  3589    20992      162      11       3      195             0 systemd-timesyn
Dec 16 19:30:23 nano kernel: [ 7648.154090] [ 3780]     0  3780    25040        2      18       3     1985             0 networkd-dispat
Dec 16 19:30:23 nano kernel: [ 7648.154095] [ 3783]   109  3783    54993        0      14       4      391             0 rsyslogd
Dec 16 19:30:23 nano kernel: [ 7648.154101] [ 3838]   103  3838     1906      433       8       4      364          -900 dbus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154106] [ 3889]     0  3889     2418      178       8       3      252             0 wpa_supplicant
Dec 16 19:30:23 nano kernel: [ 7648.154111] [ 3894]   114  3894     1538      263       7       3      129             0 avahi-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154116] [ 3902]     0  3902    59312      817      18       4      390             0 accounts-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154121] [ 3923]     0  3923    78223      676      20       4      432             0 ModemManager
Dec 16 19:30:23 nano kernel: [ 7648.154126] [ 3948]     0  3948    97557      786      24       3      888             0 udisksd
Dec 16 19:30:23 nano kernel: [ 7648.154131] [ 3953]     0  3953     2633      302       8       3      204             0 systemd-logind
Dec 16 19:30:23 nano kernel: [ 7648.154136] [ 3979]     0  3979     1660      178       8       3       75             0 cron
Dec 16 19:30:23 nano kernel: [ 7648.154141] [ 4011]     0  4011    82736      917      29       4      996             0 NetworkManager
Dec 16 19:30:23 nano kernel: [ 7648.154146] [ 4127]   114  4127     1491        0       7       3       85             0 avahi-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154151] [ 4427]     0  4427    60056      935      19       4      864             0 polkitd
Dec 16 19:30:23 nano kernel: [ 7648.154158] [ 4718]   111  4718    80509      609      26       5      467             0 whoopsie
Dec 16 19:30:23 nano kernel: [ 7648.154163] [ 4832]     0  4832     1318        0       7       3       35             0 agetty
Dec 16 19:30:23 nano kernel: [ 7648.154168] [ 4849]   105  4849     2372      347       8       4      136             0 kerneloops
Dec 16 19:30:23 nano kernel: [ 7648.154173] [ 4867]     0  4867     2398        0       8       4      130             0 xrdp-sesman
Dec 16 19:30:23 nano kernel: [ 7648.154178] [ 4882]     0  4882   307654        0      63       7     5965             0 containerd
Dec 16 19:30:23 nano kernel: [ 7648.154183] [ 4888]     0  4888     1374        0       6       3       37             0 agetty
Dec 16 19:30:23 nano kernel: [ 7648.154188] [ 4918]   105  4918     2372      332       8       4      105             0 kerneloops
Dec 16 19:30:23 nano kernel: [ 7648.154192] [ 4990]     0  4990     2605        0       8       4      193         -1000 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154197] [ 5047]   125  5047     2253      226       8       3      151             0 xrdp
Dec 16 19:30:23 nano kernel: [ 7648.154202] [ 5154]     0  5154     5375        0       8       3       69             0 nvs-service
Dec 16 19:30:23 nano kernel: [ 7648.154207] [ 5161]     0  5161     1318        0       7       4       37             0 agetty
Dec 16 19:30:23 nano kernel: [ 7648.154213] [ 5174]     0  5174     1654      137       8       3       94             0 nvmemwarning.sh
Dec 16 19:30:23 nano kernel: [ 7648.154218] [ 5221]     0  5221     1654        0       7       4       77             0 nvgetty.sh
Dec 16 19:30:23 nano kernel: [ 7648.154223] [ 5251]     0  5251     1318        0       7       3       35             0 getty
Dec 16 19:30:23 nano kernel: [ 7648.154228] [ 5310]     0  5310    20905        0      30       4     1230             0 nvargus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154233] [ 5350]     0  5350    58607      935      17       4      888             0 lightdm
Dec 16 19:30:23 nano kernel: [ 7648.154239] [ 5504]     0  5504      786        0       4       3       37             0 nvphsd
Dec 16 19:30:23 nano kernel: [ 7648.154243] [ 5514]     0  5514     3065        1       6       3       92             0 nvphsd
Dec 16 19:30:23 nano kernel: [ 7648.154248] [ 5603]     0  5603     4373      168      13       5     2440             0 dhcpd
Dec 16 19:30:23 nano kernel: [ 7648.154253] [ 5824]  1000  5824     3216      340      10       3      407             0 systemd
Dec 16 19:30:23 nano kernel: [ 7648.154258] [ 5845]  1000  5845     3994        0      12       3      668             0 (sd-pam)
Dec 16 19:30:23 nano kernel: [ 7648.154262] [ 5903]  1000  5903     1780      398       8       4      220             0 dbus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154267] [ 5934]  1000  5934    58783      513      15       4      271             0 gvfsd
Dec 16 19:30:23 nano kernel: [ 7648.154272] [ 5942]  1000  5942    94998        0      19       5      287             0 gvfsd-fuse
Dec 16 19:30:23 nano kernel: [ 7648.154277] [ 5945]  1000  5945    39547        0      12       5      144             0 gvfsd-metadata
Dec 16 19:30:23 nano kernel: [ 7648.154282] [ 6052]  1000  6052    76971        0      17       4      241             0 at-spi-bus-laun
Dec 16 19:30:23 nano kernel: [ 7648.154287] [ 6059]  1000  6059     1709      338       7       3      133             0 dbus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154291] [ 6100]  1000  6100     1046        0       6       3       83             0 ssh-agent
Dec 16 19:30:23 nano kernel: [ 7648.154296] [ 6116]  1000  6116   129575      724      51       3     1576             0 deja-dup-monito
Dec 16 19:30:23 nano kernel: [ 7648.154301] [ 6185]  1000  6185   137695      560      37       5     1089             0 zeitgeist-datah
Dec 16 19:30:23 nano kernel: [ 7648.154306] [ 6201]  1000  6201    59684      925      18       3      610             0 gvfs-udisks2-vo
Dec 16 19:30:23 nano kernel: [ 7648.154311] [ 6223]  1000  6223    95380      570      19       3      831             0 zeitgeist-daemo
Dec 16 19:30:23 nano kernel: [ 7648.154315] [ 6229]   106  6229    38092        0       9       4      113             0 rtkit-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154320] [ 6252]  1000  6252    57984      503      14       4      216             0 gvfs-goa-volume
Dec 16 19:30:23 nano kernel: [ 7648.154325] [ 6267]  1000  6267   129172      610      51       4     2045             0 goa-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154330] [ 6278]  1000  6278    66618      527      20       4      324             0 zeitgeist-fts
Dec 16 19:30:23 nano kernel: [ 7648.154334] [ 6310]  1000  6310    59954      155      20       3      345             0 goa-identity-se
Dec 16 19:30:23 nano kernel: [ 7648.154339] [ 6318]  1000  6318    78010      507      19       3      849             0 gvfs-afc-volume
Dec 16 19:30:23 nano kernel: [ 7648.154344] [ 6328]  1000  6328    58451      547      15       4      261             0 gvfs-gphoto2-vo
Dec 16 19:30:23 nano kernel: [ 7648.154348] [ 6344]  1000  6344    57953      550      14       4      202             0 gvfs-mtp-volume
Dec 16 19:30:23 nano kernel: [ 7648.154353] [ 6358]  1000  6358   124685        0      39       3     3773             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154358] [ 6359]  1000  6359   124685        0      39       3     3773             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154362] [ 6360]  1000  6360   124688        0      39       3     3778             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154367] [ 6361]  1000  6361   124688        0      39       3     3778             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154372] [ 6362]  1000  6362   124688       97      40       3     3832             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154376] [ 6363]  1000  6363   124688        0      39       3     3779             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154381] [ 6365]  1000  6365   124688        0      39       3     3779             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154386] [ 6368]  1000  6368    78058      906      19       4      360             0 gvfsd-trash
Dec 16 19:30:23 nano kernel: [ 7648.154391] [ 7738]     0  7738     2309        2       9       4      154             0 sudo
Dec 16 19:30:23 nano kernel: [ 7648.154396] [ 7741]     0  7741   226261        0      61       6     7771             0 docker
Dec 16 19:30:23 nano kernel: [ 7648.154401] [ 7750]     0  7750   304029        0      81       6    10908          -500 dockerd
Dec 16 19:30:23 nano kernel: [ 7648.154406] [ 8234]     0  8234     2246        0       8       5      296             1 containerd-shim
Dec 16 19:30:23 nano kernel: [ 7648.154411] [ 8273]     0  8273      479        0       5       4       30             0 sh
Dec 16 19:30:23 nano kernel: [ 7648.154416] [ 8325]     0  8325      793        0       5       4       55             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154420] [ 8327]     0  8327   136279        0      68       4    23223             0 jupyter-lab
Dec 16 19:30:23 nano kernel: [ 7648.154425] [ 8332]     0  8332      947        0       5       4      106             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154430] [12147]     0 12147     3034        2       9       3      245             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154435] [12184]  1000 12184     3107        0      10       3      256             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154440] [12186]  1000 12186     1996        0       8       3      415             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154445] [12853]     0 12853     3034      528       9       3      245             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154450] [12916]  1000 12916     3107      428      10       3      249             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154454] [12917]  1000 12917     1968      414       8       3      398             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154459] [12957]     0 12957     3034      536      11       4      245             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154464] [12995]  1000 12995     3107      450      11       4      263             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154469] [12996]  1000 12996     1968      433       9       3      400             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154474] [13015]     0 13015     3034      528      11       4      245             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154479] [13053]  1000 13053     3107      355      11       4      294             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154484] [13054]  1000 13054     1968      402       8       3      400             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154488] [13157]     0 13157     2309      493       8       3      154             0 sudo
Dec 16 19:30:23 nano kernel: [ 7648.154493] [13159]     0 13159      643      317       5       3       58             0 tegrastats
Dec 16 19:30:23 nano kernel: [ 7648.154499] [13385]  1000 13385    58894     1116      18       4      263             0 gnome-keyring-d
Dec 16 19:30:23 nano kernel: [ 7648.154504] [13425]     0 13425    61864     1168      19       5      346             0 upowerd
Dec 16 19:30:23 nano kernel: [ 7648.154509] [13503]   121 13503     3226      578      10       4      385             0 systemd
Dec 16 19:30:23 nano kernel: [ 7648.154514] [13504]   121 13504    40860      231      16       3      680             0 (sd-pam)
Dec 16 19:30:23 nano kernel: [ 7648.154518] [13525]   121 13525     1682      663       7       4      148             0 dbus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154523] [13527]   121 13527    76946     1574      17       4      262             0 at-spi-bus-laun
Dec 16 19:30:23 nano kernel: [ 7648.154528] [13528]   121 13528    58783     1204      17       4      230             0 gvfsd
Dec 16 19:30:23 nano kernel: [ 7648.154533] [13535]   121 13535    78614     1426      19       5      777             0 gvfsd-fuse
Dec 16 19:30:23 nano kernel: [ 7648.154538] [13542]   121 13542     1651      636       7       3      110             0 dbus-daemon
Dec 16 19:30:23 nano kernel: [ 7648.154542] [13550]   121 13550    39775     1280      13       3      199             0 at-spi2-registr
Dec 16 19:30:23 nano kernel: [ 7648.154547] [13605]     0 13605     2309      500       9       4      154             0 sudo
Dec 16 19:30:23 nano kernel: [ 7648.154552] [13606]     0 13606   226261        0      60       6     7762             0 docker
Dec 16 19:30:23 nano kernel: [ 7648.154556] [13617]     0 13617     2598        0       9       5      335             1 containerd-shim
Dec 16 19:30:23 nano kernel: [ 7648.154561] [13646]     0 13646      479        0       4       3       22             0 sh
Dec 16 19:30:23 nano kernel: [ 7648.154566] [13698]     0 13698      793        0       6       3       55             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154571] [13704]     0 13704    16394        2      36       3    11962             0 jupyter-lab
Dec 16 19:30:23 nano kernel: [ 7648.154576] [13705]     0 13705      946        2       6       3      108             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154580] [13767]     0 13767     2248      526       8       3      125             0 sudo
Dec 16 19:30:23 nano kernel: [ 7648.154585] [13768]     0 13768     1263      382       6       3       63             0 dmesg
Dec 16 19:30:23 nano kernel: [ 7648.154590] [14230]     0 14230  1608566     1155      35       5     2365             0 Xorg
Dec 16 19:30:23 nano kernel: [ 7648.154595] [14237]     0 14237    39781     1108      15       5      275             0 lightdm
Dec 16 19:30:23 nano kernel: [ 7648.154600] [14245]   121 14245      478       83       5       4       22             0 lightdm-greeter
Dec 16 19:30:23 nano kernel: [ 7648.154604] [14246]   121 14246    16306    15001      36       4        0             0 unity-greeter
Dec 16 19:30:23 nano kernel: [ 7648.154609] [14263]     0 14263     2248      473       8       3      140             0 sudo
Dec 16 19:30:23 nano kernel: [ 7648.154614] [14264]     0 14264   244630      110      63       6     8370             0 docker
Dec 16 19:30:23 nano kernel: [ 7648.154619] [14277]     0 14277     2246        0       8       5      364             1 containerd-shim
Dec 16 19:30:23 nano kernel: [ 7648.154625] [14311]     0 14311      479        0       5       3       23             0 sh
Dec 16 19:30:23 nano kernel: [ 7648.154629] [14360]     0 14360      793        0       6       3       54             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154634] [14366]     0 14366    16410        2      35       3    11962             0 jupyter-lab
Dec 16 19:30:23 nano kernel: [ 7648.154639] [14367]     0 14367      945        0       7       4      114             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154644] [14442]     0 14442  2301932    91164    1702       9   402601             0 python3
Dec 16 19:30:23 nano kernel: [ 7648.154650] [14561]     0 14561     3034      498      11       3      245             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154655] [14633]  1000 14633     3141      492      11       3      261             0 sshd
Dec 16 19:30:23 nano kernel: [ 7648.154659] [14636]  1000 14636     1999      398       9       3      408             0 bash
Dec 16 19:30:23 nano kernel: [ 7648.154666] [14878]     0 14878     1190       80       7       3       20             0 sleep
Dec 16 19:30:23 nano kernel: [ 7648.154671] Out of memory: Kill process 14442 (python3) score 87 or sacrifice child
Dec 16 19:30:23 nano kernel: [ 7648.190034] Killed process 14442 (python3) total-vm:9207728kB, anon-rss:0kB, file-rss:364656kB, shmem-rss:0kB
Dec 16 19:30:23 nano kernel: [ 7649.231135] oom_reaper: reaped process 14442 (python3), now anon-rss:0kB, file-rss:365388kB, shmem-rss:0kB
Dec 16 19:30:44 nano kernel: [ 7669.910751] start_addr=(0x8000), end_addr=(0x10000), buffer_size=(0x8000), smp_number_max=(4096)
2

I killed the GUI to free some more memory
set batch_size to 4 and still crashing

3

Hi @ssgloganvann, from these messages, it is running out of memory. In our testing, we are able to run these JupyterLab notebooks without issue on Nano 2GB, so hopefully you can get it working as well.

If you haven’t already, can you try running these commands outside of Docker and reboot?

$ sudo systemctl disable nvzramconfig
$ sudo reboot

When you mention that you have the GUI disabled, I take it that you have disconnected the HDMI from your Nano and are browsing JupyterLab remotely from your PC?

How many images are in your dataset? I noticed from the other dataset that you collected 1 thumbs-up / 1 thumbs-down. I wonder if that is throwing it off somehow by running the epochs with not enough data. We test it with roughly 100 images thumbs-up and 100 thumbs-down.

The next thing I would recommend is if you can try to run this other training example (also tested on Nano 2GB):

You can start the container for that example through the bottom link. At first I would just run it over SSH with your display still disconnected from the Nano.

4

I killed the X gui completely to free that memory and a couple more non priority processes, so X is not running at all.

I’ve tried 1/1, 25/25, 30/30, 75/75 for images

Also added a new power mode 4 cores at 1100mhz
GPU at 850mhz and memory at 1400mhz
Mode 0 and mode 1 made no improvements

I have not disabled the nvzram yet, will try that next.

I have 1 epoch running with no crash right now, but it’s been going on for 2+ hours, 23/23 images
And set batch_size to 2 in the code block

5

Tty1 tegrastats 20 second intervals
Tty2 debug data
Tty3 docker for jupiter
Tty4 terminal for any changes I need to make

Also checking ps -ax often to ensure no processes decide to come online that I miss in between sitting back down at console.

I’m about 20 years out of practice on linux, but was running it as my primary back in some of the original slackerware builds, last kernel I was using was like 4.0.24 or something. It’s been a long time and it’s all coming back rather quickly. Thankfully

6

You might want to run one of these commands to disable the desktop service from starting after reboots:

If it is still freezing, to start I would give the Hello AI World cat/dog example a try (linked to above) and see if you can get that training to work.

7

I went through and did some massive memory management on the system. Only running what I absolutely need I was able to process 200/200 images at 100 epochs, cam running smooth during the whole processing of data. No lag, no warnings, no issues.

1 Like
8

Hi, If you don’t mind, would you share with me how do you do the massive memory management?