LLaMa3.1 required an upgrade to Ollama

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

The introduction of llama3.1 caused an issue to Ollama. The ollama code maintainer already addressed the problem.

When attempting to load a instruct version the error “done_getting_tensors: wrong number of tensors; expected 292, got 291” shows up in the log file (see I can't run llama3.1 · Issue #6048 · ollama/ollama · GitHub)

We need an update to dustynv/ollama container @dusty_nv

Thank you for all your help!!!

3

Hi @pontual, see the GitHub issue here for the updated container image:

Also if you have NVME for enough storage, you can rebuild the container as needed.

4

I tried to build ollama using: jetson-containers build --name ollamatest ollama but got some issues.

For example I got

Blockquote
Step 11/21 : RUN bash gen_linux.sh
—> Running in 37685aaea9d2

  • set -o pipefail
  • compress_pids=
    Starting linux generate script
  • echo ‘Starting linux generate script’
  • ‘[’ -z /usr/local/cuda/bin/nvcc ‘]’
  • COMMON_CMAKE_DEFS=‘-DCMAKE_SKIP_RPATH=on -DBUILD_SHARED_LIBS=on -DCMAKE_POSITION_INDEPENDENT_CODE=on -DGGML_NATIVE=off -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_OPENMP=off’
    ++ dirname gen_linux.sh
  • source ./gen_common.sh
  • init_vars
  • case “${GOARCH}” in
    GOARCH must be set
    this script is meant to be run from within go generate
  • echo ‘GOARCH must be set’
  • echo ‘this script is meant to be run from within go generate’
  • exit 1
    The command ‘/bin/sh -c bash gen_linux.sh’ returned a non-zero code: 1
    Traceback (most recent call last):
    File “/usr/lib/python3.10/runpy.py”, line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
    File “/usr/lib/python3.10/runpy.py”, line 86, in _run_code
    exec(code, run_globals)
    File “/home/rp/jetson-containers/jetson_containers/build.py”, line 103, in
    build_container(args.name, args.packages, args.base, args.build_flags, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
    File “/home/rp/jetson-containers/jetson_containers/container.py”, line 143, in build_container
    status = subprocess.run(cmd.replace(NEWLINE, ’ ‘), executable=’/bin/bash’, shell=True, check=True)
    File “/usr/lib/python3.10/subprocess.py”, line 526, in run
    raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command ‘DOCKER_BUILDKIT=0 docker build --network=host --tag ollamatest:r36.3.0-ollama --file /home/rp/jetson-containers/packages/llm/ollama/Dockerfile --build-arg BASE_IMAGE=ollamatest:r36.3.0-cuda --build-arg OLLAMA_REPO=“ollama/ollama” --build-arg OLLAMA_BRANCH=“main” --build-arg GOLANG_VERSION=“1.22.1” --build-arg CMAKE_VERSION=“3.22.1” --build-arg JETPACK_VERSION=“6.0” --build-arg CMAKE_CUDA_ARCHITECTURES=“87” /home/rp/jetson-containers/packages/llm/ollama 2>&1 | tee /home/rp/jetson-containers/logs/20240821_153103/build/ollamatest_r36.3.0-ollama.txt; exit ${PIPESTATUS[0]}’ returned non-zero exit status 1.
    rp@orinnx:~/jetson-containers$ docker images
    REPOSITORY TAG IMAGE ID CREATED SIZE
    92127f8f8bcb 17 minutes ago 7.23GB
    ollamatest r36.3.0-cuda 7bb06ca0c060 18 minutes ago 6.6GB
    ollamatest r36.3.0-build-essential c70190deb319 23 minutes ago 723MB
    ubuntu 22.04 7f965f3f8aae 8 days ago 69.2MB
5

Hi @pontual, it looks like $GOARCH started to need being set in the build environment from an ollama change a couple days ago - added that in jetson-containers commit 9fd7681 and rebuilt/pushed the latest version here: dustynv/ollama:r36.2.0

1 Like
6

@dusty_nv I can now build the image, thank you. But neither the image you produce nor the one I created is capable of pulling llama3.1 images.

root@orinnx:/# ollama pull llama3.1:8b-instruct-q4_K_S
pulling manifest
Error: pull model manifest: 412:

The model you are attempting to pull requires a newer version of Ollama.

Please download the latest version at:

	https://ollama.com/download

root@orinnx:/# ollama -v gives “ollama version is 6bd8a4b”.

I am able to pull images by using your older image: dustynv/ollama:r36.3.0

7

Hi @pontual, sorry for the delay - it looks like others were able to get it working by setting the ollama version to 3.7 -

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.