Skip to content

unexpected EOF when running ollama pull #1036

Open
@BruceMacD

Description

Occasionally while pulling a model the download may get stuck waiting for a part which experienced an error.

$ ollama run llama2:70b
pulling manifest
pulling 153664158022...  99% |█████████████████████████████████████████████████████████████████████████████████ | (38/39 GB, 619 kB/s) [24m57s:9m15s]

Server log:

2023/11/07 11:36:24 download.go:160: 153664158022 part 75 attempt 0 failed: unexpected EOF, retrying

Workaround is to stop and resume the download.

Activity

killthekitten

killthekitten commented on Nov 8, 2023

@killthekitten

I've accumulated a handful of different connection errors, and EOF sometimes appeared among them. I have two terminals running the pull in parellel, one on my local machine, and another one on GCP, and the failures seem correlated.

One moment both terminals go unresponsive or throw an error, while after some time and several retries the download speed reaches hundreds of mb/s. Could it be related to this issue somehow #850?

images.go:1172: couldn't start upload: Get "https://registry.ollama.ai/v2/library/llama2/manifests/13b-chat-q5_1": context canceled
Error: pull model manifest: Get "https://registry.ollama.ai/v2/library/llama2/manifests/13b-chat-q5_1": dial tcp 34.120.132.20:443: connect: connection timed out
llm-deployments-ollama-1    | 2023/11/08 20:54:17 download.go:122: downloading 6ae280299950 in 64 64 MB part(s)
llm-deployments-ollama-1    | 2023/11/08 20:55:42 download.go:160: 6ae280299950 part 58 attempt 0 failed: unexpected EOF, retrying
llm-deployments-ollama-1    | 2023/11/08 20:55:50 images.go:1172: couldn't start upload: Get "https://registry.ollama.ai/v2/library/mistral/blobs/sha256:6ae28029995007a3ee8d0b8556d50f3b59b831074cf19c84de87acf51fb54054": dial tcp: lookup registry.ollama.ai on 127.0.0.11:53: server misbehaving
llm-deployments-ollama-1    | 2023/11/08 20:55:50 download.go:160: 6ae280299950 part 58 attempt 1 failed: Get "https://registry.ollama.ai/v2/library/mistral/blobs/sha256:6ae28029995007a3ee8d0b8556d50f3b59b831074cf19c84de87acf51fb54054": dial tcp: lookup registry.ollama.ai on 127.0.0.11:53: server misbehaving, retrying
llm-deployments-ollama-1    | 2023/11/08 20:55:58 images.go:1172: couldn't start upload: Get "https://registry.ollama.ai/v2/library/mistral/blobs/sha256:6ae28029995007a3ee8d0b8556d50f3b59b831074cf19c84de87acf51fb54054": dial tcp: lookup registry.ollama.ai on 127.0.0.11:53: server misbehaving
llm-deployments-ollama-1    | 2023/11/08 20:55:58 download.go:160: 6ae280299950 part 58 attempt 2 failed: Get "https://registry.ollama.ai/v2/library/mistral/blobs/sha256:6ae28029995007a3ee8d0b8556d50f3b59b831074cf19c84de87acf51fb54054": dial tcp: lookup registry.ollama.ai on 127.0.0.11:53: server misbehaving, retrying
cwatt

cwatt commented on Dec 6, 2023

@cwatt

I also seem to be experiencing this or a related problem while tring to pull models. Unfortunately, I haven't been able to succesfully pull any models since installing Ollama (0.1.13). Here's an example:

gpajd@WUST056705 ~ % ollama pull codellama
pulling manifest 
pulling 3a43f93b78ec... 100% ▕████████████████▏ 3.8 GB                         
Error: max retries exceeded: unexpected EOF
jmorganca

jmorganca commented on Dec 24, 2023

@jmorganca
Member

@cwatt so sorry you hit this error – wondering if this is still something you're hitting on every pull? Thanks for sharing will make sure to take a look at this

changed the title Pull may get stuck due to an EOF `unexpected EOF` when running `ollama pull` on Dec 24, 2023
cwatt

cwatt commented on Jan 2, 2024

@cwatt

@jmorganca On subsequent pull attempts I actually haven't been hitting any more EOF errors, but rather digest mismatch errors like what is described in this issue.

ollama pull codellama
pulling manifest 
pulling 3a43f93b78ec... 100% ▕████████████████▏ 3.8 GB                         
pulling 8c17c2ebb0ea... 100% ▕████████████████▏ 7.0 KB                         
pulling 590d74a5569b... 100% ▕████████████████▏ 4.8 KB                         
pulling 2e0493f67d0c... 100% ▕████████████████▏   59 B                         
pulling 7f6a57943a88... 100% ▕████████████████▏  120 B                         
pulling 316526ac7323... 100% ▕████████████████▏  529 B                         
verifying sha256 digest 
Error: digest mismatch, file must be downloaded again: want sha256:3a43f93b78ec50f7c4e4dc8bd1cb3fff5a900e7d574c51a6f7495e48486e0dac, got sha256:95e0eb0f860fe71bb37e83832c1bc1300ae827244bca7b8a89651a2c87d49770

I'm not sure what caused the change in behavior. I hope this helps!

sammcj

sammcj commented on May 19, 2024

@sammcj
Contributor

FYI the model registry seems to have been broken for 3-4 days.

(#1736 (comment))

sammcj

sammcj commented on May 24, 2024

@sammcj
Contributor

This PR fixes the issue for me: #4619

rampageservices

rampageservices commented on Aug 2, 2024

@rampageservices

There is a follow-on PR to #4619 that is yet to be pulled that seems to resolve the remaining issues that people are facing in this issue thread.

#4625

6 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingnetworkingIssues relating to ollama pull and push

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      `unexpected EOF` when running `ollama pull` · Issue #1036 · ollama/ollama