server: reduce max connections used in download #6347
Conversation
The previous value of 64 was WAY too high and unnecessary. It reached diminishing returns and blew past it. This is a more reasonable number for _most_ normal cases. For users on cloud servers with excellent network quality, this will keep screaming for them, without hitting our CDN limits. For users with relatively poor network quality, this will keep them from saturating their network and causing other issues.
|
Preemptively: With regard to configuration via some new environment variable, it seems best to "wait and see." to avoid adding more support burden for more configuration parameters if only very few if anyone will need it. |
|
Happy to test this to gather some quantitative metrics if needed (different servers/locations and local machines). In that case, just ping me. |
Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com>
Please! We would love any feedback/reports here. |
|
Already merged, sorry for the late reply, but since I'm curious about it myself, here's a quick non-scientific benchmark on different systems regardless. "Before" = Ollama 0.3.6 Test results:Domestic internet uplink (250 Mbit/s)Goal: Testing for the underlying issue of remaining bandwidth for other applications. Before: ~3 MBit/s Server internet uplink (mixed speeds)Goal: Testing for max. download speed. Some fluctuations expected between runs. Cloud provider 1 (8 vCPUs, identical machine hardware)
* larger machine with 16 vCPUs Cloud provider 2 (9 vCPUs)
* larger machine with 36 vCPUs Overall, with few exceptions, only cases with speeds over 600 MB/s will have slower downloads with the reduced connection count of 16 now compared to the previous 64. This may heavily depend on used internet connection and hardware configuration at this point. |
|
@MaxJa4 This is very helpful, thank you!! We'll continue to keep an eye on how it affects others and make changes as needed. Again, this is super helpful. Thank you so much. |
The previous value of 64 was WAY too high and unnecessary. It reached diminishing returns and blew past it. This is a more reasonable number for _most_ normal cases. For users on cloud servers with excellent network quality, this will keep screaming for them, without hitting our CDN limits. For users with relatively poor network quality, this will keep them from saturating their network and causing other issues.
The previous value of 64 was WAY too high and unnecessary. It reached diminishing returns and blew past it. This is a more reasonable number for most normal cases. For users on cloud servers with excellent network quality, this will keep screaming for them, without hitting our CDN limits. For users with relatively poor network quality, this will keep them from saturating their network and causing other issues.