Model warmup fails after adding Triton indexing kernels #2838
Comments
|
I have the same problem. I assume that you and others who reported the issues below are using the Docker image, and we have reverted to using Triton indexing kernels. Since the process involves compiling C files with calls to Python, but the Python headers are not available for C, we encounter an error. In simple terms, I assume that Python.h is not available when compiling these shared object files. After reviewing the Dockerfile, it appears that Python3.11-dev is not included in the final image, which is why Python.h is missing. Just guessing, my "sure" value is about 0.6 😁🤷🏼♂️ It seems to be the same issue as the following issues: |
#2835 not related... it's about gpu split from 2 to 4 H100, no any python stacktrace. But thanks @KreshLaDoge |
|
Update: I was able to get it working by changing the base image to devel to match the builder image I have to rebuild the image which takes time and increases the size of the image but now it works!! |
|
I don't know how to reproduce the issue Phi3.5 works perfectly under 3.0.1. I everyone here using podman ? I don't see why it should make any difference though.. |
|
can also every confirm they are using |
I can confirm the issue for me with 3.0.1, 3.0, and 2.4.0 |
|
I also had issues with 3.0.1 I suspect that it’s about missing Python.h, which would also explain why it worked for @YaserJaradeh when he changed the base image of Ubuntu to devel variant. But it can be something else. Currently, I’m forced to assing GPUs to container manually and not through the Nvidia container toolkit, so it might be related if others, experiencing the same issue, are using vGPUs for example 🤷 |
Can you elaborate ? It might be a potential culprit. |
Here is example of my docker compose and our way, how we assign GPUs - don't judge me, it has it’s own reasons why I can't use container toolkit 🤷 Anyway, I doubt that anyone else experiencing this issue has similar configuration.
|
|
I also assign the GPUs manually to my container |
System Info
I was using v2.3.1 via docker and everything was working. When I updated to later versions including the latest my TGI doesn't start due to an error:
This is my
nvidia-smioutput:Information
Tasks
Reproduction
here is the TGI env:
And here is how I'm running container (running it via podman):
Which is generated on my system from running it via a docker compose file.
Expected behavior
The TGI server to start correctly and normally as it had before adding the Triton kernels!
The text was updated successfully, but these errors were encountered: