intrenal/cri: optimize stdio create procedure #11128
Conversation
|
Hi @mingfukuang. Thanks for your PR. I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
mingfukuang
force-pushed
the
master
branch
2 times, most recently
from
4abaec7 to
961ef83
Compare
When each container is created, the containerd process will create three new threads and open fifos.stdin/fifo.stdout/fifo.stderr. The three threads in the fifos pipeline file will be blocked, and these blocked threads will only return from the fifos pipeline file is opened after the corresponding shim process of the container is started. If the container remains in a state of creation without being started, the above three threads will be permanently blocked, and each created container will consume three threads. The default number of containerd threads is 10000. In extreme scenarios, creating only the container without starting it will reach the upper limit of containerd threads, ultimately causing containerd threads to exit. Move the operation of opening the fifos.stdin/fifos.stdout/fifos.stderr for each container to the StartContainer function, so that no new threads are created in CreateContainer function. And these new threads are not constantly blocked in opening fifos.stdin/fifos.stdout/fifos.stderr file process due to the immediate started of the shim process. Thus reducing the consumption of containerd threads in this way. Prevent containerd processes from exiting due to reaching the maximum number of threads. Signed-off-by: mingfukuang <kuang.mingfu@zte.com.cn>
|
cc |
There was a problem hiding this comment.
Hmm.. breaks the following contract:
https://github.com/containerd/containerd/blob/main/internal/cri/store/container/container.go#L42-L44
see state diagram https://github.com/containerd/containerd/blob/main/internal/cri/store/container/status.go#L39
Would need to carefully consider why that was an important must have in the state transition contract.. my memory is fuzzy on it..
Thank for your response. |
This issue was encountered during the stress testing process. |
Hi @mingfukuang in reality, kubelet always starts created container immediately. It's unlikely to have 5K created containers. Wei |
Continuously creating containers without starting them will result in the containerd process exiting.
Analyzing the exit stack of containerd and discovering:
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
...
syscall.Syscall6(0xc0148b8750?, 0x13?, 0x0?, 0x1000?, 0x0?, 0x674d2a23?, 0x14533f40?)
/usr/local/go/src/syscall/syscall_linux.go:91 +0x36 fp=0xc005481e08 sp=0xc005481d80 pc=0x559c1bfb1236
syscall.openat(0x559c1d7bb240?, {0xc0148b8750?, 0xc000000005?}, 0x559c1d23970a?, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:83 +0x94 fp=0xc005481e80 sp=0xc005481e08 pc=0x559c1bfad814
syscall.Open(...)
/usr/local/go/src/syscall/syscall_linux.go:272
os.openFileNolog({0xc0148b8750, 0x13}, 0x0, 0x0)
/usr/local/go/src/os/file_unix.go:245 +0x9b fp=0xc005481ec8 sp=0xc005481e80 pc=0x559c1bfde7fb
os.OpenFile({0xc0148b8750, 0x13}, 0x0, 0x12000000?)
/usr/local/go/src/os/file.go:326 +0x45 fp=0xc005481f00 sp=0xc005481ec8 pc=0x559c1bfdc165
...
And the above number of stacks is:
$ cat containerd.stack.log |grep openFileNolog |wc -l
9954
It could be seen that about 9954 threads are blocked on openFileNolog in the contaienrd process.
When each container is created, the containerd process will create three new threads and open fifos.stdin/fifo.stdout/fifo.stderr. The three threads in the fifos pipeline file will be blocked, and these blocked threads will only return from the Fifos pipeline file is opened after the corresponding shim process of the container is started. If the container remains in a state of creation without being started, the above three threads will be permanently blocked, and each created container will consume three threads. The default number of containerd threads is 10000. In extreme scenarios, creating only the container without starting it will reach the upper limit of containerd threads, ultimately causing containerd threads to exit.
Move the operation of opening the fifos.stdin/fifos.stdout/fifos.stderr for each container to the StartContainer function, so that no new threads are created in CreateContainer function. And these new threads are not constantly blocked in opening fifos.stdin/fifos.stdout/fifos.stderr file process due to the immediate start of the shim process. Thus reducing the consumption of containerd threads in this way. Prevent containerd processes from exiting due to reaching the maximum number of threads.