本文主要是介绍k8s容器放开锁内存限制问题，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
《k8s容器放开锁内存限制问题》nccl-test容器运行mpirun时因NCCL_BUFFSIZE过大导致OOM,需通过修改docker服务配置文件,将LimitMEMLOCK设为infinity并...
参考：https://Access.RedHat.com/solutions/1257953
问题
nccl-test容器docker.io/library/nccl-tests:24.12中跑mpirun，buff设置为NCCL_BUFFSIZE=503316480
提示out of memory：
pod-1:78:91 [0] include/alloc.h:114 NCCL WARN Cuda failure 'out of memory'pod-1:78:91 [0] include/alloc.h:119 NCCL WARN Failed to CUDA host alloc -268435456 bytes
pod-1:78:91 [0] NCCL INFO transport/net.cc:517 -> 1
pod-1:78:91 [0] NCCL INFO transport/net.cc:719 -> 1
pod-1:78:93 [0] NCCL INFO transport.cc:193 -> 1
pod-1:78:93 [0] NCCL INFO group.cc:133 -> 1
pod-1:78:93 [0] NCCL INFO group.cc:75 -> 1 [Async thread]pod-1:78:91 [0] proxy.cc:162IYTjlYma0 NCCL WARN [Service thread] Error encountered progressing operation=Connect, res=3, closing connection
pod-1:78:78 [0] NCCL INFO group.cc:426 -> 1
pod-1:78:78 [0] NCCL INFO group.cc:566 -> 1
pod-1:78:78 [0] NCCL INFO group.cc:106 -> 1
pod-1: Test NCCL failure sendrecv.cu:57 'unhandled cuda error (run with NCCL_DEBUG=INFO for details) / '
.. pod-1 pid 78: Test failure common.cu:383
.. pod-1 pid 78: Test failure common.cu:592
.. pod-1 pid 78: Test failure sendrecv.cu:103
.. pod-1 pid 78: Test failure common.cu:625
.. pod-1 pid 78: Test failure common.cu:1123
.. pod-1 pid 78: Test failure common.javascriptcu:893
问题确认
容器内执行ulimit -a显示max locked memory只有64k
放开容器max locked memory限制
在 /etc/systemd/system/docker.service中js增加LimitMEMLOCK=infinity
然后重启docker：
systemctl daemon-rejavascriptload systemctl restarhttp://www.chinasem.cnt docker
总结
这篇关于k8s容器放开锁内存限制问题的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
