linux mem overcommit

2023-11-30 12:48
文章标签 linux mem overcommit

本文主要是介绍linux mem overcommit,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

 linux overcommit 机制会欺骗进程,承诺分配给他的内存空间,只有当这些page被真正touch到的时候才通过缺页机制拿到真正的物理内存。

以下摘自kernel doc

The Linux kernel supports the following overcommit handling modes0	-	Heuristic overcommit handling. Obvious overcommits ofaddress space are refused. Used for a typical system. Itensures a seriously wild allocation fails while allowingovercommit to reduce swap usage.  root is allowed to allocate slightly more memory in this mode. This is the default.1	-	Always overcommit. Appropriate for some scientificapplications. Classic example is code using sparse arraysand just relying on the virtual memory consisting almostentirely of zero pages.2	-	Don't overcommit. The total address space commitfor the system is not permitted to exceed swap + aconfigurable amount (default is 50%) of physical RAM.Depending on the amount you use, in most situationsthis means a process will not be killed while accessingpages but will receive errors on memory allocation asappropriate.Useful for applications that want to guarantee theirmemory allocations will be available in the futurewithout having to initialize every page.The overcommit policy is set via the sysctl `vm.overcommit_memory'.The overcommit amount can be set via `vm.overcommit_ratio' (percentage)
or `vm.overcommit_kbytes' (absolute value).The current overcommit limit and amount committed are viewable in
/proc/meminfo as CommitLimit and Committed_AS respectively.Gotchas
-------The C language stack growth does an implicit mremap. If you want absolute
guarantees and run close to the edge you MUST mmap your stack for the 
largest size you think you will need. For typical stack usage this does
not matter much but it's a corner case if you really really careIn mode 2 the MAP_NORESERVE flag is ignored. How It Works
------------The overcommit is based on the following rulesFor a file backed mapSHARED or READ-only	-	0 cost (the file is the map not swap)PRIVATE WRITABLE	-	size of mapping per instanceFor an anonymous or /dev/zero mapSHARED			-	size of mappingPRIVATE READ-only	-	0 cost (but of little use)PRIVATE WRITABLE	-	size of mapping per instanceAdditional accountingPages made writable copies by mmapshmfs memory drawn from the same poolStatus
------o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcementTo Do
-----
o	Account ptrace pages (this is hard)
CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'),this is the total amount of  memory currently available tobe allocated on the system. This limit is only adhered toif strict overcommit accounting is enabled (mode 2 in'vm.overcommit_memory').The CommitLimit is calculated with the following formula:CommitLimit = ([total RAM pages] - [total huge TLB pages]) *overcommit_ratio / 100 + [total swap pages]For example, on a system with 1G of physical RAM and 7Gof swap with a `vm.overcommit_ratio` of 30 it wouldyield a CommitLimit of 7.3G.For more details, see the memory overcommit documentationin vm/overcommit-accounting.
Committed_AS: The amount of memory presently allocated on the system.The committed memory is a sum of all of the memory whichhas been allocated by processes, even if it has not been"used" by them as of yet. A process which malloc()'s 1Gof memory, but only touches 300M of it will show up asusing 1G. This 1G is memory which has been "committed" toby the VM and can be used at any time by the allocatingapplication. With strict overcommit enabled on the system(mode 2 in 'vm.overcommit_memory'),allocations which wouldexceed the CommitLimit (detailed above) will not be permitted.This is useful if one needs to guarantee that processes willnot fail due to lack of memory once that memory has beensuccessfully allocated.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   for(i=0; i< 1024*1024*30; i++);{   temp_malloc[i] = i&0xff;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

 如上程序以为能分配到物理内存,可是实际上free命令结果看不到used memory增加

meminfo结果对比,申请的30M都被over committed 了,所以如上程序也没有真的touch到内存? 

 

/ # busybox pmap  2655
2655: {no such process} /mnt/mmc/sdcard/largemem
00010000       4K r-xp  /mnt/mmc/sdcard/largemem
00020000       4K r--p  /mnt/mmc/sdcard/largemem
00021000       4K rw-p  /mnt/mmc/sdcard/largemem
00022000     132K rw-p  [heap]
b50d0000   30724K rw-p    [ anon ]
b6ed1000     892K r-xp  /lib/libc-2.30.so
b6fb0000      60K ---p  /lib/libc-2.30.so
b6fbf000       8K r--p  /lib/libc-2.30.so
b6fc1000       4K rw-p  /lib/libc-2.30.so
b6fc2000      12K rw-p    [ anon ]
b6fc5000     100K r-xp  /lib/ld-2.30.so
b6feb000       8K rw-p    [ anon ]
b6fed000       4K r--p  /lib/ld-2.30.so
b6fee000       4K rw-p  /lib/ld-2.30.so
bed26000     132K rw-p  [stack]
bedd7000       4K r-xp  [sigpage]
bedd8000       4K r--p  [vvar]
bedd9000       4K r-xp  [vdso]
ffff0000       4K r-xp  [vectors]
mapped: 32108K

pmap的结果也显示出了30M的内存使用

google 查到有人用mlock 来实现防止被overcommit掉

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/mman.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   if(mlock(temp_malloc, 1024*1024*30)){   printf("mlock fail\n");free(temp_malloc);return 0;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

测试结果used 果然增加了30M

/ # freetotal         used         free       shared      buffers
Mem:        352328        42864       309464            0           40
-/+ buffers:              42824       309504
Swap:            0            0            0
/ # freestart malloc
wr donetotal         used         free       shared      buffers
Mem:        352328        73576       278752            0           40
-/+ buffers:              73536       278792
Swap:            0            0            0

至于为什么真正使用了malloc的memory也会被overcommit掉的原因,需要另开一篇认真调研下

这篇关于linux mem overcommit的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/437089

相关文章

防止Linux rm命令误操作的多场景防护方案与实践

《防止Linuxrm命令误操作的多场景防护方案与实践》在Linux系统中,rm命令是删除文件和目录的高效工具,但一旦误操作,如执行rm-rf/或rm-rf/*,极易导致系统数据灾难,本文针对不同场景... 目录引言理解 rm 命令及误操作风险rm 命令基础常见误操作案例防护方案使用 rm编程 别名及安全删除

Linux下MySQL数据库定时备份脚本与Crontab配置教学

《Linux下MySQL数据库定时备份脚本与Crontab配置教学》在生产环境中,数据库是核心资产之一,定期备份数据库可以有效防止意外数据丢失,本文将分享一份MySQL定时备份脚本,并讲解如何通过cr... 目录备份脚本详解脚本功能说明授权与可执行权限使用 Crontab 定时执行编辑 Crontab添加定

使用docker搭建嵌入式Linux开发环境

《使用docker搭建嵌入式Linux开发环境》本文主要介绍了使用docker搭建嵌入式Linux开发环境,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友们下面... 目录1、前言2、安装docker3、编写容器管理脚本4、创建容器1、前言在日常开发全志、rk等不同

linux系统上安装JDK8全过程

《linux系统上安装JDK8全过程》文章介绍安装JDK的必要性及Linux下JDK8的安装步骤,包括卸载旧版本、下载解压、配置环境变量等,强调开发需JDK,运行可选JRE,现JDK已集成JRE... 目录为什么要安装jdk?1.查看linux系统是否有自带的jdk:2.下载jdk压缩包2.解压3.配置环境

Linux搭建ftp服务器的步骤

《Linux搭建ftp服务器的步骤》本文给大家分享Linux搭建ftp服务器的步骤,本文通过图文并茂的形式给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友参考下吧... 目录ftp搭建1:下载vsftpd工具2:下载客户端工具3:进入配置文件目录vsftpd.conf配置文件4:

Linux实现查看某一端口是否开放

《Linux实现查看某一端口是否开放》文章介绍了三种检查端口6379是否开放的方法:通过lsof查看进程占用,用netstat区分TCP/UDP监听状态,以及用telnet测试远程连接可达性... 目录1、使用lsof 命令来查看端口是否开放2、使用netstat 命令来查看端口是否开放3、使用telnet

Linux系统管理与进程任务管理方式

《Linux系统管理与进程任务管理方式》本文系统讲解Linux管理核心技能,涵盖引导流程、服务控制(Systemd与GRUB2)、进程管理(前台/后台运行、工具使用)、计划任务(at/cron)及常用... 目录引言一、linux系统引导过程与服务控制1.1 系统引导的五个关键阶段1.2 GRUB2的进化优

Linux查询服务器 IP 地址的命令详解

《Linux查询服务器IP地址的命令详解》在服务器管理和网络运维中,快速准确地获取服务器的IP地址是一项基本但至关重要的技能,下面我们来看看Linux中查询服务器IP的相关命令使用吧... 目录一、hostname 命令:简单高效的 IP 查询工具命令详解实际应用技巧注意事项二、ip 命令:新一代网络配置全

linux安装、更新、卸载anaconda实践

《linux安装、更新、卸载anaconda实践》Anaconda是基于conda的科学计算环境,集成1400+包及依赖,安装需下载脚本、接受协议、设置路径、配置环境变量,更新与卸载通过conda命令... 目录随意找一个目录下载安装脚本检查许可证协议,ENTER就可以安装完毕之后激活anaconda安装更

Linux查询服务器系统版本号的多种方法

《Linux查询服务器系统版本号的多种方法》在Linux系统管理和维护工作中,了解当前操作系统的版本信息是最基础也是最重要的操作之一,系统版本不仅关系到软件兼容性、安全更新策略,还直接影响到故障排查和... 目录一、引言:系统版本查询的重要性二、基础命令解析:cat /etc/Centos-release详