linux mem overcommit

2023-11-30 12:48
文章标签 linux mem overcommit

本文主要是介绍linux mem overcommit,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

 linux overcommit 机制会欺骗进程,承诺分配给他的内存空间,只有当这些page被真正touch到的时候才通过缺页机制拿到真正的物理内存。

以下摘自kernel doc

The Linux kernel supports the following overcommit handling modes0	-	Heuristic overcommit handling. Obvious overcommits ofaddress space are refused. Used for a typical system. Itensures a seriously wild allocation fails while allowingovercommit to reduce swap usage.  root is allowed to allocate slightly more memory in this mode. This is the default.1	-	Always overcommit. Appropriate for some scientificapplications. Classic example is code using sparse arraysand just relying on the virtual memory consisting almostentirely of zero pages.2	-	Don't overcommit. The total address space commitfor the system is not permitted to exceed swap + aconfigurable amount (default is 50%) of physical RAM.Depending on the amount you use, in most situationsthis means a process will not be killed while accessingpages but will receive errors on memory allocation asappropriate.Useful for applications that want to guarantee theirmemory allocations will be available in the futurewithout having to initialize every page.The overcommit policy is set via the sysctl `vm.overcommit_memory'.The overcommit amount can be set via `vm.overcommit_ratio' (percentage)
or `vm.overcommit_kbytes' (absolute value).The current overcommit limit and amount committed are viewable in
/proc/meminfo as CommitLimit and Committed_AS respectively.Gotchas
-------The C language stack growth does an implicit mremap. If you want absolute
guarantees and run close to the edge you MUST mmap your stack for the 
largest size you think you will need. For typical stack usage this does
not matter much but it's a corner case if you really really careIn mode 2 the MAP_NORESERVE flag is ignored. How It Works
------------The overcommit is based on the following rulesFor a file backed mapSHARED or READ-only	-	0 cost (the file is the map not swap)PRIVATE WRITABLE	-	size of mapping per instanceFor an anonymous or /dev/zero mapSHARED			-	size of mappingPRIVATE READ-only	-	0 cost (but of little use)PRIVATE WRITABLE	-	size of mapping per instanceAdditional accountingPages made writable copies by mmapshmfs memory drawn from the same poolStatus
------o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcementTo Do
-----
o	Account ptrace pages (this is hard)
CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'),this is the total amount of  memory currently available tobe allocated on the system. This limit is only adhered toif strict overcommit accounting is enabled (mode 2 in'vm.overcommit_memory').The CommitLimit is calculated with the following formula:CommitLimit = ([total RAM pages] - [total huge TLB pages]) *overcommit_ratio / 100 + [total swap pages]For example, on a system with 1G of physical RAM and 7Gof swap with a `vm.overcommit_ratio` of 30 it wouldyield a CommitLimit of 7.3G.For more details, see the memory overcommit documentationin vm/overcommit-accounting.
Committed_AS: The amount of memory presently allocated on the system.The committed memory is a sum of all of the memory whichhas been allocated by processes, even if it has not been"used" by them as of yet. A process which malloc()'s 1Gof memory, but only touches 300M of it will show up asusing 1G. This 1G is memory which has been "committed" toby the VM and can be used at any time by the allocatingapplication. With strict overcommit enabled on the system(mode 2 in 'vm.overcommit_memory'),allocations which wouldexceed the CommitLimit (detailed above) will not be permitted.This is useful if one needs to guarantee that processes willnot fail due to lack of memory once that memory has beensuccessfully allocated.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   for(i=0; i< 1024*1024*30; i++);{   temp_malloc[i] = i&0xff;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

 如上程序以为能分配到物理内存,可是实际上free命令结果看不到used memory增加

meminfo结果对比,申请的30M都被over committed 了,所以如上程序也没有真的touch到内存? 

 

/ # busybox pmap  2655
2655: {no such process} /mnt/mmc/sdcard/largemem
00010000       4K r-xp  /mnt/mmc/sdcard/largemem
00020000       4K r--p  /mnt/mmc/sdcard/largemem
00021000       4K rw-p  /mnt/mmc/sdcard/largemem
00022000     132K rw-p  [heap]
b50d0000   30724K rw-p    [ anon ]
b6ed1000     892K r-xp  /lib/libc-2.30.so
b6fb0000      60K ---p  /lib/libc-2.30.so
b6fbf000       8K r--p  /lib/libc-2.30.so
b6fc1000       4K rw-p  /lib/libc-2.30.so
b6fc2000      12K rw-p    [ anon ]
b6fc5000     100K r-xp  /lib/ld-2.30.so
b6feb000       8K rw-p    [ anon ]
b6fed000       4K r--p  /lib/ld-2.30.so
b6fee000       4K rw-p  /lib/ld-2.30.so
bed26000     132K rw-p  [stack]
bedd7000       4K r-xp  [sigpage]
bedd8000       4K r--p  [vvar]
bedd9000       4K r-xp  [vdso]
ffff0000       4K r-xp  [vectors]
mapped: 32108K

pmap的结果也显示出了30M的内存使用

google 查到有人用mlock 来实现防止被overcommit掉

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/mman.h>int main(void)
{char *temp_malloc = NULL;uint64_t i=0;printf("wait malloc\n");sleep(10);printf("start malloc\n");temp_malloc=malloc(1024*1024*30);if(temp_malloc ==NULL){   printf("malloc fail\n");return 0;}   if(mlock(temp_malloc, 1024*1024*30)){   printf("mlock fail\n");free(temp_malloc);return 0;}   printf("wr done\n");sleep(100);printf("free malloc\n");free(temp_malloc);return 0;
}

测试结果used 果然增加了30M

/ # freetotal         used         free       shared      buffers
Mem:        352328        42864       309464            0           40
-/+ buffers:              42824       309504
Swap:            0            0            0
/ # freestart malloc
wr donetotal         used         free       shared      buffers
Mem:        352328        73576       278752            0           40
-/+ buffers:              73536       278792
Swap:            0            0            0

至于为什么真正使用了malloc的memory也会被overcommit掉的原因,需要另开一篇认真调研下

这篇关于linux mem overcommit的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/437089

相关文章

Linux中压缩、网络传输与系统监控工具的使用完整指南

《Linux中压缩、网络传输与系统监控工具的使用完整指南》在Linux系统管理中,压缩与传输工具是数据备份和远程协作的桥梁,而系统监控工具则是保障服务器稳定运行的眼睛,下面小编就来和大家详细介绍一下它... 目录引言一、压缩与解压:数据存储与传输的优化核心1. zip/unzip:通用压缩格式的便捷操作2.

Linux中SSH服务配置的全面指南

《Linux中SSH服务配置的全面指南》作为网络安全工程师,SSH(SecureShell)服务的安全配置是我们日常工作中不可忽视的重要环节,本文将从基础配置到高级安全加固,全面解析SSH服务的各项参... 目录概述基础配置详解端口与监听设置主机密钥配置认证机制强化禁用密码认证禁止root直接登录实现双因素

在Linux终端中统计非二进制文件行数的实现方法

《在Linux终端中统计非二进制文件行数的实现方法》在Linux系统中,有时需要统计非二进制文件(如CSV、TXT文件)的行数,而不希望手动打开文件进行查看,例如,在处理大型日志文件、数据文件时,了解... 目录在linux终端中统计非二进制文件的行数技术背景实现步骤1. 使用wc命令2. 使用grep命令

Linux如何快速检查服务器的硬件配置和性能指标

《Linux如何快速检查服务器的硬件配置和性能指标》在运维和开发工作中,我们经常需要快速检查Linux服务器的硬件配置和性能指标,本文将以CentOS为例,介绍如何通过命令行快速获取这些关键信息,... 目录引言一、查询CPU核心数编程(几C?)1. 使用 nproc(最简单)2. 使用 lscpu(详细信

linux重启命令有哪些? 7个实用的Linux系统重启命令汇总

《linux重启命令有哪些?7个实用的Linux系统重启命令汇总》Linux系统提供了多种重启命令,常用的包括shutdown-r、reboot、init6等,不同命令适用于不同场景,本文将详细... 在管理和维护 linux 服务器时,完成系统更新、故障排查或日常维护后,重启系统往往是必不可少的步骤。本文

基于Linux的ffmpeg python的关键帧抽取

《基于Linux的ffmpegpython的关键帧抽取》本文主要介绍了基于Linux的ffmpegpython的关键帧抽取,实现以按帧或时间间隔抽取关键帧,文中通过示例代码介绍的非常详细,对大家的学... 目录1.FFmpeg的环境配置1) 创建一个虚拟环境envjavascript2) ffmpeg-py

Linux脚本(shell)的使用方式

《Linux脚本(shell)的使用方式》:本文主要介绍Linux脚本(shell)的使用方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录概述语法详解数学运算表达式Shell变量变量分类环境变量Shell内部变量自定义变量:定义、赋值自定义变量:引用、修改、删

Linux链表操作方式

《Linux链表操作方式》:本文主要介绍Linux链表操作方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录一、链表基础概念与内核链表优势二、内核链表结构与宏解析三、内核链表的优点四、用户态链表示例五、双向循环链表在内核中的实现优势六、典型应用场景七、调试技巧与

详解Linux中常见环境变量的特点与设置

《详解Linux中常见环境变量的特点与设置》环境变量是操作系统和用户设置的一些动态键值对,为运行的程序提供配置信息,理解环境变量对于系统管理、软件开发都很重要,下面小编就为大家详细介绍一下吧... 目录前言一、环境变量的概念二、常见的环境变量三、环境变量特点及其相关指令3.1 环境变量的全局性3.2、环境变

Linux系统中的firewall-offline-cmd详解(收藏版)

《Linux系统中的firewall-offline-cmd详解(收藏版)》firewall-offline-cmd是firewalld的一个命令行工具,专门设计用于在没有运行firewalld服务的... 目录主要用途基本语法选项1. 状态管理2. 区域管理3. 服务管理4. 端口管理5. ICMP 阻断