【异常】记一次,排查docker容器的微服务,CPU飙高问题

2024-04-01 04:58

本文主要是介绍【异常】记一次,排查docker容器的微服务,CPU飙高问题,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

项目时 以docker为容器的 微服务项目。 其中一个docker容器,CPU很高。所以需要排查一下问题。

 
  1. [root@service-slave2 ~]# docker stats

  2. CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS NAMES

  3. fa4b3663ece4 keda6-slave2-information-plan 733.45% 919MiB / 1.5GiB 59.83% 909MB / 748MB 0B / 0B 217

CPU达到 733%
1、进入容器,查看 进程号

 
  1. [root@service-slave2 ~]# docker exec -it fa4b3663ece4 bash

  2. [root@slave2_information_plan project]# top

  3. top - 11:13:50 up 11 days, 23:26, 0 users, load average: 44.96, 41.39, 39.79

  4. Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie

  5. %Cpu(s): 96.8 us, 0.8 sy, 0.0 ni, 2.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

  6. KiB Mem : 64757920 total, 571656 free, 28601872 used, 35584392 buff/cache

  7. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29747760 avail Mem

  8.  
  9. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

  10. 8156 root 20 0 6311560 879272 13592 S 706.2 1.4 6122:00 java

  11. 1 root 20 0 43272 3072 2392 S 0.0 0.0 0:00.40 systemd

  12. 17 root 20 0 47276 9424 9308 S 0.0 0.0 0:00.81 systemd-journal

  13. 28 root 20 0 41628 1392 1276 S 0.0 0.0 0:00.15 systemd-udevd

  14. 66 root 20 0 26384 1600 1460 S 0.0 0.0 0:00.06 systemd-logind

  15. 67 dbus 20 0 58112 2008 1824 S 0.0 0.0 0:00.01 dbus-daemon

  16. 81 root 20 0 112924 3524 3332 S 0.0 0.0 0:00.13 sshd

  17. 83 root 20 0 9924 724 720 S 0.0 0.0 0:00.00 agetty

  18. 8158 root 20 0 13000 828 632 S 0.0 0.0 0:00.48 sh

  19. 8172 root 20 0 3131584 72224 12544 S 0.0 0.1 1:05.19 jstatd

  20. 15282 root 20 0 15244 2072 1656 S 0.0 0.0 0:00.01 bash

  21. 15324 root 20 0 7772 356 280 S 0.0 0.0 0:00.00 sleep

  22. 15327 root 20 0 59616 2060 1488 R 0.0 0.0 0:00.00 top

2、使用top -H -p 进程号查看异常线程、查看线程

 
  1. [root@slave2_information_plan project]# top -H -p 8156

  2. top - 11:14:20 up 11 days, 23:26, 0 users, load average: 39.80, 40.49, 39.54

  3. Threads: 193 total, 30 running, 163 sleeping, 0 stopped, 0 zombie

  4. %Cpu(s): 96.0 us, 0.5 sy, 0.0 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st

  5. KiB Mem : 64757920 total, 540364 free, 28621188 used, 35596368 buff/cache

  6. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29728468 avail Mem

  7.  
  8. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

  9. 14805 root 20 0 6311560 879272 13592 R 43.9 1.4 34:15.23 java

  10. 14847 root 20 0 6311560 879272 13592 R 37.5 1.4 29:11.78 java

  11. 14848 root 20 0 6311560 879272 13592 R 37.2 1.4 28:24.52 java

  12. 14857 root 20 0 6311560 879272 13592 R 35.9 1.4 27:59.42 java

  13. 14865 root 20 0 6311560 879272 13592 R 35.5 1.4 27:07.37 java

  14. 14692 root 20 0 6311560 879272 13592 R 33.2 1.4 30:21.00 java

  15. 14866 root 20 0 6311560 879272 13592 R 29.6 1.4 27:00.60 java

  16. 14834 root 20 0 6311560 879272 13592 R 28.6 1.4 28:56.27 java

  17. 14833 root 20 0 6311560 879272 13592 R 27.2 1.4 29:11.41 java

  18. 14883 root 20 0 6311560 879272 13592 R 26.6 1.4 26:18.58 java

  19. 14849 root 20 0 6311560 879272 13592 R 26.2 1.4 28:11.36 java

  20. 10125 root 20 0 6311560 879272 13592 R 25.6 1.4 56:53.44 java

  21. 14671 root 20 0 6311560 879272 13592 R 24.9 1.4 53:55.84 java

  22. 14818 root 20 0 6311560 879272 13592 R 23.9 1.4 31:27.83 java

  23. 14873 root 20 0 6311560 879272 13592 R 23.6 1.4 26:25.93 java

  24. 8592 root 20 0 6311560 879272 13592 R 23.3 1.4 59:16.66 java

  25. 8304 root 20 0 6311560 879272 13592 R 22.3 1.4 63:12.35 java

  26. 14685 root 20 0 6311560 879272 13592 R 22.3 1.4 30:09.24 java

  27. 8307 root 20 0 6311560 879272 13592 R 21.9 1.4 65:13.92 java

  28. 14819 root 20 0 6311560 879272 13592 R 21.9 1.4 30:13.75 java

  29. 8310 root 20 0 6311560 879272 13592 R 21.3 1.4 868:16.29 java

  30. 8311 root 20 0 6311560 879272 13592 R 19.3 1.4 868:50.86 java

  31. 8309 root 20 0 6311560 879272 13592 R 18.9 1.4 872:06.83 java

  32. 14864 root 20 0 6311560 879272 13592 R 18.9 1.4 27:53.39 java

  33. 8305 root 20 0 6311560 879272 13592 R 18.6 1.4 864:41.14 java

  34. 14658 root 20 0 6311560 879272 13592 R 18.3 1.4 59:20.44 java

  35. 14693 root 20 0 6311560 879272 13592 R 18.3 1.4 32:41.43 java

  36. 8306 root 20 0 6311560 879272 13592 R 14.3 1.4 868:35.26 java

  37. 8308 root 20 0 6311560 879272 13592 R 13.0 1.4 861:23.26 java

  38. 8175 root 20 0 6311560 879272 13592 S 3.0 1.4 46:57.17 java

  39. 15305 root 20 0 6311560 879272 13592 S 0.7 1.4 0:00.55 java

  40. 15318 root 20 0 6311560 879272 13592 S 0.3 1.4 0:00.53 java

3、使用printf "%x\n" 线程号将异常线程号转化为16进制

 
  1. [root@slave2_information_plan project]# printf "%x\n" 14805

  2. 39d5

4、使用jstack 进程号|grep 16进制异常线程号 -A90来定位异常代码的位置(最后的-A90是日志行数,也可以输出为文本文件或使用其他数字)。可以看到异常代码的位置。

 
  1. [root@slave2_information_plan project]# jstack 8156|grep 39d5 -A90

  2. "http-nio-9820-exec-18" #261 daemon prio=5 os_prio=0 tid=0x00007f3e0800f800 nid=0x39d5 runnable [0x00007f3d70580000]

  3. java.lang.Thread.State: RUNNABLE

  4. at sun.util.calendar.ZoneInfo.getTransitionIndex(ZoneInfo.java:339)

  5. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:259)

  6. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236)

  7. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340)

  8. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)

  9. at java.util.Calendar.setTimeInMillis(Calendar.java:1804)

  10. at java.util.GregorianCalendar.add(GregorianCalendar.java:1076)

  11. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.longOfTwoDate(CardPlanServiceJHImpl.java:1969)

  12. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.restartCardPlanZZZ(CardPlanServiceJHImpl.java:2272)

  13. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$FastClassBySpringCGLIB$$26beb7de.invoke(<generated>)

  14. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

  15. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

  16. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

  17. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

  18. at com.keda.util.log.LogRecordAspect.doAroundForService(LogRecordAspect.java:233)

  19. at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)

  20. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  21. at java.lang.reflect.Method.invoke(Method.java:498)

  22. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

  23. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

  24. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

  25. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  26. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

  27. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  28. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

  29. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$EnhancerBySpringCGLIB$$115419c6.restartCardPlanZZZ(<generated>)

  30. at com.keda.information.plan.service.api.ICardPlanServiceApi.restartCardPlanZ(ICardPlanServiceApi.java:259)

  31. at com.keda.information.plan.service.api.ICardPlanServiceApi$$FastClassBySpringCGLIB$$6b2eaa41.invoke(<generated>)

  32. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

  33. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

  34. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

  35. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

  36. at com.keda.util.log.LogRecordAspect.doAroundForApi(LogRecordAspect.java:177)

  37. at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)

  38. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  39. at java.lang.reflect.Method.invoke(Method.java:498)

  40. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

  41. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

  42. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

  43. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  44. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

  45. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  46. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

  47. at com.keda.information.plan.service.api.ICardPlanServiceApi$$EnhancerBySpringCGLIB$$c22f468b.restartCardPlanZ(<generated>)

  48. at sun.reflect.GeneratedMethodAccessor867.invoke(Unknown Source)

  49. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  50. at java.lang.reflect.Method.invoke(Method.java:498)

  51. at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)

  52. at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)

  53. at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

  54. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)

  55. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)

  56. at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

  57. at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)

  58. at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)

  59. at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)

  60. at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)

  61. at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)

  62. at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)

  63. at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

  64. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

  65. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  66. at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

  67. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  68. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  69. at brave.servlet.TracingFilter.doFilter(TracingFilter.java:65)

  70. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  71. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  72. at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124)

  73. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  74. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  75. at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)

  76. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  77. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  78. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  79. at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

  80. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  81. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  82. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  83. at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)

  84. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  85. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  86. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  87. at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)

  88. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  89. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  90. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  91. at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50)

  92. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

注意事项:

如果提示 -bash: jstack: command not found,则可以找到jdk的目录,进入bin目录

        在执行 ./jstack 进程号 | grep 16进制异常线程号 -A90 即可。
如果提示 well-known file is not secure,则是因为账号权限的问题。在/tmp/hsperfdata_$USER/目录,有一个以进程号命名的文件,当我们执行jmap或者jstack出现上叙信息时,先检查执行该命令的用户是否和hsperfdata_$USER这个文件所属的用户一致,如果不一致,切换至成一致再执行。执行:sudo -u 有权限的账号 ./jstack 进程号 | grep 16进制异常线程号 -A50 即可。

5、查看代码

 
  1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

  2. Calendar calendar = Calendar.getInstance();

  3. calendar.setTime(first);

  4. int cnt = 0;

  5. while (calendar.getTime().compareTo(second) != 0) {

  6. calendar.add(Calendar.DATE, 1);

  7. cnt++;

  8. }

  9. return cnt;

  10.  
  11. }

发现一个 while的死循环。修改如下:

 
  1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

  2. Calendar calendar = Calendar.getInstance();

  3. calendar.setTime(first);

  4. int cnt = 0;

  5. if(calendar.getTime().compareTo(second) <= 0){

  6. while (calendar.getTime().compareTo(second) != 0) {

  7. calendar.add(Calendar.DATE, 1);

  8. cnt++;

  9. }

  10. }

  11. return cnt;

  12.  
  13. }

 

这篇关于【异常】记一次,排查docker容器的微服务,CPU飙高问题的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/866256

相关文章

IDEA和GIT关于文件中LF和CRLF问题及解决

《IDEA和GIT关于文件中LF和CRLF问题及解决》文章总结:因IDEA默认使用CRLF换行符导致Shell脚本在Linux运行报错,需在编辑器和Git中统一为LF,通过调整Git的core.aut... 目录问题描述问题思考解决过程总结问题描述项目软件安装shell脚本上git仓库管理,但拉取后,上l

Linux创建服务使用systemctl管理详解

《Linux创建服务使用systemctl管理详解》文章指导在Linux中创建systemd服务,设置文件权限为所有者读写、其他只读,重新加载配置,启动服务并检查状态,确保服务正常运行,关键步骤包括权... 目录创建服务 /usr/lib/systemd/system/设置服务文件权限:所有者读写js,其他

解决docker目录内存不足扩容处理方案

《解决docker目录内存不足扩容处理方案》文章介绍了Docker存储目录迁移方法:因系统盘空间不足,需将Docker数据迁移到更大磁盘(如/home/docker),通过修改daemon.json配... 目录1、查看服务器所有磁盘的使用情况2、查看docker镜像和容器存储目录的空间大小3、停止dock

idea npm install很慢问题及解决(nodejs)

《ideanpminstall很慢问题及解决(nodejs)》npm安装速度慢可通过配置国内镜像源(如淘宝)、清理缓存及切换工具解决,建议设置全局镜像(npmconfigsetregistryht... 目录idea npm install很慢(nodejs)配置国内镜像源清理缓存总结idea npm in

pycharm跑python项目易出错的问题总结

《pycharm跑python项目易出错的问题总结》:本文主要介绍pycharm跑python项目易出错问题的相关资料,当你在PyCharm中运行Python程序时遇到报错,可以按照以下步骤进行排... 1. 一定不要在pycharm终端里面创建环境安装别人的项目子模块等,有可能出现的问题就是你不报错都安装

idea突然报错Malformed \uxxxx encoding问题及解决

《idea突然报错Malformeduxxxxencoding问题及解决》Maven项目在切换Git分支时报错,提示project元素为描述符根元素,解决方法:删除Maven仓库中的resolv... 目www.chinasem.cn录问题解决方式总结问题idea 上的 maven China编程项目突然报错,是

Python爬虫HTTPS使用requests,httpx,aiohttp实战中的证书异步等问题

《Python爬虫HTTPS使用requests,httpx,aiohttp实战中的证书异步等问题》在爬虫工程里,“HTTPS”是绕不开的话题,HTTPS为传输加密提供保护,同时也给爬虫带来证书校验、... 目录一、核心问题与优先级检查(先问三件事)二、基础示例:requests 与证书处理三、高并发选型:

Java服务实现开启Debug远程调试

《Java服务实现开启Debug远程调试》文章介绍如何通过JVM参数开启Java服务远程调试,便于在线上排查问题,在IDEA中配置客户端连接,实现无需频繁部署的调试,提升效率... 目录一、背景二、相关图示说明三、具体操作步骤1、服务端配置2、客户端配置总结一、背景日常项目中,通常我们的代码都是部署到远程

docker 重命名镜像的实现方法

《docker重命名镜像的实现方法》在Docker中无法直接重命名镜像,但可通过添加新标签、删除旧镜像后重新拉取/构建,或在DockerCompose中修改配置文件实现名称变更,感兴趣的可以了解一下... 目录使用标签(Tagging)删除旧的php镜像并重新拉取或构建使用docker Compose在Do

前端导出Excel文件出现乱码或文件损坏问题的解决办法

《前端导出Excel文件出现乱码或文件损坏问题的解决办法》在现代网页应用程序中,前端有时需要与后端进行数据交互,包括下载文件,:本文主要介绍前端导出Excel文件出现乱码或文件损坏问题的解决办法,... 目录1. 检查后端返回的数据格式2. 前端正确处理二进制数据方案 1:直接下载(推荐)方案 2:手动构造