【异常】记一次,排查docker容器的微服务,CPU飙高问题

2024-04-01 04:58

本文主要是介绍【异常】记一次,排查docker容器的微服务,CPU飙高问题,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

项目时 以docker为容器的 微服务项目。 其中一个docker容器,CPU很高。所以需要排查一下问题。

 
  1. [root@service-slave2 ~]# docker stats

  2. CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS NAMES

  3. fa4b3663ece4 keda6-slave2-information-plan 733.45% 919MiB / 1.5GiB 59.83% 909MB / 748MB 0B / 0B 217

CPU达到 733%
1、进入容器,查看 进程号

 
  1. [root@service-slave2 ~]# docker exec -it fa4b3663ece4 bash

  2. [root@slave2_information_plan project]# top

  3. top - 11:13:50 up 11 days, 23:26, 0 users, load average: 44.96, 41.39, 39.79

  4. Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie

  5. %Cpu(s): 96.8 us, 0.8 sy, 0.0 ni, 2.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

  6. KiB Mem : 64757920 total, 571656 free, 28601872 used, 35584392 buff/cache

  7. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29747760 avail Mem

  8.  
  9. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

  10. 8156 root 20 0 6311560 879272 13592 S 706.2 1.4 6122:00 java

  11. 1 root 20 0 43272 3072 2392 S 0.0 0.0 0:00.40 systemd

  12. 17 root 20 0 47276 9424 9308 S 0.0 0.0 0:00.81 systemd-journal

  13. 28 root 20 0 41628 1392 1276 S 0.0 0.0 0:00.15 systemd-udevd

  14. 66 root 20 0 26384 1600 1460 S 0.0 0.0 0:00.06 systemd-logind

  15. 67 dbus 20 0 58112 2008 1824 S 0.0 0.0 0:00.01 dbus-daemon

  16. 81 root 20 0 112924 3524 3332 S 0.0 0.0 0:00.13 sshd

  17. 83 root 20 0 9924 724 720 S 0.0 0.0 0:00.00 agetty

  18. 8158 root 20 0 13000 828 632 S 0.0 0.0 0:00.48 sh

  19. 8172 root 20 0 3131584 72224 12544 S 0.0 0.1 1:05.19 jstatd

  20. 15282 root 20 0 15244 2072 1656 S 0.0 0.0 0:00.01 bash

  21. 15324 root 20 0 7772 356 280 S 0.0 0.0 0:00.00 sleep

  22. 15327 root 20 0 59616 2060 1488 R 0.0 0.0 0:00.00 top

2、使用top -H -p 进程号查看异常线程、查看线程

 
  1. [root@slave2_information_plan project]# top -H -p 8156

  2. top - 11:14:20 up 11 days, 23:26, 0 users, load average: 39.80, 40.49, 39.54

  3. Threads: 193 total, 30 running, 163 sleeping, 0 stopped, 0 zombie

  4. %Cpu(s): 96.0 us, 0.5 sy, 0.0 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st

  5. KiB Mem : 64757920 total, 540364 free, 28621188 used, 35596368 buff/cache

  6. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29728468 avail Mem

  7.  
  8. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

  9. 14805 root 20 0 6311560 879272 13592 R 43.9 1.4 34:15.23 java

  10. 14847 root 20 0 6311560 879272 13592 R 37.5 1.4 29:11.78 java

  11. 14848 root 20 0 6311560 879272 13592 R 37.2 1.4 28:24.52 java

  12. 14857 root 20 0 6311560 879272 13592 R 35.9 1.4 27:59.42 java

  13. 14865 root 20 0 6311560 879272 13592 R 35.5 1.4 27:07.37 java

  14. 14692 root 20 0 6311560 879272 13592 R 33.2 1.4 30:21.00 java

  15. 14866 root 20 0 6311560 879272 13592 R 29.6 1.4 27:00.60 java

  16. 14834 root 20 0 6311560 879272 13592 R 28.6 1.4 28:56.27 java

  17. 14833 root 20 0 6311560 879272 13592 R 27.2 1.4 29:11.41 java

  18. 14883 root 20 0 6311560 879272 13592 R 26.6 1.4 26:18.58 java

  19. 14849 root 20 0 6311560 879272 13592 R 26.2 1.4 28:11.36 java

  20. 10125 root 20 0 6311560 879272 13592 R 25.6 1.4 56:53.44 java

  21. 14671 root 20 0 6311560 879272 13592 R 24.9 1.4 53:55.84 java

  22. 14818 root 20 0 6311560 879272 13592 R 23.9 1.4 31:27.83 java

  23. 14873 root 20 0 6311560 879272 13592 R 23.6 1.4 26:25.93 java

  24. 8592 root 20 0 6311560 879272 13592 R 23.3 1.4 59:16.66 java

  25. 8304 root 20 0 6311560 879272 13592 R 22.3 1.4 63:12.35 java

  26. 14685 root 20 0 6311560 879272 13592 R 22.3 1.4 30:09.24 java

  27. 8307 root 20 0 6311560 879272 13592 R 21.9 1.4 65:13.92 java

  28. 14819 root 20 0 6311560 879272 13592 R 21.9 1.4 30:13.75 java

  29. 8310 root 20 0 6311560 879272 13592 R 21.3 1.4 868:16.29 java

  30. 8311 root 20 0 6311560 879272 13592 R 19.3 1.4 868:50.86 java

  31. 8309 root 20 0 6311560 879272 13592 R 18.9 1.4 872:06.83 java

  32. 14864 root 20 0 6311560 879272 13592 R 18.9 1.4 27:53.39 java

  33. 8305 root 20 0 6311560 879272 13592 R 18.6 1.4 864:41.14 java

  34. 14658 root 20 0 6311560 879272 13592 R 18.3 1.4 59:20.44 java

  35. 14693 root 20 0 6311560 879272 13592 R 18.3 1.4 32:41.43 java

  36. 8306 root 20 0 6311560 879272 13592 R 14.3 1.4 868:35.26 java

  37. 8308 root 20 0 6311560 879272 13592 R 13.0 1.4 861:23.26 java

  38. 8175 root 20 0 6311560 879272 13592 S 3.0 1.4 46:57.17 java

  39. 15305 root 20 0 6311560 879272 13592 S 0.7 1.4 0:00.55 java

  40. 15318 root 20 0 6311560 879272 13592 S 0.3 1.4 0:00.53 java

3、使用printf "%x\n" 线程号将异常线程号转化为16进制

 
  1. [root@slave2_information_plan project]# printf "%x\n" 14805

  2. 39d5

4、使用jstack 进程号|grep 16进制异常线程号 -A90来定位异常代码的位置(最后的-A90是日志行数,也可以输出为文本文件或使用其他数字)。可以看到异常代码的位置。

 
  1. [root@slave2_information_plan project]# jstack 8156|grep 39d5 -A90

  2. "http-nio-9820-exec-18" #261 daemon prio=5 os_prio=0 tid=0x00007f3e0800f800 nid=0x39d5 runnable [0x00007f3d70580000]

  3. java.lang.Thread.State: RUNNABLE

  4. at sun.util.calendar.ZoneInfo.getTransitionIndex(ZoneInfo.java:339)

  5. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:259)

  6. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236)

  7. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340)

  8. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)

  9. at java.util.Calendar.setTimeInMillis(Calendar.java:1804)

  10. at java.util.GregorianCalendar.add(GregorianCalendar.java:1076)

  11. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.longOfTwoDate(CardPlanServiceJHImpl.java:1969)

  12. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.restartCardPlanZZZ(CardPlanServiceJHImpl.java:2272)

  13. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$FastClassBySpringCGLIB$$26beb7de.invoke(<generated>)

  14. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

  15. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

  16. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

  17. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

  18. at com.keda.util.log.LogRecordAspect.doAroundForService(LogRecordAspect.java:233)

  19. at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)

  20. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  21. at java.lang.reflect.Method.invoke(Method.java:498)

  22. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

  23. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

  24. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

  25. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  26. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

  27. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  28. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

  29. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$EnhancerBySpringCGLIB$$115419c6.restartCardPlanZZZ(<generated>)

  30. at com.keda.information.plan.service.api.ICardPlanServiceApi.restartCardPlanZ(ICardPlanServiceApi.java:259)

  31. at com.keda.information.plan.service.api.ICardPlanServiceApi$$FastClassBySpringCGLIB$$6b2eaa41.invoke(<generated>)

  32. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

  33. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

  34. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

  35. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

  36. at com.keda.util.log.LogRecordAspect.doAroundForApi(LogRecordAspect.java:177)

  37. at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)

  38. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  39. at java.lang.reflect.Method.invoke(Method.java:498)

  40. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

  41. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

  42. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

  43. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  44. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

  45. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

  46. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

  47. at com.keda.information.plan.service.api.ICardPlanServiceApi$$EnhancerBySpringCGLIB$$c22f468b.restartCardPlanZ(<generated>)

  48. at sun.reflect.GeneratedMethodAccessor867.invoke(Unknown Source)

  49. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  50. at java.lang.reflect.Method.invoke(Method.java:498)

  51. at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)

  52. at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)

  53. at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

  54. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)

  55. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)

  56. at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

  57. at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)

  58. at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)

  59. at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)

  60. at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)

  61. at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)

  62. at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)

  63. at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

  64. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

  65. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  66. at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

  67. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  68. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  69. at brave.servlet.TracingFilter.doFilter(TracingFilter.java:65)

  70. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  71. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  72. at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124)

  73. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  74. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  75. at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)

  76. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  77. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  78. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  79. at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

  80. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  81. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  82. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  83. at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)

  84. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  85. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  86. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  87. at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)

  88. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

  89. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

  90. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

  91. at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50)

  92. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

注意事项:

如果提示 -bash: jstack: command not found,则可以找到jdk的目录,进入bin目录

        在执行 ./jstack 进程号 | grep 16进制异常线程号 -A90 即可。
如果提示 well-known file is not secure,则是因为账号权限的问题。在/tmp/hsperfdata_$USER/目录,有一个以进程号命名的文件,当我们执行jmap或者jstack出现上叙信息时,先检查执行该命令的用户是否和hsperfdata_$USER这个文件所属的用户一致,如果不一致,切换至成一致再执行。执行:sudo -u 有权限的账号 ./jstack 进程号 | grep 16进制异常线程号 -A50 即可。

5、查看代码

 
  1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

  2. Calendar calendar = Calendar.getInstance();

  3. calendar.setTime(first);

  4. int cnt = 0;

  5. while (calendar.getTime().compareTo(second) != 0) {

  6. calendar.add(Calendar.DATE, 1);

  7. cnt++;

  8. }

  9. return cnt;

  10.  
  11. }

发现一个 while的死循环。修改如下:

 
  1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

  2. Calendar calendar = Calendar.getInstance();

  3. calendar.setTime(first);

  4. int cnt = 0;

  5. if(calendar.getTime().compareTo(second) <= 0){

  6. while (calendar.getTime().compareTo(second) != 0) {

  7. calendar.add(Calendar.DATE, 1);

  8. cnt++;

  9. }

  10. }

  11. return cnt;

  12.  
  13. }

 

这篇关于【异常】记一次,排查docker容器的微服务,CPU飙高问题的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/866256

相关文章

MySQL主从同步延迟问题的全面解决方案

《MySQL主从同步延迟问题的全面解决方案》MySQL主从同步延迟是分布式数据库系统中的常见问题,会导致从库读取到过期数据,影响业务一致性,下面我将深入分析延迟原因并提供多层次的解决方案,需要的朋友可... 目录一、同步延迟原因深度分析1.1 主从复制原理回顾1.2 延迟产生的关键环节二、实时监控与诊断方案

SQLyog中DELIMITER执行存储过程时出现前置缩进问题的解决方法

《SQLyog中DELIMITER执行存储过程时出现前置缩进问题的解决方法》在SQLyog中执行存储过程时出现的前置缩进问题,实际上反映了SQLyog对SQL语句解析的一个特殊行为,本文给大家介绍了详... 目录问题根源正确写法示例永久解决方案为什么命令行不受影响?最佳实践建议问题根源SQLyog的语句分

解决IDEA报错:编码GBK的不可映射字符问题

《解决IDEA报错:编码GBK的不可映射字符问题》:本文主要介绍解决IDEA报错:编码GBK的不可映射字符问题,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录IDEA报错:编码GBK的不可映射字符终端软件问题描述原因分析解决方案方法1:将命令改为方法2:右下jav

Java 中的 @SneakyThrows 注解使用方法(简化异常处理的利与弊)

《Java中的@SneakyThrows注解使用方法(简化异常处理的利与弊)》为了简化异常处理,Lombok提供了一个强大的注解@SneakyThrows,本文将详细介绍@SneakyThro... 目录1. @SneakyThrows 简介 1.1 什么是 Lombok?2. @SneakyThrows

MyBatis模糊查询报错:ParserException: not supported.pos 问题解决

《MyBatis模糊查询报错:ParserException:notsupported.pos问题解决》本文主要介绍了MyBatis模糊查询报错:ParserException:notsuppo... 目录问题描述问题根源错误SQL解析逻辑深层原因分析三种解决方案方案一:使用CONCAT函数(推荐)方案二:

在 Spring Boot 中实现异常处理最佳实践

《在SpringBoot中实现异常处理最佳实践》本文介绍如何在SpringBoot中实现异常处理,涵盖核心概念、实现方法、与先前查询的集成、性能分析、常见问题和最佳实践,感兴趣的朋友一起看看吧... 目录一、Spring Boot 异常处理的背景与核心概念1.1 为什么需要异常处理?1.2 Spring B

判断PyTorch是GPU版还是CPU版的方法小结

《判断PyTorch是GPU版还是CPU版的方法小结》PyTorch作为当前最流行的深度学习框架之一,支持在CPU和GPU(NVIDIACUDA)上运行,所以对于深度学习开发者来说,正确识别PyTor... 目录前言为什么需要区分GPU和CPU版本?性能差异硬件要求如何检查PyTorch版本?方法1:使用命

Redis 热 key 和大 key 问题小结

《Redis热key和大key问题小结》:本文主要介绍Redis热key和大key问题小结,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友参考下吧... 目录一、什么是 Redis 热 key?热 key(Hot Key)定义: 热 key 常见表现:热 key 的风险:二、

IntelliJ IDEA 中配置 Spring MVC 环境的详细步骤及问题解决

《IntelliJIDEA中配置SpringMVC环境的详细步骤及问题解决》:本文主要介绍IntelliJIDEA中配置SpringMVC环境的详细步骤及问题解决,本文分步骤结合实例给大... 目录步骤 1:创建 Maven Web 项目步骤 2:添加 Spring MVC 依赖1、保存后执行2、将新的依赖

Spring 中的循环引用问题解决方法

《Spring中的循环引用问题解决方法》:本文主要介绍Spring中的循环引用问题解决方法,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友参考下吧... 目录什么是循环引用?循环依赖三级缓存解决循环依赖二级缓存三级缓存本章来聊聊Spring 中的循环引用问题该如何解决。这里聊