客户系统健康检查分析报告.docx

1、客户系统健康检查分析报告XXXX客户系统健康检查报告一、 综述 3二、 系统环境 3核心业务系统 3三、 工作日志 4四、 系统健康检查详细列表 41. 核心业务系统 4五、 日常维护建议 10六、 故障出现时，系统处理的步骤 11七、 命令介绍 11一、综述应XXXX客户要求，倚博公司中间件技术专家针对客户的核心业务系统做了一次完整的系统健康检查。整体上讲，公司的核心系统运行比较稳定，故障率和客户业务系统的MTBF (Mean Time Between Failure)都比较小。但从系统维护的治理工作的规范和完善角度讲，还未建立完善中间件的维护，监控的方案。现时期，要紧是利用维护人员的现场应

2、急处理经验保证系统的运行正常，未合理的规划维护流程，未确认中间件的监控指标，未建立完善的监控，预警和应急处理方案。本文档总结了这次现场工作的各项情况，并依照现场维护人员的要求，增加了系统维护方法的扼要讲明。二、系统环境核心业务系统客户系统名称：核心业务系统OS CPU：8OS Memory: 16 GBOS: Red Hat AS 4 KENERL: 2.6.9-42.ELsmpJBOSS Application Server 4.2.3JDK：Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)Java H

3、otSpot(TM) Server VM (build 1.5.0_17-b04, mixed mode)三、工作日志2008年6月5日 上午 9：30 系统健康检查四、系统健康检查详细列表1.核心业务系统 操作系统配置检查系统配置：名称建议值实际值是否符合要求操作系统版本Linux/HpRed Hat AS4Y主机名N/AApp1YCPU 1GHZ8*1.9G,Y内存 2G16GY磁盘空间20G360GY系统内核参数：名称建议值实际值是否符合要求讲明文件句柄数4096或更大8192Y进程的最大文件句柄数JAVA虚拟机配置检查虚拟机安装路径：/opt/jdk1.5.0_17/名称建议值实际值是

4、否符合要求讲明JVM 版本1.5JDK 1.5.0_17YJVM HEAP初始值 -Xms514M1024MYJVM HEAP最大值 -Xmx512M-1536M1536MYPermsize64M256MJBOSS配置检查安装路径：/opt/jboss/jboss-4.2.3.GA参数值建议值讲明线程数400一个CPU对应50个线程左右线程的个数应与CPU的处理能力匹配。JSP编译的调试信息标志falsefalse生产环境中建议将JSP编译调试功能关闭，减少因编译占用系统资源对系统的阻碍。开发模式的设置falsefalse应用使用的连接池数量400400生产环境中建议将连接的数量与线程总数保持

5、统一。应用使用的连接池初始值100400最小值和最大值保持一致，幸免运行状态在创建连接时消耗系统资源。连接池Prepared Statement Cache无10缺省JBOSS服务器是可不能Cache Prepared Statement对象的，为了提高性能我们能够设定每个连接Cache的 Prepared Statement对象的个数运行期系统状态检查名称平均值峰值是否异常CPU35%以下60%未出现长时刻CPU使用率超过100%内存使用3.5G以下无页交换的情况磁盘空间20%满足应用日志或者JBOSS日志所用磁盘空间要求网络流量200K/S800K/S无网络堵塞和网络风暴运行期虚拟机状态检

6、查名称实际情况是否符合要求讲明JVM HEAP最小值280MYJVM HEAP最大值1100MY为今天的观看结果运行期JBOSS组件状态检查名称正常值实际值是否符合要求讲明线程使用情况使用率50%70%Y启动目录core文件的检查在JBOSS启动目录中发觉“core”文件和“hs_err_pid*.log”的错误日志文件，讲明系统出现过JVM core dump和异常退出故障，故障出现在一个虚拟机的解压缩操作，希望能通过进一步与开发商，维护人员沟通找出具体的故障缘故，并提出解决方案。日志检查名称建议值实际值是否符合要求讲明日志中Error信息检查Review All LogsReview Al

7、l LogsN对日志中Error消息进行检查，参考下面例子。日志文件中发觉错误信息如下：1) 启动JBoss时报如下错误：出现过一次22:53:52,907 ERROR MainDeployer Could not create deployment: file:/D:/新建文件夹/jboss-4.2.2.GA/server/default/conf/jboss-service.xml org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableExcep

8、tion) at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:196) at org.jboss.system.ServiceController.install(ServiceController.java:226) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.D

9、elegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocatio

10、n.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy4.install(Unknown Source) at org.jboss.deployment.SARDeploye

11、r.create(SARDeployer.java:249) at org.jboss.deployment.MainDeployer.create(MainDeployer.java:969) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:818) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766) a

12、t sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(Re

13、flectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invo

14、ke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke

15、(MBeanProxyExt.java:210) at $Proxy5.deploy(Unknown Source) at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482) at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362) at org.jboss.Main.boot(Main.java:200) at org.jboss.Main$1.run(Main.java:508) at java.lang.Thread.run(Unknown

16、Source) Caused by: java.lang.reflect.UndeclaredThrowableException at org.jboss.system.ServiceCreator.install(ServiceCreator.java:236) at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:451) at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:171) .

17、36 more Caused by: java.lang.Error: ERROR: it turns out the root cause is not always a runtime exception! at org.jboss.mx.server.MBeanServerImpl.handleInstantiateExceptions(MBeanServerImpl.java:1295) at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:1237) at org.jboss.mx.server

18、.MBeanServerImpl.instantiate(MBeanServerImpl.java:286) at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:344) at org.jboss.system.ServiceCreator.install(ServiceCreator.java:157) . 38 morejboss-service.x配置错误，通过修改具体错误配置，并解决了那个问题。2)NoClassDefFoundError错误，当前的类装载器不能找到那个类文件，频率：出现一次ja

19、va.lang.NoClassDefFoundError: Could not initialize class org.jboss.aspects.security.SecurityActions$ContextInfoActions at org.jboss.aspects.security.SecurityActions.getContextException(SecurityActions.java:467) at org.jboss.aspects.security.AuthenticationInterceptor.authenticate(AuthenticationInterc

20、eptor.java:131) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:66) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

21、at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInv

22、ocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.jav

23、a:84) at $Proxy147.getInitConfigAsInteger(Unknown Source) at com.police.alarm.AlarmAction.execute(AlarmAction.java:28) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236) at org.a

24、pache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.Appli

25、cationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.police.filter.SystemUserFilter.doFilter(SystemUserFilter.java:61) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilte

26、r(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.police.filter.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:37) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application

27、FilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235

28、) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.securit

29、y.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(Stand

30、ardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.

31、catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at .JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Threa