suddenly for some reason tomcat server is not runnin/starting/stopping.
below is the result I get when i run the startup command,
C:Program Filesjasperserver-3.5.0apache-tomcatbin>startup
Using CATALINA_BASE: C:Program Filesjasperserver-3.5.0apache-tomcat
Using CATALINA_HOME: C:Program Filesjasperserver-3.5.0apache-tomcat
Using CATALINA_TMPDIR: C:Program Filesjasperserver-3.5.0apache-tomcattemp
Using JRE_HOME: C:PROGRA~1JASPER~1.0javajre
The tomcat window pops up for a split of a second and goes away.
(I have another java instance installed under c:program files)
Help!!!
Update:
Logs from catalina file
Mar 22, 2011 3:41:50 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Mar 22, 2011 3:41:50 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1017 ms
Mar 22, 2011 3:41:51 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Mar 22, 2011 3:41:51 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.20
Mar 22, 2011 3:41:51 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Mar 22, 2011 3:42:06 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Mar 22, 2011 3:42:06 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Mar 22, 2011 3:42:06 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/43 config=null
Mar 22, 2011 3:42:06 AM org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
Mar 22, 2011 3:42:06 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 15374 ms
Permalink to this page: https://cwiki.apache.org/confluence/x/yColBg
Troubleshooting and Diagnostics techniques.
Table of Contents
Techniques & Reference
- How To: Capture a thread dump
- How To: Capture a heap dump
- How To: Examine a Stacktrace
- How To: Configure Tomcat for debugging
- FAQ: Developing
- FAQ: Memory
- Tomcat Memory Leak Protection
- Notes on using JMX clients
JMX Clients
- JJConsole: Documentation
- VisualVM: Documentation, Project
JDK tools
- jinfo — Prints JVM process info
- jstack — Prints thread stack traces
- jmap — Dumps heap and shows heap status
- jhat — Heap Analyzer Tool
- jcmd — Multitool intended to replace the above JDK tools
Profilers & Heap Analyzers
- Eclipse Memory Analyzer (MAT)
- YourKit Profiler
-
VisualVM Docs
Notes on using JMX clients
When running a JMX client (JConsole, VisualVM) on the same machine as the target JVM process it is possible to connect without pre-configuring a JMX port, using the local connector stub. This method relies on being able to create a protected temporary file, accessible only to a user with administrator privileges. Java processes which are accessible via the local connector will automatically appear in the client.
NB(1) On Windows, this means that the temporary directory must be located on an NTFS formatted disk. See the following link for more details.
NB(2) On Windows, if Tomcat is started using a service wrapper, this will prevent JConsole & VisualVM from using the local JMX connector stub.
Java 5 JConsole and Remote Management FAQ
From Java 6 onward a process does not need to have the management agent enabled when it starts, as the Attach API permits the management agent to be activated on demand.
Common Troubleshooting Scenario
If you have already looked into Tomcat logs, there are no error messages, and you just want to find out what is going on, you may try the following
- Look into Tomcat access log (the log file generated by AccessLogValve).
- If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).
- You will see what IP address your client is using, and whether it is using an IPv4 (
127.0.0.1
) or IPv6 address (0:0:0:0:0:0:0:1
).
Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4.
- Take a thread dump. This way you will find out what Tomcat is actually doing.
- If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress.
- Try debugging.
- A good place for a breakpoint is
org.apache.catalina.connector.CoyoteAdapter.service()
method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.
- A good place for a breakpoint is
- If you did a long-awaited upgrade, jumping over several years worth of Tomcat releases, and something broke, and you have no clue,
- Reading Migration guides may help.
- It may help to do a binary search (aka bisecting) to locate the version of Tomcat that triggered the change. If your issue is easy to reproduce, it may be pretty fast. Just 7-8 tries may cover a range of 100 versions. Once you know the version and its release date, the following resources are available:
- The release announcement.
See «former announcements» link at the bottom of the front page of the Apache Tomcat site.
An announcement mail message can also be found in the archives of the «announce@» mailing list. - The changelog. A release announcement usually has a link to it.
- Archives of the «users@» mailing list. You may look for discussions that happened a month or two after the release.
- The release announcement.
Troubleshooting unexpected Response state problems
If you encounter problems that manifest themselves as accessing a request or response that is an inconsistent state, the main suspect is your own web application (or a library that it uses) keeping a reference to Request or Response objects outside of their life cycle. Examples: BZ 61289, BZ 58457.
The lifetime of the Response object is documented in the Servlet specification. Quoting from section «5.8 Lifetime of the Response Object» of Servlet 4.0 specification:
«Each response object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method, unless the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated request is started, then the response object remains valid until complete method on AsyncContext is called.»
In case of asynchronous processing, when an error occurs Tomcat notifies all registered AsyncListener
s and then calls complete()
automatically if none of the listeners have called it yet. (Reference: 61768)
Also see sections «2.3.3.4 Thread Safety» and «3.13 Lifetime of the Request Object» of the same specification.
To troubleshoot the issue:
- Make sure that your Tomcat is configured to discard facades to its internal objects when request processing completes. This makes it easier to spot illegal access when it happens, instead of waiting until side effects of such access become visible. Essentially, it protects Tomcat internals from misbehaving web applications.
This feature is always on when you are running Tomcat with a Java Security Manager being enabled. Starting with Tomcat 10.0 this feature is enabled by default. It is disabled by default in earlier versions of Tomcat. The way this feature is configured differs between versions: it is controlled by an attribute on Connector element or by a system property.
If you are running Tomcat 9.0 or earlier, do both of the following:
— Set the following system property in Tomcat configuration:
org.apache.catalina.connector.RECYCLE_FACADES=true
— Add the following attribute to all Connector elements:
discardFacades
=»true»The Connector attribute was added in Tomcat 10.0.0-M1, 9.0.31, 8.5.51 and 7.0.100. The system property is an older way to configure this feature. In case of a doubt, or if you are switching back and forth between versions while troubleshooting the issue, it is safer to configure both of them.
This feature is also mentioned on the Security Considerations page in Tomcat documentation. You can also search the archives of the Tomcat users’ mailing lists for previous discussions mentioning the RECYCLE_FACADES flag.
Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, if the problem manifests as a security issue, see how to report it.
There are some known examples of broken libraries / APIs:
- Read about Java ImageIO issue — an issue with javax.imageio.ImageIO API. It may have already been fixed as it is an old issue, but there are no clear records of it.
- Read about an issue in PD4ML, a library that is used to generate PDF files, — fixed in their version 3.8.0, earlier versions may be affected.
Troubleshooting «Too many open file descriptors»
The code that opens the descriptors can be identified using a tool such as http://file-leak-detector.kohsuke.org/
The Apache Tomcat® software is an open source implementation of the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies.
1. The tools
- Apache Tomcat 8
2. Introduction
In this article We are going to discover some techniques to dig and troubleshoot errors on the Apache Tomcat Server.
3. Prerequisites
- JDK 8 Installed
- Tomcat 8 installed and running
4. Launch Tomcat
Go to Tomcat Install bin directory.
Type the command
start tomcat
C:JavaApache Tomcat 8.0.27bin>startup.bat
A new window opens and you get the following output
startup output
Using CATALINA_BASE: "C:JavaApache Tomcat 8.0.27" Using CATALINA_HOME: "C:JavaApache Tomcat 8.0.27" Using CATALINA_TMPDIR: "C:JavaApache Tomcat 8.0.27temp" Using JRE_HOME: "C:Javajdk1.8.0_40" Using CLASSPATH: "C:JavaApache Tomcat 8.0.27binbootstrap.jar;C:JavaApache Tomcat 8.0.27bintomcat-juli.jar"
And in the window opened by the script you get last lines like that:
Tomcat console
INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] INFO [main] org.apache.catalina.startup.Catalina.startServer startup in 2649 ms
Indicating that Tomcat has started. Check that Tomcat is started opening the link
- http://localhost:8080
You get the following output on the browser:
5. Get the tomcat PID
The JDK provides a tool to get the PID of running Java Applications. Make sure that the JDK bin directory is on your PATH.
Type the following command in the command prompt
View the PATH variable
C:JavaApache Tomcat 8.0.27bin>echo %PATH%
You get all variables in your PATH. The windows PATH is a variable that tells windows where to look for executables.
Type the following command in the command prompt
View the PATH variable
C:JavaApache Tomcat 8.0.27bin>jps
You should get an output similar to this:
These are Java processes. The process Bootstrap is Tomcat server daemon.
6. Launch jvisualvm
Java VisualVM is a tool that provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM), and for troubleshooting and profiling these applications.
Now we are going to launch jvisualvm to start inspecting Tomcat.
Type in the command console:
jvisualvm
C:Javajdk1.6.0_33bin>jvisualvm
With this command you launch the following application:
6.1 Process Overview Screen
Double click on the Apache Tomcat process and you get to the Process Overview screen:
Here you can see:
The PID of Tomcat.
The host.
The main class.
The Java Home.
The JVM Flags.
The JVM Arguments used to launch Tomcat.
And a lot of useful information about the Tomcat process.
6.2 Process Monitor Screen
You can monitor Tomcat and view real-time, data on the memory heap, thread activity, and the classes loaded.
Click on the monitor tab.
Here we can activate the garbage collector and generate head dumps.
A heap dump is a photo of the memory of a Java process. The photo contains information about the Java objects and classes in the heap at the moment the heap dump is triggered.
If your Tomcat server crashes with an OutOfMemoryError a heap dump useful to troubleshoot these errors.
6.3 Threads Screen
Here you can see all threads that Tomcat is using. Also the status of each thread and the time running.
Here we can generate a Thread dump. A thread dump is a snapshot of the state of all threads that are part of the process.
Click on the button [Thread Dump] to generate a thread dump.
We get a lot of text.
We have to make several consecutive thread dumps of Apache Tomcat. If one or more threads in Tomcat is stuck waiting for a lock to be released you need to keep a closer look to see whats happening.
In general most troubles in Tomcat are caused by a bugged library and thats why we are sniffing inside thread dumps.
7. Tomcat Log files
Inside the Tomcat install directory you can find the Tomcat log directory.
These log files can be useful to troubleshoot an error. There are multiples types of log files:
These logs are generated by date and by type. The catalina log is the global log. It is the stdout stream for the Tomcat JVM. The localhost log files is the log for a virtual host within Tomcat.
8. Conclusion
There is no standard way to troubleshoot problems with Tomcat. You need to dig inside the process in the virtual machine and carefully examine the Tomcat logs to get into the root of a problem.
Most problems are caused by libraries and lack of memory. In this tutorial we provided some tools to dig inside Tomcat.
The JDK provides a lot of tools to troubleshoot these errors and to help you know how to investigate deep inside Tomcat.
I’m a self taught programmer, I began programming back in 1991 using an IBM A10 mainframe with Pascal an Assembler IBM 360/70 emulator and Turbo C on a X86 PC, since that I work for the banking industry with emerging technologies like Fox Pro, Visual Fox Pro, Visual Basic, Visual C++, Borland C++, lately I moved out to the Airline industry, leading designing and programming in-house web applications with Flex, Actionscript, PHP, Python and Rails and in the last 7 years I focused all my work in Java, working on Linux servers using GlassFish, TomCat, Apache and MySql.
Issue
- Apache Tomcat is not running
- Error message from a Java update is received (error code 1)
Solution
- Check the Apache Tomcat service
- Error message from a Java update is received (error code 1)
I. Check the Apache Tomcat service
Apache Tomcat requires Java:
-
Make sure that Java, ESMC, and Apache Tomcat have the same bitness (32-bit or 64-bit).
-
If you have multiple Java versions installed on your system, we recommend that you uninstall older Java versions and keep only the latest Java.
-
Starting January 2019, Oracle JAVA SE 8 public updates for business, commercial or production use will require a commercial license. If you do not purchase a JAVA SE subscription, you can use this guide to transition to a no-cost alternative.
In cases where the ESET Security Management Center Web Console (ESMC Web Console) is not running, check whether the Apache Tomcat service is running:
-
Click Start → Run, type services.msc and then click OK.
Windows Server 2012 users: Click the Windows key + R, type services.msc and press Enter. -
Locate the Apache Tomcat service, and then verify that Running is listed in the Status column. If it is not running, try to start the service manually by selecting it and clicking Start.
II. Error message from a Java update is received (error code 1)
While trying to start the service, you might receive the following error message, which may result from a Java update:
«Windows could not start the Apache Tomcat on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to the service-specific error code 1.»
If you receive this error while attempting to start the Apache Tomcat service, follow the step-by-step instructions below to resolve your issue:
-
Navigate to your
%TOMCAT_HOME%bin
directory (for example,C:Program FilesApache Software FoundationTomcat 7.0bin
) and double-click tomcat7w.exe (Tomcat_folder
and tomcat7w.exe are examples — use the actual names on your system).
Tomcat version
Depending on your version of Tomcat, the version number may vary.
-
Click the Java tab, confirm the check box next to Use default is selected and click OK.
Optionally, deselect the check box next to Use default and select the path to Java virtual machine file (C:[Java folder path]binserverjvm.dll or C:[Java folder path]binclientjvm.dll) and click OK.
-
Start the Apache Tomcat service. The service should start without issue. If you are still experiencing issues, see our Knowledgebase article.
Tomcat 5.5 or later is required! Tomcat6 is recommended. Are you using an older version of Tomcat?
General troubleshooting
The first thing to do when troubleshooting tomcat is to check the tomcat logs. They are found at C:Program FilesApache Software FoundationTomcat 6.0logs — look for the one named stdout_yyyymmdd.txt where yyyymmdd is today’s date. If this log is too long and you can’t find the relevant part, you can stop the tomcat service (in Windows using control panel, administrative tools, services), rename the file to stdout_yyyymmdd_old.txt, restart tomcat and it will start writing a new log at stdout_yyyymmdd.txt.
Sluggish performance
Confirm that you have installed tomcat from the tomcat.apache.org website. The apt-get install of Ubuntu tomcat6 has terrible performance consequences.
Running Tomcat as a Service on Ubuntu
If you’re trying to run Tomcat as a serving on Ubuntu, you will likely run into permission issues. The easiest way to solve this issue is to disable the Java security manager in /etc/init.d/tomcat6.
Error:
java.security.AccessControlException: access denied (java.io.FilePermission /usr/share/tomcat6/webapps/openmrs/WEB-INF/dwr-modules.xml delete)
Solution:
# Use the Java security manager? (yes/no) TOMCAT6_SECURITY=no
Error: prunsrv.c Failed creating … jvm.dll
If you cannot start the Tomcat service, try checking the tomcat logs at C:/Program Files/Apache Software Foundation/Tomcat/logs. If these logs give errors like «Failed creating java C:Program FilesJavajre1.6.0binclientjvm.dll» then do the following:
- Search for msvcr71.dll on your hard drive
- Copy this file to C:WindowsSystem32
Tomcat hangs and stops responding when uploading the war file
When uploading the war file, Tomcat hangs and stops responding. This typically occurs if you have not defined a MySQL user account that OpenMRS can use to access the database or if you have not granted this user full access to the OpenMRS database.
The default username is test with password test. The default username/password can be overridden in the openmrs_runtime_properties.properties configuration file.
To fix this: use Navicat or the MySQL administration tool (available from MySQL) to verify that you have a user with username test and password test and this user has full access to the OpenMRS database.
Tomcat error when uploading war file — FileUploadBase$SizeLimitExceededException
When uploading the war file, you can get the following error message with Tomcat’s default settings.
description The server encountered an internal error () that prevented it from fulfilling this request. exception java.lang.IllegalStateException: org.apache.tomcat.util.http.fileupload.FileUploadBase$SizeLimitExceededException: the request was rejected because its size (54624465) exceeds the configured maximum (52428800)
To fix this: Go to the web.xml of the manager application (for instance it could be under /tomcat7/webapps/manager/WEB-INF/web.xml).
Increase the max-file-size and max-request-size:
<multipart-config> <!– 50MB max –> <max-file-size>52428800</max-file-size> <max-request-size>52428800</max-request-size> <file-size-threshold>0</file-size-threshold> </multipart-config>
Cannot connect to Tomcat on port 8080
This port is sometimes used by other programs, such as Popfile and TivoServer. You can use the Windows «netstat -ao» command to discover if another process is using port 8080.
In one case, software for a HP Laserjet 2840 printer was trying to use port 8005, which conflicts with Tomcat startup.
java.lang.OutOfMemoryError PermGen space
This error will appear after Tomcat has exhausted all of its memory. This is typically caused by using the «Update» or «Reload» function on a webapp too many times. Currently, Tomcat and/or the JVM isn’t releasing all of the memory when a webapp is destroyed/recreated. After several reloads Tomcat is out of its allotted memory and will hang or freeze. Read more at ?Troubleshooting Memory Errors.
Can’t deploy applications
This solution assumes you are using a Linux machine. If you are not, you may have to «translate» the general method into a Windows environment. First, undeploy the application from the Tomcat web interface. Next, log into the server’s command line interface as root and find the process id using
It will likely be the first item returned and you will know because it’s about three or four lines long. Find the process id (the first number on that line), and kill it with
Now restart the service with
Log back into the Tomcat web interface and deploy your application.
Reporting OpenMRS Errors by Email
OpenMRS logs all errors using the log4j framework and while this is useful, serious errors are often written to file and forgotten. Fortunately, log4j supports email alerts through the SMTPAppender.
Find your log4j.xml file, add the following.
<appender name="EMAIL" class="org.apache.log4j.net.SMTPAppender"> <param name="BufferSize" value="512" /> <param name="SMTPHost" value="localhost" /> <param name="From" value="log4j@openmrs.org" /> <param name="To" value="errors@openmrs.org" /> <param name="Subject" value="OpenMRS Error" /> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%p - %C{1}.%M(%L) |%d{ISO8601}| %m%n" /> </layout> <filter class="org.apache.log4j.varia.LevelRangeFilter"> <param name="LevelMin" value="ERROR"/> <param name="LevelMax" value="FATAL"/> </filter> </appender>
You will also have to modify the root.
<root> <level value="WARN" /> <appender-ref ref="CONSOLE" /> <appender-ref ref="EMAIL" /> </root>
Be sure to change SMTPHost, From, To, and Subject to reflect your environment. The layout of the email can be controlled using ConversionPattern (although there seems to be a bug with now stack traces are printed). LevelMin and LevelMax changes which log messages are emailed. In the above example, WARN messages are passed to CONSOLE and EMAIL. Only log messages that are classified between ERROR and FATAL (inclusive) are emailed.
For more information on log4j and email, check out
Reporting Application Errors by Email and AuthSMTPAppender.
Enabling secure (SSL) data transfer in Tomcat
SSL, or Secure Socket Layer, is a technology which allows web browsers and web servers to communicate over a secured connection. This means that the data being sent is encrypted by one side, transmitted, then decrypted by the other side before processing. This is a two-way process, meaning that both the server AND the browser encrypt all traffic before sending out data. On the web, unsecure data arise often passed over HTTP (port 80) while secure data is passed over HTTPS (port 443).
Because the security of medical information is essential, it is important for all OpenMRS data to be passed over HTTPS. Of course, in addition to securing internet communication, implementers must also ensure the local network is secure, security patches are applied, strong passwords are used, and the OpenMRS machine physically secured.
To enable SSL in Tomcat, follow the instructions at Apache’s Tomcat SSL Configuration page. InstallCert.java can help you install your self-created certificate on your client machines. GoDaddy/Verisign certificates work out of the box.
Once SSL is running, implementers may wish to redirect all unsecure requests to secure channels. To do this, a few configuration files in Tomcat must be changed.
First, make sure all Tomcat applications require HTTPS to operate. Find the $TOMCAT_HOME/conf/web.xml file. At the bottom of the file, immediately before make sure the following is uncommented (or added).
<!-- == Force all applications to use SSL == --> <security-constraint> <web-resource-collection> <web-resource-name>All applications</web-resource-name> <url-pattern>/*</url-pattern> </web-resource-collection> <user-data-constraint> <description>Redirect all requests to HTTPS</description> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> </web-app>
Then, ensure all requests to the HTTP port are forwarded to the HTTPS port. Find the $TOMCAT_HOME/conf/server.xml file. In the HTTP connector section, ensure the redirectPort is pointing towards your HTTPS connector. Finally, ensure the HTTPS connector has SSL enabled. An example minimum configuration is below.
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" />
Lost tomcat password
You can retrieve the tomcat admin password from C:Program FilesApache Software FoundationTomcat 6.0conftomcat-users.xml
«SEVERE: Error listenerStart» in tomcat logfile
You deploy the openmrs.war, but it fails to start, with the only clue two simple messages in the tomcat logfile…
SEVERE: Error listenerStart SEVERE: Context [/openmrs] startup failed due to previous errors
Well, it is most likely a problem with the application_data_directory. Investigate these questions:
- have you created the runtime properties file?
- have you specified application_data_directory in the runtime properties?
- does the directory exist?
- does tomcat have read/write permissions to the directory?
If that all checks out, then you probably have security violations with your tomcat configuration.
Error Loading Persisted Sessions
Tomcat tries to restore the exact memory state after each restart. OpenMRS does not depend on this, so you can ignore the annoying warnings printed to the logs that look like this:
SEVERE: IOException while loading persisted sessions: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException:
To turn off session persistence, Go to tomcathome/conf/server.xml and uncomment the «<Manager pathname=»» />» line
ElException in parsing some JSP pages (Updated tomcat 7 and may be other servlet containers that conform to EL Spec 2.2)
In the latest version of EL Spec v2.2, Java identifiers and reserved keywords cannot be used in EL expression. In OpenMRS we have some parts in JSP that do not conform to this new spec. To workaround this problem, you can add «-Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true» to CATALINA_OPTS environment variable — See discussion: Identifier creation doesn’t work in trunk
Troubleshooting Installation
For more troubleshooting tips see Troubleshooting Installation.