Child pages
  • How to monitor your application server with op5 Monitor and JMX4Perl
Skip to end of metadata
Go to start of metadata

In this how-to we will cover how to monitor your application server via JMX with op5 Monitor with the plugin check_jmx4perl and the agent jolokia. We will also have a glance at monitoring application  servers trough a JMX-proxy for agentless monitoring where agents are not possible to use.

Version disclamer

Icon

This article is written for op5 Appliance System 3.5.2 and op5 Monitor 5.4.3, and should work for versions up to 5.7.3 of op5 Monitor. This article does not apply to op5 Appliance System 6.0 and op5 Monitor 6.0 and later.

 

Prerequisites

 In this example the community version of Jboss 6 is used as the server of choice.  Make sure that you have Perl, gcc and make installed, and the necessary ports opened in your firewall. Basic UNIX/Linux knowledge is needed.

Additional Information

JMX4Perl: http://labs.consol.de/lang/de/jmx4perl/

Agent information: http://www.jolokia.org/reference/html/agents.html

Proxy information: http://www.jolokia.org/reference/html/proxy.html

Security configuration: http://www.jolokia.org/reference/html/agents.html#agent-war-security

Disclaimer

This how-to  is not officially supported by op5. It is just a glimpse on how to preform monitoring of application servers with op5 Monitor. And I am not a sysadmin working with application servers all day long, so there may be some glitches here and there. This is not a security document, so it is your responsibility that your system is secure. Please refer to the developers manual for more information about securing your installation.

Installation

 

Icon

If you run a distributed and/or Load Balanced setup, you will need to install the plugin on all systems that are running jmx-checks.

 When using a agentless approach these steps only applies for the server that is acting as proxy.

 

  • Download JMX4Perl

You can fetch the latest version from: http://search.cpan.org/dist/jmx4perl/

  • Extract the file:
 

 

This command will most likely complain about the following dependencies: CBuilder, ParseXS and Module-Build.

Install them via rpm as described below. We have collected these RPM’s on our download site for your convenience.

  • Download and unpack

Download and unpack
  • Installation

Installation

Now it’s time to choose which components to install from the jmx4perl-package.

The required components are jmx4perl, and check_jmx4perl, although we recommend to install “j4psh” which is a JMX-shell that you can connect to JBoss and browse around the MBeans.

  • Re-run Build.pl

Build.PL

When this is done there may still be some missing dependencies and the Build-script will try to install these by itself.

Run the following commands:

Build

If all goes well you should have JMX4Perl installed on your system.

Deployment of webapp

Next we will need to deploy the webapp “jolokia” in our Application server. In this example I used Jboss6.

Icon

 You can also install Jolokia via the JMX4Perl installer by selecting “y” for the jolokia option.

 the latest version of Jolokia can be found at: http://www.jolokia.org/download.html

jolokia

Now the web-app is deployed, and JBoss will extract it by it self.

To test if the web-app works, browse to: http://:/jolokia/version

or run the following command from your op5 Monitor server:

Now jmx4perl should return some information about JBoss and jolokia.

You can also connect with j4psh-shell to the server and browse around the MBeans:

  • Copying files

You need to copy the installed check_jmx4perl and it’s config-files to a directory that op5 monitor can browse:

  • Test check_jmx4perl

Now we can try to do a check in command-line using this example:

That should give a output similar to this:

Config files

First copy all the default configs to your created “jmx4perl” folder in /opt/plugins/custom/jmx4perl/

  • One config to rule them all

In this example we will create one config that includes all the others to make changes more simple to maintain and to shorten the check commands.

Create this config called jmx4perl.cfg:

jmx4perl.cfg

 

These files contains the variables set in “check_command_args” for the check commands that we will create.

 

I will go trough: memory.cfg, thread.cfg and some possibilities to customize your jboss.cfg.

Please have a look in the config files for further explanation of the options.

  • memory.cfg
Command variableExplanation
memory_heapRelative Heap Memory used by the application.
memory_non_heapRelative non-heap memory.
memory_pool_baseMemory pool checks, specific to a Sun/Oracle JVM.
  • threads.cfg
Command nameExplanation
thread_incCheck for a thread increase per minute
thread_countCheck for monitoring the total (absolute) count of threads
thread_deadlockFind deadlocked Threads

Check commands

  • Using agents

First we create check commands to be used with agents. You can read the pros and cons for the different usages of jolokia at:

http://www.jolokia.org/reference/html/architecture.html#agent-mode and http://www.jolokia.org/reference/html/architecture.html#proxy-mode

When using config-files we minimize the number of check commands to a bare minimum, and we will only create two of them.

One for static checks and one for an incremental check that shows how much a value has changed in a given time period.

  • Go to: Configure -> Commands

Create a new check command with the following values:

command_namecommand_line
check_jmx4perl_config$USER1$/custom/jmx4perl/check_jmx4perl -u http://$HOSTADDRESS$:$ARG1$/jolokia –config $USER1$/custom/jmx4perl/config/jmx4perl.cfg –check $ARG2$ –warning $ARG3$ –critical $ARG4$
  • Click “Apply Changes”

And the same procedure for the incremental check:

command_namecommand_line
check_jmx4perl_config_delta$USER1$/custom/jmx4perl/check_jmx4perl -u http://$HOSTADDRESS$:$ARG1$/jolokia –config $USER1$/custom/jmx4perl/config/jmx4perl.cfg –check $ARG2$ –delta $ARG3$ –warning $ARG4$

And finally, save the changes.

 

First we add host to op5 Monitor with the ip-address of our application server.

  • Go to: Configure -> New Host

Fill in the configuration information: host name, alias, address etc.

  • Click “Scan host for services” -> Click “Continue to step 3″
  • Next, click “Services for”

Select the check command that you just created, and fill in the following arguments:

  • Go to the host -> Services -> Add new service

The separators of check_command_args are: "!"

service_descriptioncheck_commandcheck_command_args
JMX4Perl – HeapMemoryUsagecheck_jmx4perl_config8080!memory_heap!70!90
  • Click “Apply Changes”

And Save applied changes

Repeat these steps for the incremental check

service_descriptioncheck_commandcheck_command_args
JMX4Perl – HeapMemoryUsageDeltacheck_jmx4perl_config_delta8080!memory_heap!300!25

This command checks how much memory usage has changed the last 5 minutes (300 seconds), with a warning threashold of 25%.

Proxy mode

When using a agentless approach we need a app-server with the jolokia webapp installed, and configured to accept connections via RMI.

This was default on my installation of Jboss6 and i won’t expand it any further.

RMI test:

 

 

If this works you should get something like this in return:

Now we will create a check command using config-files as explained earlier.

  • Check command:
command_namecommand_line
check_jmx4perl_config_proxy$USER1$/custom/jmx4perl/check_jmx4perl -u http://$ARG1$:$ARG2$/jolokia –target service:jmx:rmi:///jndi/rmi://$HOSTADDRESS$:$ARG3$/jmxrmi –config $USER1$/custom/jmx4perl/config/jmx4perl.cfg –check $ARG4$ –warning $ARG5$ –critical $ARG6$

First we add host to op5 Monitor with the ip-address of our application server.

  • Go to: Configure -> New Host

Fill in the configuration information: host name, alias, address etc. Click “Scan host for services” -> Click “Continue to step 3″

  • Next, click “Services for”

Select the check command that you just created, and fill in the following params:

service_descriptioncheck_commandcheck_command_args
Heap Memory Via Proxycheck_jmx4perl_config_proxy!8080!1090! memory_heap!80!90

Argument explanation:

First we define which host that runs the JMX-proxy, port, rmiport on the actual server we want to monitor, what checks we want to run on this server, and finally the thresholds.

These arguments is also listed in the config-files for jmx4perl just as the previous example with agents.

What you want to monitor in your environment is your call, this is just a glance at the possibilities using jmx4perl.

Jboss Labs (Optional)

This part shows some custom options that are not covered with the default configuration and is entirely optional. I just want to show examples for custom checks that you can create and include in your configs according to your needs. I found these values using the “j4psh” shell and browsing around the MBeans and editing jboss.cfg according to the MBean names and creating check commands for these:

  • jboss.cfg
Command NameExplanation
jboss_cpool_availableAvailable connections in a connection pool for a data source
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = AvailableConnectionCount
Name = Avalible Connections
Command NameExplanation
jboss_cpool_used_maxThe reverse: Max. number of connections ever in use
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = MaxConnectionsInUseCount
Name = Max. connections in use
Command NameExplanation
jboss_cpool_usedConnections currently in use
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = InUseConnectionCount
Name = Connections in use
Command NameExplanation
jboss_cpool_creation_rateRate how often connections are created per minute
Use = count_per_minute("connections")
MBean = jboss.jca:name=JmsXA,service=ManagedConnectionPool
Attribute = ConnectionCreatedCount
Name = Connection creation rate

You can of course create additional config files with your specific needs and include them in “jmx4perl.cfg”