Yourkit & Tomcat webapp on Fedora

Yourkit is a great tool for profiling Java applications.  I'm using it to do some profiling on our webapp that is deployed in a Tomcat on a remote server (virtual machine on Rackspace).  I've found that Yourkit is really simple and intuitive to use and set up, for the most part.

The only issue I've had was getting it set up on the remote server to hook my local GUI to.  Theoretically, it is a very simple process.  Download yourkit to the remote server, start up tomcat, and type the following at the command prompt:

/bin/yjp.sh -integrate

I've downloaded & installed the linux version into my /opt/ folder on my remote server.  My remote server is Fedora 14 64bit.  When I run the above command, I get the following prompts:

And here is where I run into a problem. Where is the tomcat startup.sh?? I do a search on the remote machine for the file and come up empty handed. I sift around online and come up empty handed.  Finally after trial and error, I find that the file that is used on Fedora for starting up tomcat is called:

/usr/sbin/tomcat6

I would have thought this would be the end of it.  Unfortunately, it looks the -integrate command munges the startup.sh to add an additional JAVA_OPTS to create the hooks for Yourkit.  And of course /usr/sbin/tomcat6 is not formatted how Yourkit is expecting the startup.sh to be formatted.  So no go.

The next option is to manually add the JAVA_OPTS to the file.  What I've done is add the following:

-agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001

after the {JAVA_OPTS} in the start block of the /usr/sbin/tomcat6 script (around line 30):

if [ "$1" = "start" ]; then
  ${JAVACMD} $JAVA_OPTS -agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001 $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start \
    >> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
    if [ ! -z "$CATALINA_PID" ]; then
      echo $! > $CATALINA_PID
    fi

I have added an additional startup option to this, specifying the port to use.  Pay close attention when inserting this option and be sure that is a space between $JAVA_OPTS and [agentpath] as well as a space between [agentpath] and $CATALINA_OPTS.  Because I will not always want the agent attached (overhead and performance reasons), I created two copies of the tomcat6, a tomcat6_yourkit & tomcat6_orig (original tomcat6 file) that I copy to tomcat6 as needed to swap the configurations.

Finally, the last thing that needs to be done is to open up the port in the iptables file on the remote server to allow the connection to be made.

Summary:

1. Download the correct distribution for the remote server's platform to /opt/.
   
   (In my case I need the Linux distribution)

   [root@bakeYourkit opt]# wget http://www.yourkit.com/download/yjp-9.5.6-linux.tar.bz2

2. Extract the file.
   
   (In my case it is a tar.bz2, which I always forget how to do).
3. Make a copy of /usr/sbin/tomcat6 as /usr/sbin/tomcat6_orig
   

   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_orig

4. Edit /usr/sbin/tomcat6, save, and make a copy as /usr/sbin/tomcat6_yourkit
   

   [root@bakeYourkit ~]# vim /usr/sbin/tomcat6
   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_yourkit
   

5. Edit the iptables to listen to the port specified in the startup option above (in my example here, port number 10001).  See this post for information about editing the iptables file.
6. Restart tomcat to pick up the options.
7. On your local machine, start up yourkit and connect to the remote server:
   

Ta-da!  That's all.  Now if you want to switch off the agent, copy the original /usr/sbin/tomcat6_orig over the /usr/sbin/tomcat6 file.  There is definitely a nicer way to this, possibly by adding some environment variable and setting it to.  But this works for me.

Tomcat 6 & Fedora 14 frustrations

More woes trying to set up a server image.  Current pothole (as opposed to speed bump) is being brought to you by Tomcat 6 and Fedora 14. Having just gone through a whole lot of hoop-la getting java installed, I installed Tomcat with this simple command:

root@spin-1-16r tomcat6]# yum install tomcat6

First woe:
My woes began when I tried to start up tomcat how I started it up on Rhel:

[root@spin-1-16r tomcat6]# tomcat6 start
/usr/sbin/tomcat6: line 30: /logs/catalina.out: No such file or directory

What? How is an error happening when I haven't done anything but use the package installer yum to install tomcat? This time, I read my error carefully and decided to look in the file /usr/sbin/tomcat6, line 30 for the offending code which was:

    org.apache.catalina.startup.Bootstrap start >> ${CATALINA_BASE}/logs/catalina.out

Looking through the file, I don't see CATALINA_BASE defined. And then going back to the command prompt, I discover the environment variable CATALINA_BASE isn't defined anywhere:

[root@spin-1-16r webapps]# echo $CATALINA_BASE

More searching and I find that it is defined in /etc/tomcat6/tomcat6.conf and going back to the /usr/sbin/tomcat6 file, I see that the following is commented out:

# Get the tomcat config (use this for environment specific settings)
#if [ -z "${TOMCAT_CFG}" ]; then
#  TOMCAT_CFG="/etc/tomcat6/tomcat6.conf"
#fi
#
#if [ -r "$TOMCAT_CFG" ]; then
#  . $TOMCAT_CFG
#fi

Removing the comments from in front of lines 2-8 and saving solved this first issue.

*EDIT*
Avoid first woe sub-part one:
Remember earlier we went through some troubles to set up Sun's java and even set up some environment variables? We probably want to keep using Sun's java for tomcat. Open up the tomcat config file:

[frog@caffiend ~]# less ~tomcat/conf/tomcat6.conf

At the top of the file, after the comments about what file is for, you should see these lines:

# Where your java installation lives
JAVA_HOME="/usr/lib/jvm/java"

Now I'm not 100% that this will over-write the environment variable previously set, nor do I know whether or not that link will end up at the same location or not....so to be safe, I've commented it out on my server:

# Where your java installation lives
#JAVA_HOME="/usr/lib/jvm/java"

Second woe:
Now the next woe began.  For the purposes of work, the webapp war is dropped into the /usr/share/tomcat6 folder and a symbolic link is created in the webapp folder pointing to the war.  This is done because for our testing purposes, we swap various versions of the war, including snapshots, in and out and our deployment code to push out and configure multiple nodes have a hard link to the URL.  We could always just change the name of the war to match what we have in the URL in our code, but then we would lose the quick at a glance sanity check of which version we are running.  So that is what I did.  And I started up tomcat, but EGADS, the war would NOT inflate and deploy.  Why, oh why?  After much googling on keywords like symbolic link not deploying war tomcat6 fedora14, so on and so forth...and bringing in a second pair of eyes...it hits me like a ton of bricks:

[root@spin-1-16r tomcat6]# ls -l webapps/
total 0
lrwxrwxrwx 1 root root     39 Feb 15 22:10 node-server -> /usr/share/tomcat6/node-server-1.16.war

Do you see it now? It sure took me a while...my sym link did NOT end in .war. And that...is all it took. Changing the sym link and restarting tomcat my directory now looks like:

[root@spin-1-16r webapps]# ls -l
total 8
drwxr-xr-x 2 root root 4096 Feb 15 23:52 logs
drwxr-xr-x 4 root root 4096 Feb 15 23:52 node-server
lrwxrwxrwx 1 root root   39 Feb 15 23:52 node-server.war -> /usr/share/tomcat6/node-server-1.16.war

And thus ends my woes. It looks so simple and duh all summed up here, but man, was it hard to see for me.