MacOs Snow Leopard and the disappearing pinyin candidate window

This post doesn't have very much to do with programming, but it did take me forever to figure this out, so maybe this will help out others who have had this problem.

I occasionally need to use the Pinyin-simplified on MacOsX Snow Leopard to type Chinese. I'm still fairly new to using Macs... so I've found myself accidentally pressing combinations of keys that I am used to on the Windows, that unfortunately map to entirely different things on the Mac. (I know, bad fingers ... bad). Most recently, I was in the middle of writing up a presentation for a class in Chinese using the Pinyin - Simplified input. It was late, I was tired, and somehow I pressed some combination of keys, and the little window that shows up with the different options of Chinese characters based on the pinyin typed disappeared! I have since learned that this window is called the candidate window.

I googled and googled, and came across suggestions like:

1. In the preferences for the input, switch the orientation of the input box from horizontal to vertical and then back again.

This didn't work for me.  At first, as I typed the pinyin, my letters came out like regular Roman letters.  And then after some futzing with the different preferences options, I ended up with getting letters like this:

As it finally turns out, it seems I somehow I pressed the Caps Lock while in the Pinyin mode, which according to this link means I switched on the single-byte Roman character.  Everything went back to normal after I reset my input preferences to the default.  

So now I know how to be able to type regular Roman characters (i.e. English) while in Pinyin mode, which is to press the Caps Lock.  As for the funky looking Roman characters I got, that happened when in my futzing with the preferences Options I selected "Use double-byte Roman characters":

I have to say, I do like how Windows manages this particular input style.  In the tool bar, Windows will tell you whether or not you are typing in English or Chinese when using the Pinyin input.  There doesn't seem to be any indication on the Mac to show you this, unless you know that's what the Caps Lock is about.  

Exporting RedNotebook notes to Evernote using Applescript

I've recently switched my work environment from Ubuntu to Mac OS. While I was using Ubuntu, I used RedNotebook for taking notes about the various things I was working on, code snippets, to do lists, etc. I am now using Evernote, which has some nice features and better userability (in my opinion). Sadly, there is no easy way to import my notes from RedNotebook.

What I've done to accomplish this is to use Mac's Applescript and I pieced together a hacky, but workable, way to do just this.  It's not entirely pretty, and I'm sure it could be done better, but it does work with one caveat and some instructions (it's unfortunately not an entirely automatic procedure).  So here it is for anyone else who wants an already working way to do the same thing.

First off, the caveat:  The very first note imported into Evernote ends up being an HTML attachment, instead of the text.  And the formatting gets a little bit screwy in the imported files, but in my opinion, that's acceptable since what I want is to have this information for future reference.

1. Create a new directory, this will be your input directory, and export the notebooks from RedNotebook you want imported as HTML.  This will create one large XML file for all the notes for each notebook you want imported.
2. Create a new directory, this will be your output directory, and save this file to it.  This file is the top portion of the xml file that will make up each of the individual notes in the evernote notes.
3. Download the applescript or copy and paste from below to your computer.
4. Run the script, you will be prompted for the following:
1. The header file, which is the template file you downloaded to your output directory.
2. The output directory you created.
3. The input directory containing the html files form RedNotebook
5. Once the script is done, it is necessary to manually open the first exported file in your export directory and copy and paste this into the first note imported in Evernote, as mentioned earlier.

What the script does:

1. Creates a new notebook in Evernote with the same name as the HTML file it is importing.  If you have multiple HTML files, it will create a new notebook for each of these files.
2. For each HTML file, it will create a new note in the new notebook.  The new note will be named the date the note was created and the creation date of the note will be modified to reflect this, so this metadata is not lost.

Here is the applescript code:

Setting up Apache CXF with SSL for client and server

I recently changed the JAXWS implementation from the reference implementation (RI) to Apache CXF. One of the motivating factors is that we need to be able to communicate via SSL, and to do this with the RI, the JVM system properties need to be changed.  This can, and has, caused problems when the webapp shares Tomcat with other webapps.

In order to this, I needed to set up Apache CXF.  Specifically, our webapp acts as both a client and server.


1. Dependencies
The first thing that needs to be done is to include the dependencies.  We use maven.

(Defined in our main pom's properties section)

    2.4.0

(In the pom for any package that performs client side JAXWS actions)

  
    
        org.apache.cxf
        cxf-rt-frontend-jaxws
        ${cxf-version}
    
    
        org.apache.cxf
        cxf-rt-transports-http
        ${cxf-version}
    

(In the pom for the package that performs the server side JAXWS actions)*

    
        org.apache.cxf
        cxf-rt-transports-http-jetty
        ${cxf-version}  

* I'm not 100% sure if this particular dependency is needed. It doesn't seem to be causing a problem with my build as of now. There is conflicting documentation about this. According to an out-dated how to set up CXF with Maven:

Jetty is needed if you're using the CXFServlet

However, according their first jaxws factory bean sample pom:

 Jetty is needed if you're using the CXFServlet

I'll update this post when I find a definitive answer.

2. CXF Server Configuration
The next thing that needs to be done is to register the cxf service.  This is done in the WEB-INF folder in the package that creates the web service .war file.  This folder should contain two files:

• web.xml
• Registers the services
• cxf.xml
• Configuration file for cxf that tells cxf how to resolve the endpoint

(web.xml)

    webservices-project
    
        CXFServlet
        org.apache.cxf.transport.servlet.CXFServlet
        1
    
    
        CXFServlet
        /*
    
    
        contextConfigLocation
        /WEB-INF/cxf.xml
    
    
        org.springframework.web.context.ContextLoaderListener
    

Two things to note here:

• On line 6, there is a load-on-startup field. I'll be honest, I don't know what this does. According to the wise stackoverflow:
  load-on-startup can specify an (optional) integer value. If the value is 0 or greater, it indicates an order for servlets to be loaded, servlets with higher numbers get loaded after servlets with lower numbers.
• On line 14, this is the name and location of the configuration file for cxf. If your cxf config file is named differently, say cxf-mywebapp.xml, indicate it here.

 (cxf.xml)

One thing to note here is:

• Line 11: Specify here the id/name of the web-service, the fully annotated for JaxWS implementation of the web-service and the address.

3. Initialize conduit for SSL communication 
CXF has various methods for controlling settings for the communications. In order to set up the proxy for SSL communications, settings are made to the TlsClientParameters.  I have the following method to set the TlsClientParameters for this (adapted from this post):

public final class Utils
{
    private static final TLSClientParameters tlsParams = new TLSClientParameters();
    private static volatile boolean isTlsSetForSSL = false;
    private static final Lock setTlsParamsLock = new ReentrantLock();
    
    // ----
    // Some other methods and things
    // ---
    static final TLSClientParameters getTlsParams()
    {
        return tlsParams;
    }
    
    static final void initializeConduitForSSL() throws Exception
    {
        // if the tls params have not been set up for SSL
        if(!isTlsSetForSSL)
        {
            // get the lock
            setTlsParamsLock.lock();

            // intentionally checking twice to try to minimize overhead
            // of acquiring lock
            if(!isTlsSetForSSL)
            {
                try
                {
                    isTlsSetForSSL = true;

                    // set up the keystore and password
                    KeyStore keyStore = KeyStore.getInstance("JKS");
                    // -- provide your password
                    String trustpass = "password"; 
                    // -- provide your truststore
                    File truststore = new File("truststore.jks");
                    keyStore.load(new FileInputStream(truststore), trustpass.toCharArray());

                    // should JSEE omit checking if the host name specified in the
                    // URL matches that of the Common Name (CN) on the server's 
                    // certificate.
                    tlsParams.setDisableCNCheck(true);

                    // set the SSL protocol
                    tlsParams.setSecureSocketProtocol("SSL");

                    // set the trust store 
                    // (decides whether credentials presented by a peer should be accepted)
                    TrustManagerFactory trustFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
                    trustFactory.init(keyStore);
                    TrustManager[] tm = trustFactory.getTrustManagers();
                    tlsParams.setTrustManagers(tm);

                    // set our key store 
                    // (used to authenticate the local SSLSocket to its peer)
                    KeyManagerFactory keyFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
                    keyFactory.init(keyStore, trustpass.toCharArray());
                    KeyManager[] km = keyFactory.getKeyManagers();
                    tlsParams.setKeyManagers(km);

                    // set all the needed include & exclude cipher filters
                    FiltersType filter = new FiltersType();
                    //filter.getInclude().add(".*_WITH_3DES_.*"); // XXX needed?
                    filter.getInclude().add(".*_EXPORT_.*");
                    filter.getInclude().add(".*_EXPORT1024_.*");
                    filter.getInclude().add(".*_WITH_DES_.*");
                    filter.getInclude().add(".*_WITH_NULL_.*");
                    filter.getExclude().add(".*_DH_anon_.*");
                    tlsParams.setCipherSuitesFilter(filter);
                }
                catch(final Exception e)
                {
                    throw new Exception("Failed to initialize conduit!", e);
                }
                finally
                {
                    setTlsParamsLock.unlock();
                }
            }
        }
    }
}

Things to note:

• According to CXF, proxies can be thread safe, however settings on the conduit are NOT.  The settings on the conduit are per instance.  For this reason, I have done the following:
• There is a volatile variable isTlsSetForSSL that is checked so the settings are only made once.
• The settings are done only after acquiring a lock
• The TlsClientParameters allow for setting two different stores, one for the keystore and one for the truststore.  In this example, both of these happen to be the same file.
• Line 63 is commented out.  I'm not sure if this is needed or not.  I was having some trouble getting this to work, and thought it was related to the filters that were included.  I came across a post that mentioned:
  
  2009-10-30 19:37:37:745 INFO [pool-2-thread-4] [PhaseInterceptorChain] - Interceptor has thrown exception, unwinding now org.apache.cxf.interceptor.Fault: Could not send Message. .... Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
  
  Later I found out that the new server expected to communicate over a 3DES SSL cipher suite because the new Apache configuration was set to use strong encryption (see encryption http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html).
  
  In my case, it turned out that there was one place in the unit test that did not call this initializing method before making a proxy.

4. Making a proxy to use
Prior to this change, we were using the reference implementation (Sun). In order to have SSL work properly in that implementation, it was necessary to make changes to the JVM wide system properties, which was un-desirable. When making the change to CXF, we were trying to limit the use of implementation specific code. 

CXF has an example that shows how to configure the conduit for a client, and indirectly shows how to get a proxy to work with:

import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.transport.http.HTTPConduit;
import org.apache.cxf.transports.http.configuration.HTTPClientPolicy;
...

URL wsdl = getClass().getResource("wsdl/greeting.wsdl");
SOAPService service = new SOAPService(wsdl, serviceName);
Greeter greeter = service.getPort(portName, Greeter.class);

Client client = ClientProxy.getClient(greeter);
HTTPConduit http = (HTTPConduit) client.getConduit();

Since our prior implementation used javax.xml.ws.Service to get the service, I decided to reuse that instead for getting the service:

import javax.xml.ws.Service;
import javax.xml.namespace.QName;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.transport.http.HTTPConduit;
...

final QName qServiceName = new QName(myNamespace, serviceName);
final QName qPortName = new QName(myNamespace, portName);
final Service service = Service.create(myUrl, qServiceName);
final Greeter greeter = service.getPort(qPortName, Greeter.class);

Client client = ClientProxy.getClient(greeter);
HTTPConduit http = (HTTPConduit) client.getConduit();
// !! THIS DOES NOT WORK FOR SSL !!
http.setTlsClientParameters(getTlsParams());

After removing all the methods that set the system wide JVM properties and trying this out... it turns out that it DOES NOT WORK. The following is the stacktrace on the server's log of the failure:

Caused by: javax.xml.ws.WebServiceException: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
 at org.apache.cxf.jaxws.ServiceImpl.(ServiceImpl.java:149)
 at org.apache.cxf.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:90)
 at javax.xml.ws.Service.(Service.java:56)
 at javax.xml.ws.Service.create(Service.java:680)
... more
Caused by: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
 at org.apache.cxf.wsdl11.WSDLServiceFactory.(WSDLServiceFactory.java:93)
 at org.apache.cxf.jaxws.ServiceImpl.initializePorts(ServiceImpl.java:203)
 at org.apache.cxf.jaxws.ServiceImpl.(ServiceImpl.java:147)
 ... 15 more
Caused by: javax.wsdl.WSDLException: WSDLException: faultCode=PARSER_ERROR: Problem parsing 'https://localhost:9081/node-server/node?wsdl'.: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.ibm.wsdl.xml.WSDLReaderImpl.getDocument(Unknown Source)
 at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
 at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
 at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:239)
 at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:186)
 at org.apache.cxf.wsdl11.WSDLServiceFactory.(WSDLServiceFactory.java:91)
 ... 17 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
 at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
 at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
...
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:323)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:217)
 at sun.security.validator.Validator.validate(Validator.java:218)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1185)
 ... 41 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
 ... 47 more

As it turns out, it is failing at line 10 in the above, the javax.xml.ws.Service call does something that tries to connect before getting to the conduit set up. This call relies on the JVM system wide properties, and because I had removed this bit of code, it was failing because it could not longer find the keystore information on the properties.

When I changed the code to use the implementation specific way of setting up a proxy, it worked:

final JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();

factory.setServiceClass(Greeter.class);
factory.setAddress(myUrl.toString());
final Greeter greeter = (Greeter) factory.create();

final Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
// make sure to initialize tlsParams prior to this call somewhere
http.setTlsClientParameters(getTlsParams());

One final thing to note. In our unit test, we originally had:

private static final String endpointA = protocol + "://localhost:9080/" + serverContext + "/node";

This resulted in the following error when trying to run the unit test after switching to CXF:

javax.xml.ws.WebServiceException: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
 at org.apache.cxf.jaxws.ServiceImpl.(ServiceImpl.java:149)
 at org.apache.cxf.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:90)
 at javax.xml.ws.Service.(Service.java:56)
 at javax.xml.ws.Service.create(Service.java:680)
... more
Caused by: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
 at org.apache.cxf.wsdl11.WSDLServiceFactory.(WSDLServiceFactory.java:93)
 at org.apache.cxf.jaxws.ServiceImpl.initializePorts(ServiceImpl.java:203)
 at org.apache.cxf.jaxws.ServiceImpl.(ServiceImpl.java:147)
 ... 32 more
Caused by: javax.wsdl.WSDLException: WSDLException: faultCode=PARSER_ERROR: Problem parsing 'https://localhost:9080/server/node'.: java.io.IOException: Server returned HTTP response code: 500 for URL: https://localhost:9080/server/node
 at com.ibm.wsdl.xml.WSDLReaderImpl.getDocument(Unknown Source)
 at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
 at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
 at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:239)
 at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:186)
 at org.apache.cxf.wsdl11.WSDLServiceFactory.(WSDLServiceFactory.java:91)
 ... 34 more
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://localhost:9080/server/node
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
 at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)
 at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
 at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
 at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
 at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
 ... 40 more

Chaniging the endpoint to add a ?wsdl to the end of the string fixed this problem:

private static final String endpointA = protocol + "://localhost:9080/" + serverContext + "/node?wsdl";

And that's it.  Hopefully this helps.

Yourkit & Tomcat webapp on Fedora

Yourkit is a great tool for profiling Java applications.  I'm using it to do some profiling on our webapp that is deployed in a Tomcat on a remote server (virtual machine on Rackspace).  I've found that Yourkit is really simple and intuitive to use and set up, for the most part.

The only issue I've had was getting it set up on the remote server to hook my local GUI to.  Theoretically, it is a very simple process.  Download yourkit to the remote server, start up tomcat, and type the following at the command prompt:

/bin/yjp.sh -integrate

I've downloaded & installed the linux version into my /opt/ folder on my remote server.  My remote server is Fedora 14 64bit.  When I run the above command, I get the following prompts:

And here is where I run into a problem. Where is the tomcat startup.sh?? I do a search on the remote machine for the file and come up empty handed. I sift around online and come up empty handed.  Finally after trial and error, I find that the file that is used on Fedora for starting up tomcat is called:

/usr/sbin/tomcat6

I would have thought this would be the end of it.  Unfortunately, it looks the -integrate command munges the startup.sh to add an additional JAVA_OPTS to create the hooks for Yourkit.  And of course /usr/sbin/tomcat6 is not formatted how Yourkit is expecting the startup.sh to be formatted.  So no go.

The next option is to manually add the JAVA_OPTS to the file.  What I've done is add the following:

-agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001

after the {JAVA_OPTS} in the start block of the /usr/sbin/tomcat6 script (around line 30):

if [ "$1" = "start" ]; then
  ${JAVACMD} $JAVA_OPTS -agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001 $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start \
    >> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
    if [ ! -z "$CATALINA_PID" ]; then
      echo $! > $CATALINA_PID
    fi

I have added an additional startup option to this, specifying the port to use.  Pay close attention when inserting this option and be sure that is a space between $JAVA_OPTS and [agentpath] as well as a space between [agentpath] and $CATALINA_OPTS.  Because I will not always want the agent attached (overhead and performance reasons), I created two copies of the tomcat6, a tomcat6_yourkit & tomcat6_orig (original tomcat6 file) that I copy to tomcat6 as needed to swap the configurations.

Finally, the last thing that needs to be done is to open up the port in the iptables file on the remote server to allow the connection to be made.

Summary:

1. Download the correct distribution for the remote server's platform to /opt/.
   
   (In my case I need the Linux distribution)

   [root@bakeYourkit opt]# wget http://www.yourkit.com/download/yjp-9.5.6-linux.tar.bz2

2. Extract the file.
   
   (In my case it is a tar.bz2, which I always forget how to do).
3. Make a copy of /usr/sbin/tomcat6 as /usr/sbin/tomcat6_orig
   

   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_orig

4. Edit /usr/sbin/tomcat6, save, and make a copy as /usr/sbin/tomcat6_yourkit
   

   [root@bakeYourkit ~]# vim /usr/sbin/tomcat6
   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_yourkit
   

5. Edit the iptables to listen to the port specified in the startup option above (in my example here, port number 10001).  See this post for information about editing the iptables file.
6. Restart tomcat to pick up the options.
7. On your local machine, start up yourkit and connect to the remote server:
   

Ta-da!  That's all.  Now if you want to switch off the agent, copy the original /usr/sbin/tomcat6_orig over the /usr/sbin/tomcat6 file.  There is definitely a nicer way to this, possibly by adding some environment variable and setting it to.  But this works for me.

Perl script to "copy" a Rackspace image

I recently found myself putting together a perl script that performs an rsync of one server in the Rackspace cloud to another server on another account.  Why?  See my next post about restrictions found with Rackspace for details.  Long story short, I needed a way to copy an image from one account to another and Rackspace recommends using rsync.

Since I will be needing to do this at least once a month for my testing, I thought it best to write up a script using perl that does the rsync.  The first thing was to figure out to successfully do this rsync before putting things together in a script.  I followed the instructions and when I rebooted my new server with the files rsync'ed over, it failed to accept ssh anymore.  On a whim, I decided to go through the basic exclude file from the instructions to see if those directories/files really lived where the file says they would be.

Ah-ha! That was my problem. I'm not sure which distro was used to create the exclude file, but some of the recommended files/directories to be excluded were located in different places.

#Suggested by Rackspace
/etc/hostname

#Actual location on Fedora 14 64bit
/bin/hostname
 

#Suggested by Rackspace
/etc/modules 

#Actual location on Fedora 14 64bit
/etc/sysconfig/modules

What I did was to do a

find / -name [sometext]

Where [sometext] I started from the end of the path and worked my way in to find the location. It was definitely trial and error.  Here is what I have for my final exclude file for Fedora 14 64bit:

#exclude.txt
/proc
/sys
/tmp
/dev
/root/copyImage.log
/root/.ssh/
/var/lock
/etc/fstab
/etc/mtab
/etc/resolv.conf
/usr/share/dbus-1/interfaces
/etc/networks
/etc/sysconfig/network
/etc/sysconfig/network-scripts
/etc/sysconfig/iptables-config
/lib/modules/
/usr/share/selinux/devel/include/kernel
/bin/hostname
/usr/lib64/gettext/hostname
/etc/hosts
/etc/modprobe*
/etc/selinux/targeted/modules
/etc/selinux/targeted/modules/active/modules
/etc/sysconfig/modules
/etc/selinux/targeted/modules
/etc/selinux/targeted/modules/active/modules
/usr/lib64/gio/modules
/lib/udev/devices/net
/usr/lib/ruby/1.8/net
/etc/init/

So now that I have a working exclude file, the next thing was to code up a perl script to do this for me.  In order to automate this through the script I needed to be able to:

• Create an ssh tunnel from the "sending" server to the "receiving" server and be able to generate an ssh key on the "receiving" server.
• Add the "sending" server's public key to the "receiving" server's authorized_key file
• In my case, as part of the image, there is an authorized_key file with some public keys in it, which I want available to all servers generated with the image. I added the "sending" server's public key to this file and rsync'ed this file over first
• Rsync the "sending" server data to the "receiving" server.

Two of the largest hurdle is:

1. Open the ssh connection to the "receiving" server
2. Know when the "receiving" server is asking for the password and supply it

There are many many perl modules available that do just this in various ways. I didn't think to make a note of which ones I did try, unfortunately.  I ended up using Net::OpenSSH, it had the best documentation and did everything I needed.  I used this in conjunction with Expect module.


Neither of these modules are included with the Perl package that comes with Fedora 14 64 bit.  In trying to add these modules, I learned some more things.  For example, I learned about CPAN.  This is a little piece of software that manages installation and dependency resolution for perl modules.  Other benefits I found are:

• Access to modules that may not be in repos yet, i.e. Net::OpenSSH is not available via yum on Fedora 14
• Type in instmodsh at the command prompt and you can see a list of all installed modules and get information about the modules
• If module is installed via yum install , will not show up in instmodsh

Here is a link describing how to install CPAN.  After you follow the instructions at that link, you should also check to see if gcc is installed.  In my case, it was also not included in my default Fedora 14 64bit installation.  It is needed to compile some of the modules (in this case, one of the dependencies for Expect).  The error I got before I installed it is:

ERROR: cannot run the configured compiler 'gcc'
(see conf/compilerok.log). Suggestions:
1) The complier 'gcc' is not in your PATH. Add it
   to the PATH and try again. OR
2) The compiler isn't installed on your system. Install it. OR
3) You only have a different compiler installed (e.g. 'gcc').
   Either fix the compiler config in the perl Config.pm
   or install a perl that was built with the right compiler
   (you could build perl yourself with the available compiler).

Note: this is a system-administration issue, please ask your local
admin for help. Thank you.

Warning: No success on command[/usr/bin/perl Makefile.PL]
'YAML' not installed, will not store persistent state
  TODDR/IO-Tty-1.10.tar.gz
  /usr/bin/perl Makefile.PL -- NOT OK
Could not read metadata file. Falling back to other methods to determine prerequisites
Failed during this command:
 TODDR/IO-Tty-1.10.tar.gz                     : writemakefile NO '/usr/bin/perl Makefile.PL' returned status 6400

Finally, my last hurdle for putting together this script is trouble using the open function in perl.

#DOES *NOT* WORK
open(CAT_FILE,"cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n";

#DOES WORK
open(CAT_FILE,"| cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n";

Can you spot the difference?
So apparently that "|" is required ... oops...guess the documentation was right. *insert chagrin here*.

So without further ado, please find below my full script to do the rsync step of copying an image to another server on Rackspace:

Tomcat 6 & Fedora 14 frustrations

More woes trying to set up a server image.  Current pothole (as opposed to speed bump) is being brought to you by Tomcat 6 and Fedora 14. Having just gone through a whole lot of hoop-la getting java installed, I installed Tomcat with this simple command:

root@spin-1-16r tomcat6]# yum install tomcat6

First woe:
My woes began when I tried to start up tomcat how I started it up on Rhel:

[root@spin-1-16r tomcat6]# tomcat6 start
/usr/sbin/tomcat6: line 30: /logs/catalina.out: No such file or directory

What? How is an error happening when I haven't done anything but use the package installer yum to install tomcat? This time, I read my error carefully and decided to look in the file /usr/sbin/tomcat6, line 30 for the offending code which was:

    org.apache.catalina.startup.Bootstrap start >> ${CATALINA_BASE}/logs/catalina.out

Looking through the file, I don't see CATALINA_BASE defined. And then going back to the command prompt, I discover the environment variable CATALINA_BASE isn't defined anywhere:

[root@spin-1-16r webapps]# echo $CATALINA_BASE

More searching and I find that it is defined in /etc/tomcat6/tomcat6.conf and going back to the /usr/sbin/tomcat6 file, I see that the following is commented out:

# Get the tomcat config (use this for environment specific settings)
#if [ -z "${TOMCAT_CFG}" ]; then
#  TOMCAT_CFG="/etc/tomcat6/tomcat6.conf"
#fi
#
#if [ -r "$TOMCAT_CFG" ]; then
#  . $TOMCAT_CFG
#fi

Removing the comments from in front of lines 2-8 and saving solved this first issue.

*EDIT*
Avoid first woe sub-part one:
Remember earlier we went through some troubles to set up Sun's java and even set up some environment variables? We probably want to keep using Sun's java for tomcat. Open up the tomcat config file:

[frog@caffiend ~]# less ~tomcat/conf/tomcat6.conf

At the top of the file, after the comments about what file is for, you should see these lines:

# Where your java installation lives
JAVA_HOME="/usr/lib/jvm/java"

Now I'm not 100% that this will over-write the environment variable previously set, nor do I know whether or not that link will end up at the same location or not....so to be safe, I've commented it out on my server:

# Where your java installation lives
#JAVA_HOME="/usr/lib/jvm/java"

Second woe:
Now the next woe began.  For the purposes of work, the webapp war is dropped into the /usr/share/tomcat6 folder and a symbolic link is created in the webapp folder pointing to the war.  This is done because for our testing purposes, we swap various versions of the war, including snapshots, in and out and our deployment code to push out and configure multiple nodes have a hard link to the URL.  We could always just change the name of the war to match what we have in the URL in our code, but then we would lose the quick at a glance sanity check of which version we are running.  So that is what I did.  And I started up tomcat, but EGADS, the war would NOT inflate and deploy.  Why, oh why?  After much googling on keywords like symbolic link not deploying war tomcat6 fedora14, so on and so forth...and bringing in a second pair of eyes...it hits me like a ton of bricks:

[root@spin-1-16r tomcat6]# ls -l webapps/
total 0
lrwxrwxrwx 1 root root     39 Feb 15 22:10 node-server -> /usr/share/tomcat6/node-server-1.16.war

Do you see it now? It sure took me a while...my sym link did NOT end in .war. And that...is all it took. Changing the sym link and restarting tomcat my directory now looks like:

[root@spin-1-16r webapps]# ls -l
total 8
drwxr-xr-x 2 root root 4096 Feb 15 23:52 logs
drwxr-xr-x 4 root root 4096 Feb 15 23:52 node-server
lrwxrwxrwx 1 root root   39 Feb 15 23:52 node-server.war -> /usr/share/tomcat6/node-server-1.16.war

And thus ends my woes. It looks so simple and duh all summed up here, but man, was it hard to see for me.

Ubuntu (v10.04) update frustrations

*EDIT*
I wrote this a few months back and I'm not quite sure why I never published it...might have been one of those days...hopefully it's correct, if it's not, please let me know.
 

Finally decided to do the updates that my Ubuntu has been asking me about for a week or so now.  And of course, my Eclipse breaks because Ubuntu keeps trying to change my java to OpenJDK, which does not play together happily with Eclipse (v3.3). I'm sure others must also have this problem, so here is my step by step troubleshooting:

1. Did  the update change the environment variables?
• Check the bashrc file:
  

  frog@caffiend:~$ vi ~/.bashrc
  

  
  Where vi is your console editor. You can use any editor you like. The file should contain the following, most likely at the end:
  

  export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20/
  

2. Did the update change the symbolic link?
• First find your Sun Java directory. I think it usually gets installed in /usr/lib/jvm. One way is to do a search from the FileSystem using the File Browser for "sun".  You're looking for a folder with a name like: java-x-sun-n.n.nn, where x is the version of java (nowadays, most likely 6...possibly 5) and n.n.nn is the release. 
• Now check to make sure the java folder is pointing correctly to the alternatives folder:
  

  frog@caffiend~:$ ls -l /usr/bin/java
  lrwxrwxrwx 1 root root 22 2010-11-02 12:26 /usr/bin/java -> /etc/alternatives/java
  

• Now check where the alternatives folder is pointing to:
  

  frog@caffiend~:$ ls -l /etc/alternatives/java
  lrwxrwxrwx 1 root root 36 2010-11-02 12:26 /etc/alternatives/java -> /usr/lib/jvm/java-6-sun/jre/bin/java
  

• If one or both of these symbolic links are not pointing to the correct location:
  

  frog@caffiend~:$ cd /usr/bin/
  frog@caffiend~:$ sudo ln -f -s /etc/alternatives/java java
  frog@caffiend~:$ cd /etc/alternatives/
  frog@caffiend~:$ sudo ln -f -s /usr/lib/jvm/java-6-sun/jre/bin/java java
  

3. Did the update change which Java is being used?

• frog@caffiend~:$ udo update-alternatives --config java
  [sudo] password for frog: 
  There are 2 choices for the alternative java (providing /usr/bin/java).
  
    Selection    Path                                      Priority   Status
  ------------------------------------------------------------
    0            /usr/lib/jvm/java-6-openjdk/jre/bin/java   1061      auto mode
    1            /usr/lib/jvm/java-6-openjdk/jre/bin/java   1061      manual mode
  * 2            /usr/lib/jvm/java-6-sun/jre/bin/java       63        manual mode
  
  Press enter to keep the current choice[*], or type selection number: 2
  

4. Confirm that the correct Java is being used:

• frog@caffiend~:$ which java
  /usr/bin/java
  frog@caffiend:/usr/bin$ java -version
  java version "1.6.0_22"
  Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
  Java HotSpot(TM) Server VM (build 17.1-b03, mixed mode)
  

Now try starting Eclipse.  There is a chance you may still get an error if you use Maven, subclipse.  A solution can be found here.

Errors Installing Java JDK on Linux.

Several hours that I will never get back have just been lost due to errors received trying to install Sun's Java (specifically the JDK) on a virtual machine.  I had previously been able to install the JDK without much of a hiccup last fall on the virtual machine at this particular vendor, and was quite puzzled as to why this was no longer working.  Googling the error and various keywords didn't point out any obvious issues, everything seemed to say that Fedora 14 and Sun Java played well together, with the exception of having to configure the alternatives to point to the correct java (which I knew).

[root@spin-1 opt]# sh ./jdk-6u22-linux-i586.bin 
Unpacking...
Checksumming...
Extracting...
./jdk-6u22-linux-i586.bin: ./install.sfx.20270: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

Fine, let's try the self extracting rpm version:

[root@spin-1 opt]# sh ./jdk-6u22-linux-i586-rpm.bin 
Unpacking...
Checksumming...
Extracting...
./jdk-6u22-linux-i586-rpm.bin: ./install.sfx.20566: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
Installing JavaDB
error: open of sun-javadb-common-10.5.3-0.2.i386.rpm failed: No such file or directory
error: open of sun-javadb-core-10.5.3-0.2.i386.rpm failed: No such file or directory
error: open of sun-javadb-client-10.5.3-0.2.i386.rpm failed: No such file or directory
error: open of sun-javadb-demo-10.5.3-0.2.i386.rpm failed: No such file or directory
error: open of sun-javadb-docs-10.5.3-0.2.i386.rpm failed: No such file or directory
error: open of sun-javadb-javadoc-10.5.3-0.2.i386.rpm failed: No such file or directory
 
Done.

After much googling, I kept coming back to this one forum post. In the middle of it, there is the following information from a poster:

"The library you are missing is comes from the 32bit version of the glibc. You can try installing the i386 version of glibc using yum. That would install /lib/ld-linux.so.2 for you."

Great, so let's try installing the glibc ... (some of you may be seeing where this is going, but sadly I did not yet).

[root@spin-1 opt]# yum install glibc*

Alright, so now that's installed, I tried installing java again...to no avail. Scrolling back up the console to at the yum output from installing the glibc* I see:

=============================================================================

 Package                       Arch     Version             Repository Size
=============================================================================

Installing:
 gcc                           x86_64   4.5.1-4.fc14        fedora     14 M
Installing for dependencies:
 cloog-ppl                     x86_64   0.15.7-2.fc14       fedora     93 k
 cpp                           x86_64   4.5.1-4.fc14        fedora     4.0 M
 glibc-devel                   x86_64   2.13-1              updates    968 k
 glibc-headers                 x86_64   2.13-1              updates    599 k
 kernel-headers                x86_64   2.6.35.11-83.fc14   updates    742 k
 libmpc                        x86_64   0.8.1-1.fc13        fedora     44 k
 mpfr                          x86_64   2.4.2-1.fc13        fedora     158 k
 ppl                           x86_64   0.10.2-10.fc12      fedora     1.1 M

Transaction Summary
=============================================================================

Now wait just a hot minute, why is it installing the x86_64 and not the i386 32bit ones....^Insert light bulb.

That's right folks, the virtual machine is 64 bit and I was trying to install a 32bit version of Java.  Some time between when I last tried this particular exercise out on the vendor and now, the vendor moved from 32bit OS's to 64bit OS's.  To verify:

[root@spin-1 opt]# uname -m
x86_64

Well would you look at that, my vm's architecture is x86_64. :: sigh :: downloading the 64 bit version of Java JDK resolved the issues.

For completeness of this post, some links to get started with Sun JDK Java and Tomcat 6 on Fedora 14, in order of things to do.

1. Configuring yum repositories on fedora
• On the remote machine, as root user, do the following:
  

  yum --nogpgcheck install http://rpm.livna.org/livna-release.rpm http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

2. On the remote machine, as root user, do the following:
   

   yum update 

   
   
3. Download the 64 bit version of Java JDK desired onto local machine and then scp the file to the /opt directory of the remote machine
   
   
4. On the remote machine, possibly need to be remote user, do the following:
   

   [root@spin-1 opt]# sh ./opt/jdk-6u22-linux-x64.bin 

   
   
*EDIT*
(realized I also did the following and it is not included in the link at step 6, not entirely sure of the necessity, but I did this anyways just to make sure and everything is working)
5. Add the following environment variables (not included as part of the directions in 6, may not be 100% necessary, but I did this and everything work) to the end of ~/.bashrc:
   

   export JAVA_HOME=/opt/jdk1.6.0_22
   export JRE_HOME=${JAVA_HOME}/jre
   export PATH=${JAVA_HOME}/bin:${PATH}
   export CLASSPATH=${JAVA_HOME}/jre/lib:${CLASSPATH}
   

   
   This is what my ~/.bashrc looks like:
   

   # .bashrc
   
   # User specific aliases and functions
   
   alias rm='rm -i'
   alias cp='cp -i'
   alias mv='mv -i'
   
   # Source global definitions
   if [ -f /etc/bashrc ]; then
           . /etc/bashrc
   fi
   
   export JAVA_HOME=/opt/jdk1.6.0_22
   export JRE_HOME=${JAVA_HOME}/jre
   export PATH=${JAVA_HOME}/bin:${PATH}
   export CLASSPATH=${JAVA_HOME}/jre/lib:${CLASSPATH}
   

   
   Save the file and from the command prompt, source the file:
   

   [root@newImage ~]# source ~/.bashrc 
   

   
   Now do an echo to make sure the environment variable exists and points to the correct location:
   

   [root@newImage ~]# echo $JAVA_HOME
   /opt/jdk1.6.0_22

   
   You should see the location your jdk is installed, my jdk is installed in /opt/.
   N.B.For the testing that we're doing here at work, we're always either logged in as root or tomcat is logged in as tomcat. I'm not familiar enough with environment variables and bash scripts to know where they need to be or what needs to be done to ensure the variable persists through all users.
   
   
6. Follow the directions found here for setting up the alternatives in Fedora to point to the correct java distro
   
   
7. Install tomcat:
   

   yum install tomcat6*.*

   
   

Remote debugging webapp in Tomcat (Rhel) through Eclipse (Ubuntu)

Background to my current issue:
There appears to be something odd going on the server side of our network code. Our server is a virtual machine hosted on GoGrid and the oddness is only occurring in release 1.14 and later of our network code.  The current trunk of our code is 1.16-SNAPSHOT and the network code on the vm is 1.15 (release).

I've spent the better of today getting right set of convocations together for my particular set of system set ups.  Specifically:
Locally
* Ubuntu 10.04 (Lucid Lynx)
* Eclipse 3.5

Server Side
* Rhel 5.4
* Tomcat 6

Hopefully the following represents an accurate list of steps to assist anyone else with setting up a remote debugging of a webapp running on a Tomcat elsewhere through Eclipse on your local machine.

• Setting up tomcat to start up in debug mode:
• Most of the websites I came across suggest adding a command after the script:
  
  

   [host]$ catalina.sh jpda start 

  (Mulesoft's blog)
 
• For my particular set up, I did not have a catalina.sh script file to manipulate.  Instead, Rhel's installation of Tomcat6 creates a conf folder and a conf file where the options are set.  In my case, the specific file name is tomcat6.conf and was found in ~tomcat6/conf/
• In this file, I added the following lines after setting where the installation lives (noting that this should be the first JAVA_OPTS set):
  

   # Java option to turn on debugger FIRST
  JAVA_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n" 

  
  NB: that there can not be any spaces around the initial "=". The above is NOT equivalent to:
  

  JAVA_OPTS = "-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n" 



• Now it's time to restart (bounce) the tomcat for the tomcat to pick up the changes in the JVM arguements:
  

  [host]$ /etc/init.d/tomcat6 restart 



•  Setting up the server to listen to the port that has been designated in the above step for the debugger (in this example, 8000, see the numbers after "address")
• Now the appropriate ports need to be opened on the iptables:
  

  [host]$ vim /etc/sysconfig/iptables 

  
  And add this line into the file and save it:
  

  -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT
  

  
  This line should go along with the other similar statements in the file. Once the file has been saved, the iptables need to be restarted for the changes to take affect:
  

  [host]$ /etc/init.d/iptables restart



• Since my server is hosted on GoGrid, I know that they have a firewall in place. The next step is to create an ssh tunnel to my server. For the sake of usability, I installed "Gnome SSH Tunnel Manager - a front-end to ssh tunneling" and added the information for my server and opened up the tunnel with the followinig for the properties:
• Name: GoGrid 252 [This can be anything that is easy for you to recognize]
• Login: me [This is the login you would use to login to the server]
• Host: ###.##.###.252 [This is the IP address of the server]
• Port: 22 [This is the port that you have designated on your server as your ssh port, default is 22]


• Finally, on my local machine in Eclipse, I had to do the following:
• I needed the specific version of the code base project opened that the server was running.  In this case, I had been working on 1.16 SNAPSHOT, which I needed to close and open up the 1.15 release that is running on the server.
• Click on any file in the project that contains the webapp code
• From the Eclipse menu, navigate: Run -> Debug Configurations
• In the left handle panel, navigate up to Remote Java Applications, right click over it and select New from the pop-up menu.
• In the right hand panel you should now see 3 tabs and a name field.  The name field is for your reference and will show up on the left hand panel for you to select later.  Under the Connect tab you should fill in the following:
• Project: [The project containing the source code for the webapp you are trying to debug in the remote Tomcat]
• Connection Type: Standard (Socket Attach)
• Host: ###.##.###.252 [This is the IP address of the server]
• Port: 8000 [This is the designated debugging port that was set up on the server]
• Click Apply and then Debug


• In the source code for the webapp, put a breakpoint somewhere that you know will be hit and then send a request to your tomcat.  You should see be able to now debug in Eclipse.

And that summarizes up what I did to get the debugging to work on my local machine for a remote webapp on Tomcat.

Here are some links to blogs which assisted me in getting to this point:

http://forums.alfresco.com/en/viewtopic.php?f=10&t=1914&start=15
http://wiki.apache.org/tomcat/FAQ/Developing?highlight=%28CategoryFAQ%29

Hope this helps :).