Monday, November 8, 2010

Scala & Eclipse...Just keep cleaning...

Today's aggravation is most likely brought to you by the Scala IDE (v1.0.0.201009232358) plugin for Eclipse (v3.5.2.R35).  A simplified version of the offending code is:
  def nestedDoesntWork(num: Int) = {
    if (true)
      for (x <- 0 until num)
        println(x)
  }

Where I have some condition that is checked and if true, proceeds into a for-loop.  The error that I repeatedly get is:
  • Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.Range$ByOne.foreach$mVc$sp(Lscala/Function1;)V
        at org.spin.node.gogrid.Client$.doesntWork(Client.scala:62)
        at org.spin.node.gogrid.Client$.main(Client.scala:72)
        at org.spin.node.gogrid.Client.main(Client.scala)
However, this seems to work:
  def nestedWorks(num: Int) = {
    if (true)
      for (x <- (0 until num).toList)
        println(x)
  }

Very odd. Even odder is if I define either of these functions on the Scala interpreter in Eclipse, they work fine. It's only when the nestedDoesntWork is defined within a class and I try to run a main function calling that class. The same happens if I take out the if(true) line.

Solution? Clean project...and look, everything is fine and dandy. Make a change, get the error? Clean the project, and hurray.  Note that the nestedWorks always works...without cleaning.

Annoying to say the least, but today's lesson is if it doesn't work when it should the first time, clean the project.  If it still doesn't work, clean it again. and again. and again.

Monday, November 1, 2010

Scala & Json (Lift-web): Trouble with "mapping" field name to another name

This is a trivial mistake that just ate up the last 2 hours of my life that I will never get back again.

One thing that is important is to be able to take the "raw" parsed json object and rename some of the fields since GoGrid uses some names that are reserved and/or have other Scala meanings.  For example, GoGrid uses Object and Option to mean things specific to their environment.  

This is possible by using the JValue's map function to postprocess AST.  Following the example in the readme documenation, I reproduced the following:
package org.gogrid.sandbox
abstract class jsonSandbox
{ }

case class Person(firstname: String)

And then started up the Scala interpreter:
Welcome to Scala version 2.8.0.final (Java HotSpot(TM) Server VM, Java 1.6.0_20).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.gogrid.sandbox._

scala> import net.liftweb.json.JsonParser._
import net.liftweb.json.JsonParser._

scala> implicit val formats = net.liftweb.json.DefaultFormats 
formats: net.liftweb.json.DefaultFormats.type = net.liftweb.json.DefaultFormats$@1c9fe7e

scala> val jsonString = """ { "first-name" : "Jim" } """
jsonString: java.lang.String =  { "first-name" : "Jim" } 

scala> val json = parse(jsonString)
json: net.liftweb.json.JsonAST.JValue = JObject(List(JField(first-name,JString(Jim))))

scala> json map {
     | case JField("first-name", x) => JField("firstname", x)
     | case x => x
     | }
<console>:16: error: not found: value JField
       case JField("first-name", x) => JField("firstname", x)
            ^

Heh? Why is JField not found???

My "Oh DUH moment":

I did NOT import the package where things like JField, JArray are defined.
 
Insert wall bash here.
scala> import net.liftweb.json.JsonAST._
import net.liftweb.json.JsonAST._

scala> json map {
     | case JField("first-name", x) => JField("firstname", x)
     | case x => x
     | }
res4: net.liftweb.json.JsonAST.JValue = JObject(List(JField(firstname,JString(Jim))))

Dang it.

Scala & Json (Lift-web): Trouble with getting started

I'm abandoning using XML parsing for this project.  While I can get the results back from GoGrid now, I can't seem to figure out a simple, clean way to get the result string transformed into the objects for manipulation in our code.  GoGrid has the XML set up as basically key value pairs, where for each object 'type' is specified as an attribute to the element, which makes squishing the result string through something to pop out an object at the end kind of an ugly hassle.

GoGrid does provide the default response format to be Json, so I'm now chasing down this avenue.

First up is to find an appropriate library to handle the heavy lifting, and for this I'm using Lift-Web's Json library.  Lift-web is a large project made up of several projects, for my purposes, I only need the Json module. It took a little bit of scrounging around the net, but I found that what I need is the lift-json module.  The getting started documentation that I found on lift-web's site tells you how to set up a project from scratch, which is not what I need.  I just needed access to this particular lib.  After browsing their source tree, I decided to just put the information into my project's pom that I found here combined with the announcement that Lift-Web 2.1 has been released and inserted the following into my pom:
//DOES NOT WORK

 net.liftweb
 lift-json
 2.1

//DOES NOT WORK
This (as the comments note) does not work.  I ended up with the following error when I tried to save the pom:

Missing artifact net.liftweb:lift-json:jar:2.1:compile

Okay, so that didn't work.  So then I tried using version 2.0 instead to see if that will work:
//SORT OF WORKS

    net.liftweb
    lift-json
    2.0

//SORT OF WORKS

Well this seemed promising.  I was able to save the pom with no errors.  So the next step was to give the JsonParse -ing a whirl in the command line Scala interpreter (Eclipse v3.5.2 with Scala IDE for Eclipse v1.0.0.201009232358 plugin).  I found an example on Stackoverflow and tried it out for myself:
scala> import net.liftweb.util.JSONParser
:8: error: value util is not a member of package net.liftweb
       import net.liftweb.util.JSONParser

Um, okay. So that didn't work.  Looking through the library that I added, I decided to try this next:
scala> import net.liftweb.json
import net.liftweb.json

scala> val jStr = """ { "name" : "Jim Bob" } """
jStr: java.lang.String =  { "name" : "Jim Bob" } 

scala> json.JsonParser.parse(jStr)
error: error while loading JsonParser, Scala signature JsonParser has wrong version
 expected: 5.0
 found: 4.1 in /home/sophia/.m2/repository/net/liftweb/lift-json/2.0/lift-json-2.0.jar(net/liftweb/json/JsonParser.class)
Heh?! What the heck is going on? After some more fruitless searching on the web, I looked back at this site and saw that there was also a lift-json_2.8.0 package. I am using Scala 2.8.0 ... so I changed my pom to:
//SUCCESS!!

 net.liftweb
 lift-json_2.8.0
 2.1

//SUCCESS!!
And... Hurray!  That did the trick:
scala> import net.liftweb.json
import net.liftweb.json

scala> val jStr = """ { "name" : "Jim Bob" } """
jStr: java.lang.String =  { "name" : "Jim Bob" } 

scala> json.JsonParser.parse(jStr)
res5: net.liftweb.json.JsonAST.JValue = JObject(List(JField(name,JString(Jim Bob))))

Another note for getting started is that the string that is being parsed can be escaped or not, they both work:
//Un-escaped quotes in json string
scala> val jStr = """ { "name" : "Jim Bob" } """
jStr: java.lang.String =  { "name" : "Jim Bob" } 

scala> json.JsonParser.parse(jStr)
res5: net.liftweb.json.JsonAST.JValue = JObject(List(JField(name,JString(Jim Bob))))

//Escaped quotes in json string
scala> val badJstr = " { \"name\" : \"Jim Bob\" } "
badJstr: java.lang.String =  { "name" : "Jim Bob" } 

scala> json.JsonParser.parse(badJstr)
res6: net.liftweb.json.JsonAST.JValue = JObject(List(JField(name,JString(Jim Bob))))
So the next thing on the list is to take a Json string, parse it and shove it into an object.  An example of this can be found on lift-web's source code site and scroll down to extraction. I decided to go for a very simple test, using just a list of numbers (I'm still on the console in Eclipse with Scala interpreter):
scala> import net.liftweb.json.JsonParser._
import net.liftweb.json.JsonParser._

scala> implicit val formats = net.liftweb.json.DefaultFormats 
formats: net.liftweb.json.DefaultFormats.type = net.liftweb.json.DefaultFormats$@7db754

scala> case class Winn(numbers : List[Int])
defined class Winn

scala> val jsonStr = """{ "numbers" : [1,2,3,4] } """
jsonStr: java.lang.String = { "numbers" : [1,2,3,4] } 

scala> val json = parse(jsonStr)
json: net.liftweb.json.JsonAST.JValue = JObject(List(JField(numbers,JArray(List(JInt(1), JInt(2), JInt(3), JInt(4))))))

scala> json.extract[Winn]
net.liftweb.json.MappingException: Parsed JSON values do not match with class constructor
args=
arg types=
constructor=public Winn(scala.collection.immutable.List)
 at net.liftweb.json.Meta$.fail(Meta.scala:128)
 at net.liftweb.json.Extraction$.instantiate$1(Extraction.scala:200)
 at net.liftweb.json.Extraction$.newInstance$1(Extraction.scala:222)
 at net.liftweb.json.Extraction$.build$1(Extraction.scala:240)
 at net.liftweb.json.Extraction$.extract(Extraction.scala:284)
 at net.liftweb.json.Extraction$.extract0(Extraction.scala:172)
 at net.liftweb.json.Extraction$.extract(Extraction.scala:40)
 at net.liftweb.json.JsonAST$JValue.extract(JsonAST.scala:288)
 at .(:17)
 at .()
 at RequestResult$.(:9)
 at RequestResult$.()
 at...

So close...but not quite there.  Back to the googling board.  I eventually found this in lift-web's markmail and they say that:

"The extraction feature uses paranamer to reflectively read the constructor parameter names. Unfortunately paranamer lib does not work with Scala console. Your example should work if you put your case class definitions into .scala file and then import those after compilation."

Okay, so now I created a new scala file in my project that contains the following case class:
package org.gogrid.sandbox
abstract class jsonSandbox
{ }

case class Winn(numbers : List[Int])

And then stopped the previous Scala interpreter and then right clicked over the package containing my sandbox class in the package explorer and started up a new Scala interpreter from there:
Welcome to Scala version 2.8.0.final (Java HotSpot(TM) Server VM, Java 1.6.0_20).
Type in expressions to have them evaluated.
Type :help for more information.

import org.gogrid.sandbox._

scala>
scala> Winn
res0: org.gogrid.sandbox.Winn.type = &ltfunction1>

scala> import net.liftweb.json.JsonParser._
import net.liftweb.json.JsonParser._

scala> implicit val formats = net.liftweb.json.DefaultFormats 
formats: net.liftweb.json.DefaultFormats.type = net.liftweb.json.DefaultFormats$@166de01

scala> val jsonStr = """{ "numbers" : [1,2,3,4] } """
jsonStr: java.lang.String = { "numbers" : [1,2,3,4] } 

scala> val json = parse(jsonStr)
json: net.liftweb.json.JsonAST.JValue = JObject(List(JField(numbers,JArray(List(JInt(1), JInt(2), JInt(3), JInt(4))))))

scala> val w = json.extract[Winn]
w: org.spin.node.gogrid.objects.Winn = Winn(List(1, 2, 3, 4))

scala> w.numbers
res3: List[Int] = List(1, 2, 3, 4)

scala> w.numbers(0)
res4: Int = 1

That's the best "Int = 1" I've seen in a long time :) .

So to summarize, to get started:
  1. Add a dependency on lift-json_2.8.0, version 2.1
    
     net.liftweb
     lift-json_2.8.0
     2.1
    
    
  2. You can choose to escape or not the quotes within the json string, they both work.
    scala> val jStr = """ { "name" : "Jim Bob" } """
    jStr: java.lang.String =  { "name" : "Jim Bob" } 
    
    scala> val badJstr = " { \"name\" : \"Jim Bob\" } "
    badJstr: java.lang.String =  { "name" : "Jim Bob" } 
    
  3. Case classes that represent the objects you are munging the string into cannot be defined in the class you are using them (and also can not be defined in the Scala interpreter of Eclipse).
  4. Must include the implicit formats line of code
    implicit val formats = net.liftweb.json.DefaultFormats 

    Otherwise you will get this error:
    error: could not find implicit value for parameter formats: net.liftweb.json.Formats

Monday, October 25, 2010

Scala, Jax-RS, and GoGrid: Trouble w/calculating signature

Today's lesson: Not all time is created equally.

One important task for connecting to the GoGrid API is calculating your signature.  From their documentation, this is calculated by:

"generating an MD5 hash made up by concatenating the API key, the API user's shared secret, and a UNIX timestamp reflecting the number of seconds since the Unix Epoch (January 1 1970 00:00:00 GMT) when the request is made."

The keyword in this definition is: seconds since the Unix Epoch.

Unfortunately, I did not read this definition closely, and naively translated my already working Python implementation:
import md5
import time

class GoGridClient:
  """sample gogrid api client"""
  api_key = 'xxxxxxxxxxx'
  secret = 'someonesSecret'
  
  ....
  
  def getSignature(self,key,secret):
    """ create sig from md5 of key + secret + time """
    m = md5.new(key+secret+str(int(time.time())))
    return m.hexdigest()

to the following Scala implementation, with a little help from Code Redefined's post for calculating the MD5 sum in Scala:
import java.security.MessageDigest

object BrokenClient {
  val apiKey = "xxxxxxxxxxx"
  val apiSecret = "someonesSecret"
  val apiSignature: String = {
    //        m = md5.new(key+secret+str(int(time.time())))
    val signature = MessageDigest.getInstance("MD5")
    val timeStamp = System.currentTimeMillis().toString //<-- WRONG!!!!
    signature.update(apiKey.getBytes)
    signature.update(apiSecret.getBytes)
    signature.update(timeStamp.getBytes)
    signature.digest().map(0xFF & _).map { "%02x".format(_) }.foldLeft("") { _ + _ }
  }
}

And sadly kept getting a 403: Authentication Failed error. The correct way to implement this is:
import java.security.MessageDigest

object BrokenClient {
  val apiKey = "xxxxxxxxxxx"
  val apiSecret = "someonesSecret"
  val apiSignature: String = {
    //        m = md5.new(key+secret+str(int(time.time())))
    val signature = MessageDigest.getInstance("MD5")
    val timeStamp = ((java.util.Calendar.getInstance(TimeZone.getTimeZone("GMT-0:00")).getTimeInMillis()) / 1000).toString //<-- YAY ^_^
    signature.update(apiKey.getBytes)
    signature.update(apiSecret.getBytes)
    signature.update(timeStamp.getBytes)
    signature.digest().map(0xFF & _).map { "%02x".format(_) }.foldLeft("") { _ + _ }
  }
}

Note that there is some kinda weirdness going on, in that timeStamp works even if you don't pass in a time zone, or if you pass in a random time zone.

Scala, Jax-RS, and GoGrid: Trouble with annotations


At the moment, I am trying to create a scala client that talks to GoGrid using their REST-like API.  I am using Apache's Jax-RS implementation.  There are examples (here and here) available online on how to use the Apache implementation to create a Java client and I have been trying to adapt these examples to Scala.

One annoyance I have encountered so far is how to convert:
@Path("/myResource")
@Produces("text/plain")
public class SomeResource {
    @GET
    public String doGetAsPlainText() {
        ...
    }
}

My first pass at this is:
import javax.ws.rs.{ Path, GET, Produces }

@Path("grid/server/list")
@Produces("text/plain")
class ListRequest {
    @GET
    var plainText : String = _
}

Which produces an obscure error (or at least obscure to me):
Error: "annotation argument needs to be a constant; found: "text/plain" {<error>}
After trying a variety of things to make this acceptable for Scala and a lot of time searching to no avail, I randomly came across this post, where the author is trying to do something similar and found out that the code needs to look like this:
import javax.ws.rs.{ Path, GET, Produces }

@Path("grid/server/list")
@Produces(Array("text/plain"))
class ListRequest {
    @GET
    var plainText : String = _
}

Note that "text/plain" is now an element of an array...so apparently when the error says that the argument needs to be a constant...it meant it needs to be in an Array.  Okay...

Friday, October 22, 2010

SyntaxHighlighter and Blogger

Ironically, it took me a little while to futz around with syntaxHighlighter to get it to work nicely with blogger so I could post my first post...which will now be my second post since my first post is now going to be about syntaxhighlighter and blogger. After a little bit of googling, I found myself at this very helpful post from Tips for software engineer. However, as I'm sure many of you fellow googlers have found out, sometimes these helpful posts are slightly dated and don't have up to date information on how to get things done. Following the directions on his post, I attempted my first test post by copying and pasting from some xml from Gedit. And for some reason, I ended up with:
<configuration> 

     <sources> 

       <source>src/main/scala</source>

    </sources> 

</configuration> 

For whatever reason, there was an extra line break inserted between each of these lines...which was driving me crazy. That's not very useful at all for copying and pasting. After a bit more searching, it turns out SyntaxHighlighter now has a fix for the extra <br /> that Blogger inserts into posts which causes the extra line breaks:


Note the "dp.SyntaxHighlighter.BloggerMode();" on line 2. Now this is looks all great and dandy, however...my first tests posts *still* had the annoying extra line breaks. The solution appears to be changing some 'default' blogger settings. Expand the Post Options underneath the box where you are entering your post and make sure that [Use <br / > tags] is selected for Edit HTML Line Breaks.

 <configuration> 
     <sources> 
       <source>src/main/scala</source>
    </sources> 
 </configuration> 


Ah, much better.

Of course while I am trying to write this post, I realized that SyntaxHighlighter has a new website with a new version and new functionality, which for whatever reason is *not* the first result returned in google...okay, it's the third.  Anyways, to add to the plethora of available step by step instructions available for getting SyntaxHighlighter to work with Blogger that worked for me:

  1. Download the latest release here as well as an older release here.
  2. Unzip the files and then upload the following to a host of your choice (I use Google Sites, yay for free! EDIT: Site must be viewable by public):
    • From the current release folder:
      • shcore.css - Core stylesheet
      • shThemeDefault.css - At least one theme, I chose the default theme. 
      • shCore.js - Core javascript
      • shBrushXML.js - As many javascript brushes as you want,
    •  From the older release folder:
      • clipboard.swf - code to enable clipboard, not sure why is not included in the current release folder.
  3. In your Blogger account, go to the Design tab, and then select the Edit HTML sub-tab.
  4. Download your template as it is now somewhere in case everything goes whacky and you need to go back to the original.
  5. Insert the following code snippet into the Edit Template box just before the </body> tag.
        
        <link href='http://where.your.stuff/is/hosted/syntaxhighlighter/shCore.css' rel='stylesheet' type='text/css'/>
        <link href='http://where.your.stuff/is/hosted/syntaxhighlighter/shThemeDefault.css' rel='stylesheet' type='text/css'/>
        
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shCore.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushBash.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushJava.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushJScript.js'/>   
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushPerl.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushPhp.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushPython.js'/>
        <script type="text/javascript" src='http://where.your.stuff/is/hosted/syntaxhighlighter/shBrushXml.js'/>
        
        <script language='javascript'>
          SyntaxHighlighter.config.bloggerMode = true;
          SyntaxHighlighter.config.clipboardSwf = 'http://where.your.stuff/is/hosted/syntaxhighlighter/clipboard.swf';
          SyntaxHighlighter.all();
        </script>
    
    
  6. Save the template
  7. When you go to make your first post using this, make sure that under Post Options, Use <br / > tags is selected for Edit HTML Line Breaks.
  8. When you are editing your post and need to insert a bit of code for the syntax highlighter, switch to the Edit HTML tab of the editor and insert the following:
      
    
      
    
        Some other xml stuff
    
     
    Where you substitute "brush:xml" with "brush:[whichever brush you are using]".
One weirdness I also noticed is that sometimes it looks likes it's not working, but waiting a few seconds and refreshing, everything is okay again.

Hopefully this was helpful to someone...and hopefully I haven't been too redundant.

another test

 <execution>
   <id>add-test-source</id>
   <phase>generate-sources</phase>
   <goals>
    <goal>add-test-source</goal>
   </goals> 
    <configuration>
      <sources>
      <source>src/test/scala</source>
     </sources>
    </configuration>
  </execution>