Moved to Hugo

Blog just moved to aimplicits.com, please go there for new content. The reason is that static site generators (such as Hugo which is my choice) are more convinient for tech blog as they directly support Markdown and let version-control the blog, for example using Git. Github Pages offer free hosting for static sites (such as blog generated by Hugo), and I find this approach much more convinient. See you on new place!

Mockito and default parameters in Scala

Suppose we need to mock such service and verify that doSomething is called exactly once, and then no interaction happens with the service:

class SomeService {
    def doSomething(from: Int = 0, to: Int = 10): Unit = {
      //... implementation ...
    }

    //... other methods ...
}

A common approach to do it:

val serviceMock = Mockito.mock(classOf[SomeService])

//run tested code which invokes this:
serviceMock.doSomething() 

Mockito.verify(serviceMock).search(0, 10)
Mockito.verifyNoMoreInteractions(serviceMock)

But here’s surprise: it fails with such error message:

Argument(s) are different! Wanted:
doSomething.search(0, 10);
-> at .....
Actual invocation has different arguments:
doSomething.search(0, 0);
-> at ....

Somehow, default params weren’t picked - instead doSomething was called with zeros. The mistery continues if we replace Mockito.verify line with this one:

Mockito.verify(serviceMock).search(Matchers.anyInt(), Matchers.anyInt())

Now error message tells us:

No interactions wanted here:
-> at ...
But found this interaction:
-> at ...
***
For your reference, here is the list of all invocations ([?] - means unverified).
1. [?]-> at ...
2. [?]-> at ...
3. -> at ...

We onestly call doSomething() one time and never call other methods of SomeService. Where from did other 2 calls appear then? Time to look at actual Java representation of default params. Go to the target folder of the project and find the folder with SomeService.class. Then run `javap SomeService’. You’ll find something like this in the output:

public void doSomething(int, int);
public int doSomething$default$1();
public int doSomething$default$2();

That pretty much explains our issues: when you omit parameters and expect to have default values, Scala compiler adds behind the scenes calls to the methods which return those parameters. That’s how additional 2 calls appear. And as the mock doesn’t have return values specified for those methods, it just returns zeros.

Play, ScalaTest and afterAll: There is no started application

Recently I was trying to do some cleanup of shared DB after the tests (with ScalaTest) of Play application using afterAll method of BeforeAndAfterAll trait. DB credentials were contained in Play’s config files so cleanup code relied on running play application. But when trying to run the test I’ve got an exception:

java.lang.RuntimeException: There is no started application

Looks like application was shut down before afterAll gets started. Turns out that traits inclusion order matters here. Following code throws an exception:

class MySpec extends PlaySpec 
             with OneServerPerSuite 
             with BeforeAndAfterAll {
   //tests go here
   override def afterAll(): Unit = {
       //cleanup code goes here
   }
}

while this works perfectly:

class MySpec extends PlaySpec
             with BeforeAndAfterAll 
             with OneServerPerSuite ...

Same applies to OneAppPerSuite. The reason is order of overriding SuiteMixin.run() method. BeforeAndAfterAll attaches there handling of beforeAll and afterAll methods, and OneServerPerSuite attaches testServer.start() and testServer.stop(). Didn’t check it, but appears that due to this ordering only one of beforeAll and afterAll can have access to Play application. So, now beforeAll will fail to access application configs.

Why I dislike Scala enums

I came to conclusion that even in fully-Scala projects it’s better to define enums using Java. Even third-party libraries that have support for Scala enums suffer from the fact that in runtime all Scala enums have the same type. So if you get some enum value - you have no way to tell to which enum type does it belong. Let me show you how bad it can be with json4s. If you have such enums:

object FooVersion extends Enumeration {
  val V1, V2, V3 = Value
}
object BarVersion extends Enumeration {
  val V1, V2, V3 = Value
} 

and such formats:

implicit val formats = org.json4s.DefaultFormats +
  new org.json4s.ext.EnumNameSerializer(FooVersion) +
  new org.json4s.ext.EnumNameSerializer(BarVersion)

you will be able to serialize and deserizlize the data:

val d1 = SampleData(name = "first foo", version = FooVersion.V1)
println("original: " + d1)
val s1 = Serialization.write(d1)
println("serialized: " + s1)
val ds1 = Serialization.read[SampleData](s1)
println("deserialized: " + ds1)

and it will look the same:

original: SampleData(first foo,V1)
serialized: {"name":"first foo","version":"V1"}
deserialized: SampleData(first foo,V1)

but the objects are not going to be equal!

println("original == deserialized: " + (d1 == ds1))

prints following result:

original == deserialized: false

With debugger you may inspect private fields and discover that type of d1.version.scala$Enumeration$$OuterEnum is FooVersion$, and of ds1.version.scala$Enumeration$$OuterEnum - BarVersion$.

Generics with Play JSON

As Play JSON library uses implicits in order to know how to serialize/deserialize some object the type of that object has to be known at compile time. This makes serializing/deserializing generic classes not that straightforward - even if you know at compile time type which is substituted in generic class type parameter, you need to provide somehow reads/writes for each such type, not just for single generic class. The trick is to use def instead of val. Here is an example

Say, we have reads and writes for some classes Foo and Bar, and we’d like to serialize/deserialize such containers with them:

case class MyContainer1[T](data: Seq[T])
case class MyContainer2[T](name: String, data: Seq[T])

You’ll need to define writes like this:

implicit def myContainer1Writes[T](implicit dw: Writes[T]) = 
  new Writes[SeqContainer[T]] {
    override def writes(o: SeqContainer[T]): JsValue =
      Json.toJson(Map("data" -> o.data))
}

implicit def myContainer2Writes[T](implicit dw: Writes[T]) = 
  new Writes[SeqContainer[T]] {
    override def writes(o: SeqContainer[T]): JsValue =
      Json.toJson(Map("name" -> o.name, "data" -> o.data))
}

and reads like this:

implicit def myContainer1Reads[T](implicit dr: Reads[T]):
  Reads[SeqContainer[T]] = 
     (JsPath \ "data").read[Seq[T]].map(MyContainer2[T])

implicit def myContainer2Reads[T](implicit dr: Reads[T]):
  Reads[SeqContainer[T]] = (
     (JsPath \ "name").read[String] and
     (JsPath \ "data").read[Seq[T]]
   )(MyContainer2[T] _)

And now having these implicits in your scope you can write:

val foos1: Seq[Foo] = ...
val json: JsValue = Json.toJson(MyContainer1(foos))
val foos2: Seq[Foo] = json.as[MyContainer1[Foo]].data
foos2 must be(foos1)

Cross-version testing with SBT

I’d like to share my experience on testing versions compatibility with sbt. The system which had to be tested consists of server (Play application) and a client - library that exposes server functionality to other apps. Not all apps that use client library may be updated fast enough with new version of server, so before deploying new server version we must check that client versions which are in use currently keep working. Moreover, tests for the client should demonstrate using it in Play application. Therefore, 2 Play apps should run simultateously during the tests, often with different versions. Read on if you’re interested for solution for such task.

Project structure

Both server, client and cross-version tests are sub-projects inside of common project. They share the same version via version.sbt. Here are the most important dirs and files:

/my-proj
 +-/project
 |  +-MyTestUtils.scala
 +-/my-client
 |  +-build.sbt
 +-/my-client-tests
 |  +-/my-server-launcher
 |  |  +-build.sbt
 |  +-/app
 |  +-/conf
 |  |  +-clientTest.conf
 |  |  +-clientTest.routes
 |  +-/it
 |  |  +-/scala
 |  +-build.sbt
 +-/my-server
 |  +-/app
 |  +-/conf
 |  +-build.sbt
 +-build.sbt
 +-version.sbt

my-server and my-client are nothing special, just regular Play project and regular Scala project. Other files deserve specific attention.

my-client-tests/my-server-launcher/build.sbt

name := "my-server-launcher"

val serverVersion = System.getProperty("serverVersion")

if(serverVersion != null) {
    libraryDependencies += "my.org" %% "my-server" % serverVersion
} else {
    libraryDependencies += "my.org" %% "my-server" % version.value
}

This makes possible to launch specific version of server, as long as it’s available in artifactories known to your project. Stepping forward and assuming that my-server-launcher is known as myServerLaucher to root project:

sbt ";project myServerLauncher; run 9000" -DserverVersion=1.2.3

my-client-tests/build.sbt

name := "my-client-tests"

libraryDependencies ++= Seq(
  ws,
  "org.scalatestplus" %% "play" % "1.4.0" % "it,test"
)

sourceDirectory in IntegrationTest := baseDirectory.value / "it"

Keys.fork in IntegrationTest := true
javaOptions in IntegrationTest += "-Dconfig.resource=clientTest.conf"

routesGenerator := InjectedRoutesGenerator

As my-client-tests is a Play application which demostrates how to use my-client, it requires usual Play configuration - like routesGenerator. Play plugin for this subproject is enabled in root project, I’ll describe that a bit later. Important point is thatKey.fork is set to true - it enables to run fake app for integration/functional tests with different config file then my-server. Otherwise javaOptions would be ignored.

project/MyTestUtils.scala

import sbt.Keys._
import sbt._

object MyTestUtils {
  private val serverPort = 9001
  lazy val acceptanceCommand: Command = Command.command("acceptance")({(state) =>
    var s = state
    s = Command.process("project myServerLauncher", s)
    s = Command.process("set PlayKeys.playInteractionMode := play.sbt.StaticPlayNonBlockingInteractionMode", s)
    s = Command.process(s"run $serverPort", s)   
    s = Command.process("project myClientTests", s)
    val serverUrl = s"http://localhost:$serverPort"
    s = Command.process(s"""set javaOptions in IntegrationTest ++= Seq(""-DtestServerUrl=$serverUrl") """.trim, s)
    s = Command.process(s"it:test", s)
    s
  })
}

MyTestUtils.acceptanceCommand will be attached to root project and run the tests together with server. Sbt task wouldn’t work here because we need to execute Play run which is a command itself. Pay attention to playInteractionMode which makes Play app run in background, together with tests. Also important point is that testServerUrl environment variable is passed to the tests and can be used to correctly configure the client.

root project build.sbt

name := "my-proj"

scalaVersion := "2.11.7"
organization := "my.org"

lazy val root = (project in file(".")).
aggregate(myServer,
  myClient,
  myClientTests).
settings(
  aggregate in test := false /*to avoid running tests from myClientTests in incorrect environment*/
  ).
configs(IntegrationTest).
settings(Defaults.itSettings: _*)


parallelExecution in Test := true
parallelExecution in IntegrationTest := false

lazy val myServer = project.in(file("my-server"))

lazy val myClient = project.in(file("my-client"))

lazy val myServerLauncher = project.in(file("my-client-tests/my-server-launcher")).enablePlugins(PlayScala)

lazy val myClientTests = project.in(file("my-client-tests")).
  enablePlugins(PlayScala).
  configs(IntegrationTest).
  settings(publish := {}).
  settings(Defaults.itSettings: _*).
  aggregate(myClient).
  settings(aggregate in test := false).
  dependsOn(myClient)

commands += MyTestUtils.acceptanceCommand

Here sub-projects gain the names that can be used in sbt and acceptanceCommand is configured. Now tests from my-client-tests can be launched with current version of my-server:

sbt publishLocal
sbt acceptance

or with custom version of my-server:

sbt acceptance -DserverVersion=1.2.3

In order to check that latest changes are compatible with old clients you can publish locally latest version of server, checkout older version of my-project by tag and run sbt acceptance with freshly published latest version.

Scala Parser Combinators Quickstart

Parsing may be a challenging task. And it’s amazing how Scala Parser Combinators make it doable for regular human beings. You can easily describe how to parse simple things like identifier, string constant or argument list, and then combine these simple parsers into more comples ones. I’d like to describe here step-by-step how to get that brilliant library working for you. No deep theory or specific features, just bare minimum to get started with typical tasks. We’re going to develop boolean expression parser and use it to evaluate some expressions. The complete project is available here: https://github.com/paul-lysak/ParserCombinatorsDemo

Include the library in your build

Add "org.scalatest" %% "scalatest" % "2.2.1" % "test" to libraryDependencies in your build.sbt file

Create your model (abstract syntax tree)

Model doesn’t have any dependencies on library classes, you can write here anything that makes sense for your domain - you’ll be instantiating these classes later by hands. Here what I’ve got at this stage:

package parsers

object Ast {
  sealed trait Expression 
  case class And(expressions: Seq[Expression]) extends Expression 
  case class Or(expressions: Seq[Expression]) extends Expression 
  case class Not(expression: Expression) extends Exp
  ression 
  case class Variable(name: String) extends Expression 
}

Whenever there’s a need to use these classes, following import simplifies the job: import parsers.Ast._

Create a place for your parsers

Typically you’d like to extend from scala.util.parsing.combinator.JavaTokenParsers or scala.util.parsing.combinator.RegexParsers. They define all the necessary DSL for creating the parsers. JavaTokenParser extends RegexParsers with some sugar like definition of what is identifier or string constant in Java world - usually it’s quite useful. Here is my approach:

package parsers
import scala.util.parsing.combinator.JavaTokenParsers

object BooleanExpressionParser extends JavaTokenParsers {
  import Ast._

  def parse(str: String): Ast.Expression =
    parseAll(expression, str) match {
      case Success(result, _) => result
      case failedOrIncomplete => throw new RuntimeException(failedOrIncomplete.toString)
    }

  private def expression: Parser[Expression] = ??? //will define it later
//...
//... here go all your parsers
//...
}

The entry point is private dev expression - it defines how to get the root of your AST from the string. Returning type for parsers (like Parser[Expression]) is optional, Scala almost always correctly detects it. However, here I’ll keep fhe types for the sake of clarify.

Define first parser

Let’s parse variable names. For the beginning we’ll follow Java approach, so x, X and someVar are valid names, while 123var not.
Add following parser to BooleanExpressionParser:

private def variable: Parser[Variable] = ident  ^^ (Variable(_))

and change expression to this:

private def expression: Parser[Expression] = variable

Here ident is a regex-based parser (it’s type is Parser[String]) from JavaTokenParsers which checks if a string is a valid identifier and if is - returns the string. ^^ is a combinator defined in scala.util.parsing.combinator.RegexParsers for transforming parsing result. Here ident returns a String, but we cant a Variable. If target class doesn’t need additional argument processing and has only one argument then simples approach like (Variable(_)) may be used.

Following check is satisfied now:

BooleanExpressionParser.parse("x") mustBe(Variable("x"))

Now suppose we need some processing of the string before creating target object. For example, we’d like variables to be case-insensitive. Then variablecan be changed this way:

private def variable: Parser[Variable] = ident  ^^ {case v => Variable(v.toLowerCase)}

This check is satisfied as well now:

BooleanExpressionParser.parse("X") mustBe(Variable("x"))

Parser that checks the prefix/suffix

Add this one:

private def not: Parser[Not] = "not" ~> expression ^^ (Not(_))

and change expression to this:

private def expression: Parser[Expression] = not | variable

Few points to note here:

• | defines alternatives - parsers from the list are tried in order, and first successful is applied
• "not" is also a parser here. Matches string “not” and nothing else. If matched - returns that string
• ~> is a combinator telling that we’re not interested in result of parser on the left side - we just want to know that it matches the input. So result of the left part is discarded
• There’s a counterpart <~ discarding the right side. You’ll see it later in the code

Parse repetitive fragment

Now let’s implement logical operation and, so that the following check would be satisfied:

BooleanExpressionParser.parse("x and y and z") mustBe(And(Seq(Variable("x"), Variable("y"), Variable("z"))))

Define following 2 parsers (here type qualifier in pattern matching are also optional - they’re here just to make the types more clear):

private def leftExpression: Parser[Expression] = not | variable
private def and: Parser[And] = leftExpression ~ ("and" ~> leftExpression).+ ^^
{case (left: Expression) ~ (right: Seq[Expression]) => And(left +: right)}

And change expression to: and | not | variable

Key points here are:

• () - brackets wrap a part of parser into kind of “sub-parser”
• something.+ - quantifier, requires underlying parser to match at least once. Returns Seq[] of what underlying parsers return
• Other quantifiers are also available. To name a few:
  
  • something.* or rep(something)- match zero or more times,
  • repsep(something, separatorChar) - match something zero or more times, separated by separatorChar
  • something.? - match something zero or 1 time, returns Option
• something1 ~ something2 - combines a parsers into a chain, every parser must match successfully and every parser result is preserved. Returns a special data structure that can be deconstructed with the same character ~ in pattern matching part.
• leftExpression has to be introduced. If simply use expression as underlying element of and, then we end up with undefined recursion and StackOverflowException. and expression can’t start at the first character of another and expression - they have to be separated by at least one level of some sub-expressions. We’re going to use brackets later to resolve situation when there are nested expressions which require recursive parser: and -> or -> and

Parser with constant output

Sometimes you’d like to produce the same output for all inputs that match certain criteria. As an example - let’s consider everything that starts with /* and ends with */ as a comment and convert it to an empty string:

private def comment: Parser[String] = """/\*([^*]|\*[^/])*\*/""".r ^^^ ""

Here ^^^ completely discards the result of parsing, just makes sure that parsing was successful. Result is always takebn from the right side - an empty string in this case. You’ll see later how it’s integrated into the whole picture.

The whole picture

Now you have enough knowlege of parser combinators to understand the remaining parts. So here’s the final code of all parsers:

private def expression: Parser[Expression] = combinationExpression | leftExpression

private def combinationExpression: Parser[Expression] =  comment.? ~> or | and <~ comment.?

private def leftExpression: Parser[Expression] = comment.? ~> not | brackets | variable <~ comment.?

private def brackets: Parser[Expression] = "(" ~> expression <~ ")"

private def variable: Parser[Variable] = ident  ^^ {case v => Variable(v.toLowerCase)}

private def not: Parser[Not] = "not" ~> expression ^^ (Not(_))

private def and: Parser[And] = leftExpression ~ ("and" ~> leftExpression).+ ^^
{case (left: Expression) ~ (right: Seq[Expression]) => And(left +: right)}

private def or: Parser[Or] = (and | leftExpression) ~ ("or" ~> (and | leftExpression)).+ ^^ {case left ~ right => Or(left +: right)}

Evaluation of the AST is out of scope of this article, you can see the code on GitHub.

What’s left behind

Some important aspects didn’t find a place in the demo project

Case-insensitive string matching

If you want “true”, “True” and “TRUE” to be regarded as the same value, you need to use regular expressions. You may write your wrapper like this:

private def r(str: String) = ("(?i)" + str).r

And then use it like this:

private def boolConst: Parser[String] = r("true") | r("false")

Exclude reserved words from matching

You may use guard to check the input, but don’t consume it, and then not for making a parser fail if sub-parser succeeds:

private def reservedWords: Parser[String] = guard(“if” | “case” | “class” | “object” | “trait”)
private def notReservedIdentifier: Parser[String] = not(reservedWords) ~> ident

Hope it will make a start with parser combinators easier for you!