sweet mustard compiler



Running Scala On AWS Lambda

2020-06-14 / AWS Lambda / Scala / 5 minutes

How to run a simple Scala function on AWS Lambda. Since no Scala runtime is provided out-of-the-box in AWS Lambda, the Scala program needs to be compiled for the Java 8 runtime. Function input and output are JSON strings which are encoded and decoded using Scala’s circe library.

Prerequisites

This post assumes you have

  • an AWS account,
  • an AWS role that permits executing AWS Lambda functions,
  • an IntelliJ IDEA installation with Scala plugin,
  • and a basic understanding of Scala.

Implementation

The Scala Project

Create a new Scala (sbt) project in IntelliJ IDEA called scalaOnAWSLambda. Make sure to select a JDK for version 1.8. If you don’t have one, you can install it right from the IDE. In this example, we use Amazon Corretto 8.

setup

The Scala Function

The function we want to run looks pretty much like the functions you might be familiar with from school

f(x, z) = 500 + 1 x + 0.05 z

Think of this as a scoring function, where x and z represent numerical attributes of a client (like age and income) and the output is the predicted monthly expenditure.

The Scala code for this function is shown below. Instead of using x and z, we are using the more descriptive argument names age and income, and we call the function predictExpenditure rather than f. The code for Main.scala looks as follows:

package scoring

object Main extends App {
  def predictExpenditure(age: Double, inc: Double): Double = {
    500 + 1 * age + 0.05 * inc
  }

  println(predictExpenditure(age = 50, inc = 1000)) // for debugging purposes
}

Next, let’s check if this code successfully compiles. In IntelliJ IDEA, go to sbt shell at the bottom and type run to compile and run the program. You will get an output similar to

[IJ]sbt:scalaOnAWSLambda> run
[info] Compiling 1 Scala source to C:\Users\flori\OneDrive\Scala\scalaOnAWSLambda\target\scala-2.13\classes ...
[info] Done compiling.
[info] running scoring.Main
600.0
[success] Total time: 1 s, completed 11.06.2020 10:41:23
[IJ]sbt:scalaOnAWSLambda>

The fifth line shows the output of the function that we printed to the console (600.0), which is the correct output.

The Java Function

Note that the previous compilation was targeted for the Scala runtime, not the Java runtime. You see a file scalaonawslammda_2.13-0.1.jar in target\scala-2.13. Check that the program only runs on Scala runtime, but not directly on Java runtime by running the jar in Scala and Java from the console:

C:\Users\flori\OneDrive\Scala\scalaOnAWSLambda>scala target\scala-2.13\scalaonawslambda_2.13-0.1.jar
600.0

C:\Users\flori\OneDrive\Scala\scalaOnAWSLambda>java -jar target\scala-2.13\scalaonawslambda_2.13-0.1.jar
Error: A JNI error has occurred, please check your installation and try again
[...]

As AWS Lambda does not provide a Scala runtime, we first need to compile our Scala program for the JVM (version 1.8 or “Java 8). We can do this by creating a “fat” jar that includes standard Scala libraries necessary to run Scala code.

To achieve this, first create a file scalaOnAWSLambda\project\plugins.sbt and add the line

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")

This adds command assembly to sbt. This will allow us to create the fat jar.

Next, open scalaOnAWSLambda\build.sbt and add line

javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")

to tell sbt to compile for Java 8 version.

Finally, in sbt, run compile and assembly. The latter creates a second jar scalaOnAWSLambda-assembly-0.1.jar in target\scala-2.13, which can be run in JVM directly:

C:\Users\flori\OneDrive\Scala\scalaOnAWSLambda>java -jar target\scala-2.13\scalaOnAWSLambda-assembly-0.1.jar
600.0

Integration to AWS Lambda

Unfortunately, the program as we have it now, will not run in AWS lambda. This is due to the fact that Lambda expects a function with a standard signature. Hence, we create function lambdaHandler that takes care of parsing the input and formatting the output:

def lambdaHandler(in: InputStream, out: OutputStream): Unit = {
    val inputString = scala.io.Source.fromInputStream(in).mkString
    ...
    val result = ...
    out.write(result.getBytes("UTF-8"))

The input received is a (Java) InputStream byte-stream object, the output is an Outputstream. Using Scala’s standard libary scala.io.Source we convert the input to a string object. The result is also a string, which gets sent to the output.

As input and output are expected to be JSON strings, we need a simple way to parse JSON. We will use Scala’s circe package for that. To add circe to our project, we need to add the following packages to our build.sbt file, using

val circeVersion = "0.12.3"

libraryDependencies ++= Seq(
  "io.circe" %% "circe-core",
  "io.circe" %% "circe-generic",
  "io.circe" %% "circe-parser"
).map(_ % circeVersion)

With case classes Client and ScoredClient describing the fields for input and output, respectively, parsing and compiling JSON will be easy:

case class Client(age: Double, income: Double)
case class ScoredClient(expenditure: Double, age: Double, income: Double)

Solution

The final Scala code is:

package scoring

object Main extends App {
  import java.io.{InputStream, OutputStream}
  import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._

  case class Client(age: Double, income: Double)
  case class ScoredClient(expenditure: Double, age: Double, income: Double)

  def predictExpenditure(age: Double, inc: Double): Double = {
    500 + 1 * age + 0.05 * inc
  }

  def lambdaHandler(in: InputStream, out: OutputStream): Unit = {
    val inputString = scala.io.Source.fromInputStream(in).mkString
    val input = parse(inputString).getOrElse(Json.Null).
      as[Client].getOrElse(Client(0,0))

    val age = input.age
    val inc = input.income
    val exp = predictExpenditure(age = age, inc = inc)

    val result = ScoredClient(exp, age, inc).asJson.spaces4
    out.write(result.getBytes("UTF-8"))
  }
}

Again, run compile and assembly to create a fat jar and upload the jar to AWS Lambda. Make sure to specify scoring.Main::lambdaHandler as the handler in the basic settings.

Test your function with sample input event

{
  "age": 60,
  "income": 800
}

which should return

{
  "expenditure": 600,
  "age": 60,
  "income": 800
}

In a later post, I will show how to use this lambda function as an endpoint for AWS API Gateway. This will allow you to call this function using HTTP requests, e.g. using data input via a web form.