/var/log/mind

Analysing Tourism Data with Kotlin

2018-01-09T00:00:00+05:30

Here's a brief example of some of the functional processing capabilities of Kotlin.

To get some raw data I headed out to http://data.gov.in

The original data set can be found at Foreign Tourist Arrivals In India From Top 15 Source Countries

The modified CSV with the continents added is here

Thus the CSV has the following columnar structure.

Column 1: Country of Tourist
Column 2: Continent
Columns 3 onwards: Number of tourists visiting India starting 2001 to 2015

The problem statement I made up is kind of contrived to be able to demonstrate some interesting coding aspects. In any case the statement is as follows:

Read the file. Group all the data by continents. For each continent compute the following

Total number of tourists from all countries in that continent in 2015
Percentage growth for total number of tourists for that continent from 2001 to 2015
Country from whom the maximum number of tourists visited India in 2015 from that continent

Display the data in the descending order of the percentage growth rate from that continent

The resultant Kotlin code is as follows

import java.io.File

data class CountryData(val name: String, val visitors: List<Int>)
data class Result(val tourists2015: Int, val pctGrowth: Int, val maxCountry: String)

fun main(args: Array<String>) {
//  Open file
  File("tourists-to-india.csv")
      // Read all lines
    .readLines(Charsets.US_ASCII)
      // Drop the first line (column headers)
    .drop(1)
      // Drop the last line (file totals)
    .dropLast(1)
      // For each remaining row in the file
    .map { row ->
      // split into cells using a comma as the delimiter
      row.split(",")
          // for the array of cells in each row
          .let { array ->
            // create a pair. The first value is the continent name (array[1])
            // The second value is the CountryData ie.
            //    list of tourists from that country each year starting 2011
            array[1] to CountryData(array[0], array.drop(2).map { it.toInt() })
          }
    }
      // collate all country data for each continent into a list of countrydata
      // for that continent
    .groupBy({it.first}, { it.second })
      // for each continent
    .map { (continent, countriesData) ->
      // compute tourists from across the continent in 2001
      val tourists2001 = countriesData.sumBy { it.visitors[0] }
      // compute tourists from across the continent in 2015
      val tourists2015 = countriesData.sumBy { it.visitors[14] }
      // compute percentage growth
      val pctGrowth = (tourists2015 - tourists2001) * 100 / tourists2001
      // now we want to find out which country in that continent sent
      // the maximum number of tourists
      val maxCountry = countriesData
          // sort data based on 14th element in the list ie. visitors for 2015
          .sortedByDescending { it.visitors[14] }
          // take the first country data and specifically its name attribute
          .first().name
      // construct a pair of continent to result
      continent to Result(tourists2015, pctGrowth, maxCountry)
    }
      // sort the continent result pairs based on the percentage growth
    .sortedByDescending { it.second.pctGrowth }
      // display the results
    .forEach { println(it) }
}

Exercises in Kotlin: Part 5 - Classes

2016-04-29T08:20:00+05:30

After the last post Exercises in Kotlin: Part 4 - Control flows and return, we shall be exploring classes and objects in the exercises in this post. While retaining many similarities to Java classes, there are actually significant differences in syntax as well and these help classes become a little easier to work with in Kotlin. And save a lot of unnecessary boilerplate code.

Minimalist Syntax

Probably the minimalistic (and for all practical purposes useless) declaration of a class in Kotlin will be

class Foo

A typical class

Thats it. No constructor, no fields, no body. However as is obvious this is not particularly useful. Let us look at a more useful one.

class Account (val number: String, var balance: BigDecimal) {
    fun deposit(amount: BigDecimal) {
        balance += amount
    }
    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

Here Account is the class name. It has two properties number which is a String and balance which is a BigDecimal. Another aspect is number is a read-only property ie. you can't change the value within the body of the class (say in a method) after one is provided at the construction time. As mentioned in an earlier post read-only properties (aka final in Java terms) are identified by the keyword val, while read-write properties are identified by var. Notice the succinct constructor. The properties of the class that are required by the primary constructor are specified immediately after the class name within parenthesis. You can have variables declared in the constructor that are neither vals nor vars. However they can be used only during construction time and do not become properties of the class (more about that later). Thus unlike Java you don't need to write constructors just in order copy values from the constructor parameter to a member field. Also unlike Java you do not need to write getter/ setter methods in most cases (though you could if you had a specific reason to write custom getter/setters).

Client code for the account class could perhaps use it as follows. Do note that the object is instantiated using the ClassName(parameter_list) construct. It is essentially similar to Java except that there is no new keyword preceding it.

    // Instantiate object
    val acc1 = Account("123", BigDecimal("456.70"))

    // get value of a property and print it
    println(acc1.number)

    // invoke a method
    acc1.deposit(BigDecimal("234.56"))

    // change the value of a property
    acc1.balance = BigDecimal("567.89")

Member Visibility

Now balance needs to be a var since it changes. But some might be disappointed at how it can be modified from outside the class. One solution would be to make it a private var as follows

class Account (val number: String, private var balance: BigDecimal) {
    fun deposit(amount: BigDecimal) {
        balance += amount
    }
    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

fun somewhereElse() {
    val acc1 = Account("123", BigDecimal("456.70"))

    // The following wouldn't compile
    acc1.balance = BigDecimal("567.89")

    // But neither would the following line
    println(acc1.balance)
}

What if you wanted it to be read-only from outside the class but read-write within the class? You could choose to use the following approach of passing in a parameter to the constructor, using it to declare another property within the body of the class and then declaring that variable to have a private setter. Note that in this case initBalance is simply a parameter passed to the constructor (since it is not declared as either a val or a var), and can be accessed in a read only fashion only during the construction stage. (Thus other methods cannot use this parameter)

class Account (val number: String, initBalance: BigDecimal) {
    var balance = initBalance
        private set

    fun deposit(amount: BigDecimal) {
        balance += amount
    }

    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

Multiple constructors

Now let us assume that we wish to allow the constructor to use a String based value for balance though the balance property on the account should continue to be a BigDecimal. You could modify the class as follows.

class Account (val number: String, initBalance: String) {
    var balance = BigDecimal(initBalance)
        private set

    fun deposit(amount: BigDecimal) {
        balance += amount
    }
    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

Here balance is being initialised by constructing a BigDecimal using the String value. But we know if the value passed in is not a number, a NumberFormatException will be thrown. But we can't see that in the constructor declaration anywhere. Huh?

I won't dwell long on that, but note that Kotlin (unlike Java) does NOT use checked exceptions. Any method could throw any exception at any point in time and it does not have to be documented as a part of the method signature. A fuller discussion of this is beyond the scope of this post, but it should be stated that the conventional wisdom tends towards preferring unchecked exceptions and kotlin supports that style of programming.

Now let us assume we wish to have both types of constructors, one taking the initial balance as a String and another as a BigDecimal. Each class can have one primary and zero or more secondary constructors, each one of them eventually calling the primary constructor via the keyword this. Note the parameter lists to secondary constructors cannot declare properties using val or var since these are anyways in turn declared by the primary constructor.

class Account (val number: String, initBalance: BigDecimal) {
    constructor(number: String, initBalance: String) :
        this(number, BigDecimal(initBalance))

    var balance = initBalance
        private set

    fun deposit(amount: BigDecimal) {
        balance += amount
    }
    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

So far our constructors have been only initialising properties. But what if you wanted them to do more. I have shown the way to do so by adding additional println() statements during both the primary and secondary construction phases as below.

class Account (val number: String, initBalance: BigDecimal) {
    init {
        println("Into primary constructor")
    }
    constructor(number: String, initBalance: String) :
        this(number, BigDecimal(initBalance)) {
        println("Into secondary constructor")
    }

    var balance = initBalance
        private set

    fun deposit(amount: BigDecimal) {
        balance += amount
    }
    fun withdraw(amount: BigDecimal) {
        if (balance < amount) {
            throw Exception("Insufficient funds in the account")
        }
        balance -= amount
    }
}

As can be seen the non property initialisation of primary constructor can be done in the init{...} block. Similarly secondary constructors can also include a block at the end which lists out the necessary statements to be performed.

Data Classes

Very often we need basic classes which are primarily place holders for a collection of different values. Though java does not have an equivalent, in C, you would have called it a struct and in scala you would call it a case class. In Kotlin we have data class. eg.

data class Point(val x: Double, val y: Double)

These classes are tailored for using them as collection of other properties.

A data class automatically generates a sensible equals()/hashcode() functions for the class based on the properties declared in the primary constructor.
A sensible toString() is automatically provided
Another helper method copy() to selectively override some property values when creating a copy is also automatically provided.
You can also use a data class similar to a record of values easily since it also generates component1(), component2(), ... componentN() functions which can treat the class as a record of sequence of properties (in the same order as declared in the primary constructor).

Lets take a usage at a sample usage of the class above

    val p1 = Point(1.1,2.2)
    println(p1)
    println(p1.x)
    println(p1.component1())

    val p2 = p1.copy(x=3.3)
    println(p2)

    val (x, y) = p1
    println("" + x + ":" + y)

The output for the above will be as follows

Point(x=1.1, y=2.2)
1.1
1.1
Point(x=3.3, y=2.2)
1.1:2.2

Note, how a second copy (p2) was easily created from the first while allowing the ability to override a few values. Also how the point object can be destructured into its component values, in this case vals x and y. If the Point class had 3 properties, we would have had to provide 3 variables on the left hand side of the assignment operator for it to succeed

Enum classes

Let us go back to the Account class example. Let us assume we wanted to add an account type member. Account type can either be Savings or Current (aka Checking). In order to allow the capability we will declare an enum class

enum class AccountType {
    Savings, Current
}

Now we can go ahead and modify the Account class as follows

class Account (val number: String, initBalance: BigDecimal, val type: AccountType){ ... }

fun somewhereElse() {
    val acc1 = Account("123", BigDecimal("456.7"),AccountType.Savings)
    println(acc1.type)
}

Now let us take a look at some of the sample usages of the enum class

fun somewhereElse() {
    // Iterate through all the enums
    for( accType in AccountType.values()) {
        println(accType)
    }

    // Every enum instance has two properties: name and ordinal
    println(AccountType.Savings.name)
    println(AccountType.Current.ordinal)

    // You can recreate an enum instance based on a String
    println(AccountType.valueOf("Savings"))
}

This program will result in the following output

Savings
Current
Savings
1
Savings

Thus as you can see, You could iterate through all the enum types by getting an array of all its possible types using Every enum instance has two properties, a name which is a string and an ordinal which allows the enum values to be used a set of ordered values and allows them to be comparable. The name of course is the name as was provided in the declaration and ordinal is the sequence in the declaration of values. * You can use the string value of the enum to obtain an enum instance using valueOf()

Finally you can provide additional properties to each of the enum instances as follows

enum class Color(r: Int, g: Int, b: Int) {
    Red(255,0,0),
    Green(0,255,0),
    Blue(0,0,255)
}

We have looked at the basics of classes in this post. We will look at inheritance next.

Exercises in Kotlin: Part 4 - Control flows and return

2016-04-27T21:20:00+05:30

After Exercises in Kotlin: Part 3 - Functions we now take a look at control flows and actually doing some basic exercises.

if/else

As mentioned earlier, if/else is not just a statement but can also be used as an operator (in lieu of the ternary ? : operator.

As an example, a familiar way to have if/else statements would be as follows.

fun greeting(hours: Int): String {
    var greeting: String = null
    if (hours < 12) {
        greeting = "Good Morning"
    } else if (hours < 16) {
        greeting = "Good Afternoon"
    } else {
        greeting = "Good Night"
    }
    return greeting
}

But since if/else can be an expression as well, another way to write the function above would be

fun greeting2(hours: Int): String =
    "Good " + if (hours < 12) {
        "Morning"
    } else if (hours < 16) {
        "Afternoon"
    } else {
        "Night"
    }

when

There is another expression that can be used instead of multiple nested if/else expressions or situations where you might use a switch/case block in Java. Using when the function above could be written as

fun greeting3(hours: Int): String =
    "Good " + when {
        hours < 12 -> "Morning"
        hours < 16 -> "Afternoon"
        else -> "Night"
    }

Note that when is an expression and evaluates to a value (in the case above it would be "Morning", "Afternoon" or "Night". You could also use it more conventionally just to write other blocks of code eg.

fun greeting4(hours: Int): Unit {
 when {
        hours < 12 -> {
            println("Morning")
        }
        hours < 16 -> {
            println("Afternoon")
        }
        else -> {
            println("Night")
        }
    }
}

As you can note, there is no break statement (unlike switch case), and the right hand side of the conditions can be blocks which consist of multiple statements. If used as a part of a when expression, the value returned by a block is the one returned by the last expression in the block.

When used as an expression, the compiler expects the conditions to be exhaustive and thus many times you might need to add an else block. However the else block is not required when used as a statement.

There are many other things you can do including checking the actual instance or just the value as shown in the rather arbitrary example below

fun anotherWhen(any: Any): Any =
    when(any) {
        (any is Int && any < 10000) -> "Small Number"
        is Int -> "Big Number"
        is String -> "A string"
        else -> "Its just another Any"
    }

You might note that in the first condition, the right hand side of the condition ie. (any < 10000) actually compiles, because the compiler is able to figure out that any will be an Int if reaches the comparison with 10000 and allows it to be used as an Int. This smart casting is actually quite nice and helps save on verbose typecast statements (or for that matter explicit typecasts) when used in when expressions as above.

while and do-while

The familiar constructs of while and do while are also available. eg.

fun countDown(n: Int) {
    var counter = n
    while(counter >= 0) {
        println(counter)
        counter--
    }
}

Note that in kotlin the while block will not allow you to perform an assignment within the while expression unlike java code. eg. while((n = next()) > 0) will not compile. In some of these situations you might actually find the do while construct useful. eg. the val n declared within the do while block remains in scope while evaluating the while expression.

do {
    val n = next()
    // do something
} while(n > 0)

for loop

The for loop is much nicer and often more helpful than in Java. eg. to print each character in a string you could write

for(c in "String") {
    println(c)
}

One of the useful ability of for loop is to iterate over a range which can be written as startValue..endValue eg

// will print 100, 101, 102... 110
for (n in 100..110) {
    println(n)
}

There are many other aspects of the for loop which are better covered after some other topics, so other sample usages will be taken up later.

break and continue

break and continue are also available and continue to have the same meaning as in java. For more advanced uses such as breaking to a or returning to a label, see Returns and jumps from the kotlin reference guide.

Exercise - Counting Sundays

Here's an exercise for you to try out. (Note: the solution follows immediately after, so you might wish to avoid reading it if you want to attempt to do the exercise yourself first).

This one comes from Project Euler Exercise #19

The problem is stated as follows

You are given the following information, but you may prefer to do some research for yourself.


1 Jan 1900 was a Monday.
Thirty days has September,
April, June and November.
All the rest have thirty-one,
Saving February alone,
Which has twenty-eight, rain or shine.
And on leap years, twenty-nine.
A leap year occurs on any year evenly divisible by 4, but not on a century unless it is divisible by 400.

How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)?

One of the possible solutions to this problem is described below

fun sundaysInTwentiethCentury(): Int {
    fun daysInMonth(month: Int, year: Int) = when(month) {
        1,3,5,7,8,10,12 -> 31
        4,6,9,11 -> 30
        2 -> when {
            year % 400 == 0 -> 29
            year % 100 == 0 -> 28
            year % 4 == 0 -> 29
            else -> 28
        }
        else -> throw Exception("Invalid Month ${month}")
    }
    fun daysInYear(year: Int): Int {
        var days = 0
        for(month in 1..12) {
            days += daysInMonth(month, year)
        }
        return days
    }
    fun dayOfWeekAfter(currentDow: Int, numberOfDays: Int) = ((currentDow-1) + numberOfDays) % 7 + 1
    var weekDayOnFirstOfMonth = dayOfWeekAfter(2, daysInYear(1900))
    var totalSundays = 0
    for (year in 1901..2000) {
        for(month in 1..12) {
            if (weekDayOnFirstOfMonth == 1) totalSundays++
            weekDayOnFirstOfMonth = dayOfWeekAfter(weekDayOnFirstOfMonth, daysInMonth(month,year))
        }
    }
    return totalSundays
}

We shall explore classes in the next post Exercises in Kotlin: Part 5 - Classes

Exercises in Kotlin: Part 3 - Functions

2016-04-20T08:00:00+05:30

We looked at a bit of syntax and variable declarations in the last post Exercises in Kotlin: Part 2 - High level syntax and Variables. We explore functions in this post. Here's how a typical function in Kotlin would look like.

fun add(first: Int, second: Int): Int {
    // will not compile
    // first = first + 2. Because first is a val and cannot be reassigned
    return first + second
}

As noted earlier, the arguments are declared with the names followed by the types. Also even though val is not specified, each argument is automatically a val and can not be reassigned within the function. Also the return type is specified after the argument list followed by a colon.

In this case, since the body of the function is an expression it can also alternatively be specified as

fun addExpr(first: Int, second: Int) = first + second

Note in particular that the curly brackets are replaced by =. And because the compiler can infer the return type, it is no longer required to provide it (though it would work just as well if you did provide one)

Functions can have default arguments

fun powerOf(value: Double, power: Double = 2.0) = Math.pow(value, power)

The second argument power is no longer required to be specified, (when not specified its value will default to 2.0. Thus allowing both the following statements

    powerOf(3.0)         // This will return 9.0
    powerOf(3.0, 4.0)    // This will return 81.0

This makes it convenient to require fewer overloaded functions. The order of the arguments needs to be retained, and all the arguments after the ones specified will need to have default values. While this makes a lot of defaulting easy, it still is a little problematic for selectively overriding defaults for only few of the arguments. This is helped by the next feature.

Arguments can also be named

fun contrived(first: String = "First",
              second: String = "Second",
              third: String = "Third",
              fourth: String = "Fourth",
              fifth: String = "Fifth") = 
        first + " " + second + " " + third + " " + fourth + " " + fifth

// ....

fun someOtherFunction() {
    contrived(second="2nd", fourth="4th") 
    // the above returns "First 2nd Third 4th Fifth"
}

As shown in the example above using a combination of default and named arguments we can choose to have smaller function invocations by specifying only the non defaults. Thus only the arguments whose default values need to be overridden, making the code both terse and more readable

Functions can have variable arguments

fun showStrings(suffix: String, vararg items: String) {
    for(item in items) {
        print(item + " ")
    }
    print(suffix)
    println()
}

fun somewhereElse() {
    showStrings("!", "Hello", "World") // prints "Hello World !"
}

Function can have variable number of arguments so long as all of them are of the same type. The type of the declared argument then becomes an array of that type. Thus in the above example, items is an array of String or Array<String>.

The above function also introduces you to a for statement when used with an array. The syntax is very simple and natural, for itemVar in arrayVar { ... }

(I imagine there can be only one vararg type argument and that has to be declared at the end but never attempted to do anything differently)

Functions can have nested functions

fun showStrings2(suffix: String, vararg items: String): String {
    val sb = StringBuffer()
    fun appendToBuffer(item: String) {
        sb.append(item)
        sb.append(" ")
    }
    for(item in items) { appendToBuffer(item) }
    appendToBuffer(suffix)
    return sb.toString()
}

....

fun somewhereElse() {
    showStrings2("!", "Hello", "World") // returns "Hello World !"
}

As shown above, appendToBuffer is a nested function (called local function) It can access variables in the parent functions namespace (in the above situation - sb.).

Aside: if - else can also be an expression

In java, if-else is a statement. And has a complementary ternary operator ? : which is an expression. Kotlin merges both by allowing use of if - else as an expression, and such if/else expressions can be used as a part of the function expression body (or a part of regular block bodies as well). eg.

fun isBlankString(arg: String) = if (arg.trim() == "") true else false

For an if/else to be an expression, both if and else branches have to be specified and followed by an expression each

Functions can be tail recursive.

Yes! They can be.

fun factorial(n: Int): Int {
    tailrec fun factorial(accumulator: Int, n: Int): Int = if (n == 1) accumulator else factorial(accumulator * n, n - 1)
    return factorial(1, n)
}

If a function is written in a form that lends itself to tail recursion the compiler generates optimised code that uses a loop based version which no longer risks a stack overflow.

Extension functions.

This is a very interesting and very useful feature to Kotlin. It is the ability to seemingly externally add a method to a class. Thus the function isBlankString introduced above could be rewritten as

fun String.isBlankString() = this.trim() == ""

fun somewhereElse() {
    " ".isBlankString() // returns true
}

Note that in an extension function, you can use this to refer to the receiver of the function (in this case the String object on whom the isBlankString function is called). Also that the receiver class is resolved statically and not dynamically. So if you had an extension function defined for a Base class and a Derived class that inherits from Base, and called it on a variable declared of type Base but actually referencing an instance of type Derived the Base version of the extension function will be called.

Infix functions

Class member functions or extension functions with a single argument, can in turn be declared as infix which allows from syntactic sugar in terms of how that function is invoked. Thus the function

infix fun Int.toThePowerOf(n: Int) = Math.pow(this.toDouble(), n.toDouble()).toInt()

could be invoked using either of the two ways, the latter becoming feasible only because of the keyword infix being used in the function declaration above.

    5.toThePowerOf(3)
    5 toThePowerOf 3

Higher order functions

Higher order functions are functions which take functions as parameters or as a return type. For that let us first understand how a function can be alternatively represented as a variable using a function type.

fun double(n: Int) = n * 2

What if we wish to represent the function above as a variable. We need to understand the type of the function. In this case it happens to be of the type which takes a single Int and returns an Int. Knowing that we can now declare it as a variable

val doubleAsArg = { n : Int -> n * 2}

The way to invoke the function above would be similar to the way double is invoked

doubleAsArg(5) // would return 10

But the more interesting capability that is now possible is that you can write a function which takes another function as an argument, and in turn pass doubleAsArg as an argument to the newly written function. eg.

fun twice(function: (Int) -> Int, arg: Int) = function(function(arg))

fun somewhereElse() {
    twice(doubleAsArg,5) // will return 20
}

doubleAsArg could have been also declared using an alternative syntax as follows. For all practical purposes they are identical. However the declaration below makes the function type very explicit (it was inferred in the declaration above)

val doubleAsArgRedux: (Int) -> Int = { n -> n * 2}

Aside: Function Types

Just like String, Int etc, since functions are also things that can be passed around, they also have types. As is shown above, doubleAsArg above has the type (Int) -> Int. In general a function taking n arguments each of type Type1..TypeN, and returning a type TypeReturn will have the type (Type1, Type2,..TypeN) -> TypeReturn

Lambdas

A lambda function is a function that is not declared upfront but is passed in immediately to another function. In the situation below the function that is passed as the first argument to the function twice is a lambda. A lambda expression is always surrounded by curly braces, its parameters (if any) are declared before -> while the body goes after -> (if required).

fun somewhereElse() {
    twice({n: Int -> n * 10}, 5) // returns 500
}

Note that for a lambda function the declaration of the parameter types is optional, and there is no way to declare the return type of the function (it has to be inferred). And it is always wrapped with curly braces. Thus the general syntax for a lambda function with n arguments is

{ p1: Type1, p2: Type2 .. pn: TypeN -> /* do something */ }

Note that the lambda function body is an expression and does not allow a return statement. It could consist of multiple statements and/or branches so long as they end with an expression. So the following is a valid lambda.

{ n: Int -> val twoN = 2 * n; if (twoN < 100) twoN * 2 else twoN * 3}

Single parameter lambdas

If a lambda has a single argument and its type can be inferred, then the argument declaration along with its type and the subsequent -> can be omitted. In such a case that parameter will have the default name it.

val quadruple: (Int) -> Int = { it * 4}

Anonymous functions

An anonymous function is a function with no name. For obvious reasons you cannot declare it by itself. But you could say use it as the right hand side of an assignment to a variable or pass it as an argument to another function eg.

val triple = fun(n: Int): Int = n * 3

fun somewhereElse() {
    twice(triple, 5)
    twice(fun(n: Int): Int { return n * 3}, 5)
}

This is particularly useful when the return type cannot be contextually inferred

Closures

Note that local functions, lambda functions and anonymous functions can access variables declared in their outer scope, ie. their closure. They can also modify such variables (assuming they are vars).

Receivers

Lambdas and anonymous functions can be used as extension functions as well. Their function type then is ReceiverType.(p1: Type1 .. pn: TypeN) -> ReturnType

Alternative invocation syntax

If a function has one or more arguments, but exactly one argument which is of a function type, then it can be invoked with an alternative syntax where the function is not specified within the parenthesis, but is instead specified after them wrapped in curly braces.

fun doubleAndThen(n: Int, then: (Int) -> Int): Int = then(n * 2)

fun somewhereElse() {
    doubleAndThen(5) { it * 5 } // returns 50
}

Inline functions

You can specifically request that a function be inlined using the inline keyword. eg. you could've alternatively declared the doubleAndThen function above as

inline fun doubleAndThen(n: Int, then: (Int) -> Int): Int = then(n * 2)

In this case the compiler will emit the function (and the lambda passed to it) inlined at the call site where doubleAndThen is called. This causes your compiled code to grow in size but could pay off in performance.

If you do not want a particular lambda passed to the function as inlined, then you could mark it as noinline. eg.

inline fun doubleAndThen(n: Int, noinline then: (Int) -> Int): Int = then(n * 2)

If a function is inlined, then the lambdas passed to it are allowed to have a return

The next post in the series is Exercises in Kotlin: Part 4 - Control flows and return

Exercises in Kotlin: Part 2 - High level syntax and Variables

2016-04-19T09:20:00+05:30

This post first discusses the code that was written so far in the last post, Exercises in Kotlin: Part 1 - Getting Started. Subsequently it focuses on the basic syntax of variables, discusses a bit about packaging and also about Type Inference

A review of code so far

Let us take a look at the code we wrote so far in the last blog post first part of this series and glean what we can from the same.

helloworld.kt

package com.example.kotlin.learning

fun main(args: Array<String>) {
    println("Hello World!")
}
fun greetings() = "Hello World!"

Some of the things that can be noted from this program are follows

Similar package convention: Kotlin follows similar package naming conventions as Java.
Standalone functions: Yes, Kotlin has them. That means there is no requirement to have a class just in order to wrap functions, that frankly have no reasons to be part of a class.
Keyword fun: Functions are declared using the keyword fun.
Variable declaration order: When declaring variables (or arguments to a function as in this case of the arguments to main), the order of declaration is reversed, with the type specification following the variable/argument name and a colon used to separate the two.
Array is a generic type: Arrays are not represented by [] but a generic type Array<T> instead where T is the type whose array is being represented. Thus at a syntactic level there is no necessity for using [] for arrays.
There is a helper function called println() similar to System.out.println()
Semicolon as a statement terminator is optional: A semi-colon does not seem to be mandatory
Optional function return types: Return type of the function is not required. to be specified if the return type is a Unit. Which is why the function main does not require a return type declaration.
Alternative function representation: As the function declaration for greetings() shows, function bodies can be substituted by expressions if the separator = is used between the function declaration and the expression. When the function body is specified as an expression, the compiler will attempt to infer the return type and in most cases no return type declaration will be required.

testgreet.kt

package com.example.kotlin.learning

import org.junit.Test
import org.junit.Assert.assertEquals

class TestGreet() {
    @Test
    fun testGreetings() {
        assertEquals("Greeting should be 'Hello World!'",
                "Hello World!", greetings())
    }
}

From this we can glean the following

Class declaration: A Class declaration is similar to java in the sense it starts with the keyword class followed by the class name. The class body is also wrapped within { and }.
Constructor: There is no explicit constructor that seems to be required. That is not true. The parenthesis after the class name, in this case with empty content, are in fact the argument list for the primary constructor. More details about that later.
Method declarations: Methods are also declared using the fun keyword.
Annotations: Annotations can also be used just like they are used in java (in this case the @Test annotation.

The above gives us some sort of understanding about Kotlin syntax, Let us look at a few more examples to flesh out some of the more commonly used details. This has a main function. So you can copy this code into a file and compile and run it.

Example 1 : Comments and variables

/**
* This is a comment
*/

/*
   So is
   this */

// As is this

val PI = 3.14159

fun circleArea(r: Double) = PI * r * r 

fun main(args: Array<String>) {
    var radius: Double = 1.0 
    val increment = 1 
    // increment = increment + 1 <-- will not compile since increment is a val
    println("The radius of a circle with radius ${radius} is ${circleArea(radius)}")
    radius = radius + increment
    println("The radius of a circle with radius ${radius} is ${circleArea(radius)}")
}

The output of the above program is

The radius of a circle with radius 1.0 is 3.14159
The radius of a circle with radius 2.0 is 12.56636

Note the following :

Comments

Kotlin follows the same syntax for comments as Java does. Enough said.

Primitive types

Kotlin has primitive types. Though they are not named identical to the ones in Java. They are in fact the same names with the first character capitalised viz Boolean, Byte, Int, Long, Float, Double etc. (not sure if there is an exception to that).
There are member methods available for these primitives unlike Java. Thus you can call a toString() on an Int
Although they seem to be like objects, their storage space is the same as primitives in Java unless they are declared as nullable (more about that later) or used by generic classes in which case their storage is like boxed primitives.

Variables

As mentioned earlier, variables can be declared at the top level (eg PI)
Functions can also have local variables. eg. radius and increment
Variables are of two types. val ie. read-only variables and var ie. read-write variables. In Java all variables by default are read-write and those that are to be treated as read-only are marked using the keyword final. However in Kotlin it is required to use one of the two keywords val or var explicitly. The statement // increment = increment + 1 was commented out, because since increment is a val such a statement would not compile. radius on the other hand being a var can change its value, as is done at radius = radius + increment.
A predominantly functional style of programming will almost always use vals and rarely if any vars. The rationale for that is beyond the scope of this post.
A variable may or may not have an explicitly type declared. The compiler will attempt to infer it based on the value it is initialised to. However often that may not be feasible, and the programmer may be required to specify the type explicitly.

Parameters

Parameter declarations for functions are similar to a variable declaration. However they are not preceded by a val or a var. They are always vals. They always need an explicit type declaration.
A function may return no value whatsoever (represented using the type Unit). The return type of the functional is optional for functions having an expression body (as in case of circleArea) or functions that have a block body but have a Unit return type. In other situatons the compiler will not attempt to infer the return type, and the programmer is required to provide it explicitly.

String templates

String interpolation is supported by allowing expressions to be embedded in strings using the ${expr} syntax as in the arguments to println(). This feature is called string templates in Kotlinspeak

(Optional) Some more advanced aspects about variables

There are some nuanced aspects of variables that you might use only occasionally. These are discussed here. You could chose to skip this section and instead move on to the next post since these might entangle you into learning and thinking about infrequently leveraged capabilities.

Consider the following code, which is an extended version of the code we saw earlier. It is kind of a little odd, but has been deliberately constructed to talk about the advanced features.

const val PI = 3.14159

fun circleArea(r: Double) = PI * r * r

var defRadius = 5.0

var defaultRadius: Double
    get() = defRadius
    set(value) { defRadius = value }

val area: Double
    get() = circleArea(defaultRadius)

fun main(args: Array<String>) {
    var radius = 1.0
    val increment = 1
    println("The radius of a circle with radius ${radius} is ${circleArea(radius)}")
    radius = radius + increment
    println("The radius of a circle with radius ${radius} is ${circleArea(radius)}")
    println("The radius of a circle with radius 5.0 is ${area}")
    defaultRadius = 6.0 
    println("The radius of a circle with radius 6.0 is ${area}")
}

The output of the code above is as follows

The radius of a circle with radius 1.0 is 3.14159
The radius of a circle with radius 2.0 is 12.56636
The radius of a circle with radius 5.0 is 78.53975
The radius of a circle with radius 6.0 is 113.09723999999999

Things to be noted.

getters: Variables can have getter functions as shown for the variable area above. Frankly, not sure why they would be used for top level variables. Though I could imagine them to be used to work around variable visibility constraints by having a private var being accessible as a read-only variable function from another file by using a getter function.
setters: We can also similarly use setters as in the case of defaultRadius. Again this is a bit of a contrived example, since one could have just as easily directly used defRadius. However do note, getters and setters to tend to be more useful when used with class member variables, a topic we will get to later.
const variables: Variables can be prefixed with a const as in case of PI above. However this capability is restricted to top level variables only. In addition such variables have to be initialised to a String or a primitive type. and may not have a custom getter. In turn such consts can be used as parameters to annotations.
package namespaces and visibility: Just like top level functions, you can also have top level variables/constants being declared. Also note that all the standalone functions get declared in the namespace of the package. So if you have multiple files with the same package and also same function (or top level variable) names, that will be treated as an error since one of them will end up attempting to redeclare the other. On the other hand if you declare a top level variable as private, it is not private to that package namespace, but is instead treated as private within a file

We shall be reviewing functions in the next post Exercises in Kotlin: Part 3 - Functions

Exercises in Kotlin: Part 1 - Getting Started

2016-04-18T07:50:00+05:30

Kotlin is an exciting new language I have been working with for a couple of months now. I believe it has a lot of features going for it, and I think many will be benefited from learning a bit more about it. Hence this series of posts.

This post helps you with getting the necessary software installed on your desktop for compiling and running kotlin code

Presumed Audience :

The reader is presumed to have a reasonable working knowledge of Java. While one of the reasons Kotlin is particularly exciting is because it is such a useful language for the Android platform, the early parts of this series shall focus on the desktop based JVMs. Since those learnings are common irrespective of whether you wish to use Kotlin for desktop/server or Android programming.

It is also assumed that the reader has working java development kit installed on his desktop. Most or all of the discussion about syntax will also likely compare Kotlin with Java.

Information :

Some of the useful sites for Kotlin are

Kotlin Programming Language : Language Home page.
Kotlin Reference : Reference - Kotlin Programming Language
Intellij Idea : IDE with built in kotlin support. From the same authors who bring you kotlin, ie. JetBrains
Kotlin Eclipse Plugin : Kotlin plugin for eclipse. Note: I haven't used it myself

Getting started using an IDE

Note: if you do not prefer to use an IDE and are likely to code using text editors and build and run using command line, you can skip this section entirely and go to Command Line Usage

Go to the Intellij idea page above and download the latest version. You can choose to download the community version. It has the necessary support for Kotlin. You may alternatively choose to use the Eclipse plugin described above, but I haven't used it so cannot help with getting it to work. Install the intellij idea ide and launch it. In case unsure how to do so, consult the official guide for downloading, installing and launching the ide.

Start a new project. Click on gradle in the left hand column. You should reach the new project dialog box as follows

I always use the gradle build tool, so I end up making the selections as shown. Select any appropriate JVM between 1.6 and 1.8 (built in one should do). You could use the dialog as shown above. You should next see a dialog box for groupid/artifactid as required by gradle (these mean the same thing as they do for maven, so fill up some group id eg. com.example.kotlin and artifactid eg. learning). The rest of this post assumes these two values, so if you use different values, please replace them appropriately in the code as necessary.

On the next two screens you could choose the defaults or customise as necessary

Since gradle defaults to maven like directory structures, let us create the directory necessary for our work using the groupid/artifactid used earlier and maven conventions. Select the "learning" folder at the top of the left pane, and select from main or right click menu, File -> New Directory and specify the directory as follows. (If the menu options File -> New Directory do not appear, you have selected the wrong "learning", select the one at the top of the left pane, not the one in the toolbar above it).

Similarly also create another directory for writing our test cases

You should see the appropriate packages for the main and test areas appear in the left hand pane as follows

Right click on the package name under the src/main/kotlin directory and select "New Kotlin File/Class". It will pop up a dialog where you can enter helloworld

This will create a new file "helloworld.kt" and open it into your ide

Enter the following code in the just opened file

package com.example.kotlin.learning

fun main(args: Array<String>) {
    println("Hello World!")
}

Run the program using "Run -> Run". It will pop up a very small dialog box, select HelloWorldKt from it. (You could also use Alt-Shift-F10 instead of Run -> Run, though not sure if the same key bindings work across operating systems, though thats what it is on Linux). This is the canonical Hello World program for Kotlin. Don't worry about understanding the syntax yet. We will get to it in the very next post, this post is to get you started.

One more thing to do is to get a sample test case running. So, let us add a function to the file we just created that we shall be testing, as follows

fun greetings() = "Hello World!"

Next, select the package in the src/test/kotlin directory, and after a right click select "New -> Kotlin File/Class". This time select the name as "testgreet", which will create a new file called testgreet.kt. Add the following code to the file.

package com.example.kotlin.learning

import org.junit.Test
import org.junit.Assert.assertEquals

class TestGreet() {
    @Test
    fun testGreetings() {
        assertEquals("Greeting should be 'Hello World!'", 
                "Hello World!", greetings())
    }
}

Again select "Run -> Run..." or (Alt-Shift-F10 or equivalent). This will pop up a small dialog box. Select TestGreet. This will run the test case and should show an all green bar near the bottom of your IDE as shown in the image below.

Now, we shall repeat the exercise using text editors and command line for building and running the program. Even if you always use IDEs, I very much recommend that you at least do the remainder of the activities described on this post at least once. It is always helpful to have a general sense of what is happening at a command line level. However if you skip it, or it doesn't work for you for whatever reason, it won't constrain you from being able to continue with the rest of this series.

In either case, I shall not be covering the Intellij Idea IDE any further. If you are new to it or to the Eclipse plugin, learning it is beyond the scope of this series of posts, so I encourage you to play around with them, since that is likely to be useful as we go further along in this series.

Installation for command line usage

The instructions for working with Kotlin from the command line are documented on the page Working with the Command Line Compiler

Goto Kotlin latest release to download the Kotlin compiler. It will likely be a file called kotlin-compiler-<version>.zip. The current version is 1.0.1-2 at the point in time this post was written. Unzip the file into an appropriate directory. And ensure that the executables kotlin and kotlinc (or their windows equivalent with the .bat extensions) are in your path.

Create a work directory for this exercise and change directory to it.

Create a file helloworld.kt with the following content.

package com.example.kotlin.learning

fun main(args: Array<String>) {
    println("Hello World!")
}

fun greetings() = "Hello World!"

Compile it using the command

$ kotlinc helloworld.kt

Notice that it will have created a file com/example/kotlin/learning/HelloworldKt.class. This is the compiled .class file. It is Java byte code though you wouldn't be able to run it as you might be used to just yet. In fact go ahead and try to run it as follows

$ java com.example.kotlin.learning.HelloworldKt

It will fail because you do not have the necessary kotlin runtime jar in your classpath. You should've seen an error message similar to the one below

Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at com.example.kotlin.learning.HelloworldKt.main(helloworld.kt)
Caused by: java.lang.ClassNotFoundException: kotlin.jvm.internal.Intrinsics
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more

That runtime is located in the kotlin zip you had just downloaded. Let us point the classpath to it and attempt to run the program again. (Replace <folder_where_you_unzipped_kotlin_zip> below with the appropriate directory where you had unzipped the file you downloaded)

$ java -cp <folder_where_you_unzipped_kotlin_zip>/kotlinc/lib/kotlin-runtime.jar:. com.example.kotlin.learning.HelloworldKt

You should see "Hello World!" getting printed to the console. Kotlin provides an easier way to do this as well, so let us repeat the process, just a little differently.

Compile the source again, but this time create a .jar file

$ kotlinc helloworld.kt -include-runtime -d helloworld.jar

Now run it as follows

$ java -jar helloworld.jar

The program should run and "Hello World!" should get printed to the console.

Now let us run the test case. Create a file testgreet.kt as follows

package com.example.kotlin.learning

import org.junit.Test
import org.junit.Assert.assertEquals

class TestGreet() {
    @Test
    fun testGreetings() {
        assertEquals("Greeting should be 'Hello World!'",
                "Hello World!", greetings())
    }
}

Now for compiling and running this you will need the jar for junit (which in turn requires hamcrest). The instructions for that are provided on the Download and Install - junit4 wiki page. For now just go ahead and download the two following jars for junit and hamcrest.

You will have downloaded junit-4.12.jar and hamcrest-all-1.3.jar. Move both of them into your current directory after download.

$ kotlinc helloworld.kt -cp junit-4.12.jar -include-runtime -d helloworld.jar

$ java -cp junit-4.12.jar:hamcrest-all-1.3.jar:helloworld.jar org.junit.runner.JUnitCore com.example.kotlin.learning.TestGreet

You should see something like the follows (the time obviously could vary) that indicates success.

JUnit version 4.12
.
Time: 0.004

OK (1 test)

You should also be aware that Kotlin has a REPL that you can run using the command kotlinc-jvm as follows

$ kotlinc-jvm
Welcome to Kotlin version 1.0.1-2 (JRE 1.8.0_77-b03)
Type :help for help, :quit for quit
>>> println("Hello World!")
Hello World!
>>>

Congratulations you are now able to compile and run Kotlin programs. In the next post in this series, we shall attempt to understand the code that we just wrote, and explore the Kotlin language further. You can continue to the next post Exercises in Kotlin: Part 2 - Syntax: Variables

Few thoughts about Kotlin and why I like it so much

2016-04-15T15:00:00+05:30

It doesn't matter how many languages you have learnt, learning yet another one is always a wonderful experience. I have been working for a couple of months with Kotlin. Have got one app to ready for production state and another in the works.

To cut to the chase, I am really impressed by this language. As in really really impressed. And so I decided that I should write a few posts that can help readers get a better idea and learn about this language. So I started writing out the series and am already onto the third post now. But as I was writing it, I realised what I was writing was really about the mechanics of the language. Not so much about its spirit. Cos when one gets into the tactical nitty gritty details, one tends to miss the forest for the trees. So I decided to pen that here before I start that series (next week), and here's my few thoughts about why I really enjoy the language.

One of the reasons there is a lot of enthusiasm with the language is because it can be readily used for programming to the Android platform. That is how I ended up learning it. But very soon, my enthusiasm spread into the desktop/server space where I think Kotlin deserves a shout out for a very very credible effort.

[ EDIT: Deleted a few paragraphs]

So what does Kotlin bring to the table?

Runs on the JVM
Is extremely good at interoperating with existing java code and libraries.
Statically typed (thus far far easier to refactor)
Has a bit more evolved type system than Java.
Is much safer than Java. Nullability is a first class compiler level construct - something that helps address a large number of likely defects aka the billion dollar bug). There is a lot to write home about here.
Has a clear system of distinguishing between mutable and immutable structures. While it doesn't make an opinionated stand in favour of immutability, it gives you the programmer enough helpful constructs to chart your course down that path.
Has support for type inference which helps reduce your boiler plate and keeps code shorter
Has a wonderful story to tell in terms of it providing programmers an ability to write extension functions to existing classes (or for that matter even primitives).
Support for Higher Order Functions, Higher kinded types
Support for Android programming and JavaScript execution engine (experimental).

Does that make it the better language? I personally think it has achieved amongst the best set of tradeoffs that are attractive to me (obviously YMMV). And here's why I enjoy it so much.

Its expressivity is comparable to and superior than python keeping your code short and sweet
It being a statically typed language helps enormously speed up refactoring exercises. Even though I have been using it for only two months, I went through at least two brutal refactoring cycles and found Kotlin code to be a pleasure to change.
It is safer, lot lot safer than python / java and delegates one of the most frequent tasks of programming ie. null checking to the compiler. Thats a huge effort off the back along with reduction in number of probable defects. Surely nothing to complain about here. And believe me this is a big big plus point.
Uses the JVM which is a fine and fast runtime engine
It is easy to learn. I actually went through the reference from start to end in 2 days and was a half decent programmer in about a weeks time. Since then I have written functional libraries for Option, Either, Stack, List, HTTP Clients, XML parsing helper libraries, rather complex X.509 certificate trust chain verifiers and a complete Java FX based app. Have started playing around with more functional constructs such as currying, functors etc. Look forward to working more with typeclasses. I just don't think I could have had such a pace of progress with any other language.
It is non opinionated. For XML parsing I wanted to ensure very low memory footprint because I wanted to use it on android. I was able to write a library which heavily used mutation even as it maintained very low memory usage by keeping extra object allocation counts down. For another situation I wanted to use immutable functional paradigm, and Kotlin happily let me create my own Option/Either/List classes and use that paradigm end to end. It doesn't get in your way. And focuses on getting work done.
I have been interfacing with a large number of java libraries. The interoperability has been an absolute breeze. It is actually refreshing to go through that experience. Doesn't matter if you are dealing with HTTP client APIs, integrating into a Slack HTTP service, interfacing with Java PKCS API, Kotlin does it easily and safely.

In a nutshell, I can write code that is safer, shorter, uses better building blocks, is easier to grok, gives me all the nice capabilities such as HOFs, lambdas, pretty competent type system that gives me all the feedback necessary when writing code and makes refactoring a breeze, and interoperates nicely and leverages the JVM universe.

The Draft Encryption Policy - An impractical exercise

2015-09-22T04:00:00+05:30

The government of india published a new draft encryption policy . Frankly a part of me still imagines this to be an April fools prank gone awry. And that I will wake up tomorrow to news, that this was a screwup and we can all forget about it.

[EDIT: The government has since published an addendum, issuing clarifications. PROPOSED ADDENDUM TO THE DRAFT ENCRYPTION POLICY This post has been updated at the end to reflect the same. In brief while it has reduced the coverage of the applications and services, the essential implementation issues as pointed out to in this post still remain and the policy seems just as silly and onerous after the addendum was published.]

Be as it may, this policy has received substantial criticism early on, not the least of which is about the various privacy issues. This post attempts to look at this policy within the context of security, practicality and law.

Note that encryption as referred to in this policy applies to encryption as used for transmission AND storage. Also the quoted policy text may refer to G, B and C. This refers to Government, Business and Citizens respectively.

An imaginary conversation

I am frankly dumbstruck by the policy draft. I wonder how it even resulted into its current state. My best guess is somewhere in the policy corridor a conversation along the lines below happened. Once again, to emphasise, the conversation below is imaginary.

Politician: We need to have a better ability to intercept all the messages flowing on the internet. Very needed to take on terrorism and is also useful for political intelligence and rapid reaction.

Bureaucrat: Sir, most of the internet sites and mobile apps encrypt all the conversations. So even as we can readily intercept the messages, we cannot easily decrypt them.

Politician: But the US does it. Why can't we as well?

Bureaucrat: Ahh, but they have the NSA and all its resources, and they can arm twist the large internet giants into making backdoor concessions via programs such as PRISM as Edward Snowden showed. They even introduced backdoors into the Dual Elliptic Curve random number generation written by RSA to easily crack the encryption. We cannot do that.

Politician: Lets have the chinese model then.

Bureaucrat: Ahh .. but there there is far more interception of messages and an ability to do better offline monitoring of suspicious citizens, and the government can easily get to the data of a suspicious guy, because there is lesser judicial oversight. We being a democracy with an active judiciary cannot implement that model either.

Politician: OK, I don't really care for the encryption per se. And I don't wish to be lambasted for too much snooping. Just make sure that we can easily access the non encrypted data directly from one of the parties when we need to and there is enough of a legal framework to support that.

Bureaucrat: OK Sir.

The said bureaucrat then sets in motion a series of actions which results in this new draft policy document

Comments on the draft

Lets first take a tour through some of the interesting provisions in the new so called policy.

Use of encryption algorithms and protocols

Use of Encryption technology for communications between G group and B / C groups (i.e. G2B and G2C sectors) with protocols and algorithms for encryption, key exchange, Digital Signature and hashing will be as specified through notification by the Government from time to time.

So the government will decide what kind of algorithms and protocols can be used. Businesses may no longer retain the liberty to choose which algorithms and protocols they may use or use any custom algorithms (even though that is not such a good idea, the freedom to use it is), or innovate to create better algorithms. The government dicates and businesses shall oblige.

Ability to reproduce encryption using software/hardware

On demand, the user shall be able to reproduce the same Plain text and encrypted text pairs using the software / hardware used to produce the encrypted text from the given plain text.

Whenever business do any kind of encryption, they must be able to reproduce both the plain text and the encrypted text.

First of all, if the key has any random or varying element to it the encrypted text may not be the same each time. But no, the government now mandates that you must be able to produce the same encrypted text from the given plain text as you did earlier. Randomness be damned. Which means in case of random keys, the keys will need to get stored as well.

Secondly the user needs control over the software and hardware used to produce the encrypted text. So if you had a cloud based service to do it for you, you would need to find a way to ensure that you are able to re-do the same using the cloud based service on the day the government makes that request of you, and if your cloud based service cannot do it for you, that will now be you the consumer's fault. Or if you are using a mobile app you should be able to reproduce both the plain text and the encrypted text pair along with the key.

Storage of plain text for 90 days

This is a huge huge huge deal and perhaps the hardest part of this policy to deal with. As per the policy

Such plain text information shall be stored by the user/organisation/agency for 90 days from the date of transaction and made available to Law Enforcement Agencies as and when demanded in line with the provisions of the laws of the country

It further goes on to add

In case of communication with foreign entity, the primary responsibility of providing readable plain-text along with the corresponding Encrypted information shall rest on entity (B or C) located in India.

Elsewhere the document states

All information shall be stored by the concerned B / C entity for 90 days from the date of transaction and made available to Law Enforcement Agencies as and when demanded in line with the provisions of the laws of the country. In case of communication with foreign entity, the primary responsibility of providing readable plain-text along with the corresponding Encrypted information shall rest on entity (B or C) located in India

We will get into a number of issues this creates later on. But just the basic implementation is impractical. The browser exchanges literally hundreds if not thousands or more encrypted messages with various servers every hour. How is one expected to keep a log of all these exchanges? Both the plain text and encrypted bodies. And you have to store the keys as well since these keep on changing. How will this work with mobile apps ? Most internet services and mobile app servers are based abroad. So the entire burden of responsibility comes down on the lay user residing in India. What if the mobile is stolen and the government asks for that data in less than 90 days since it was generated? What if you delete some data (eg. messages) after you had exchanged them? What about applications that use different protocols for encryption? This is a complete nightmare. Lay users will simply not be able to even comprehend what needs to be done and how, forget actually taking on the responsibility to implement this. The only practical way out is to stop using computers and modern smart phones and go back to the PCs of the 80s without the internet and old Nokia phones which maintain an address book and an ability to make calls.

Service provider registration

Service Providers located within and outside India, using Encryption technology for providing any type of services in India must enter into an agreement with the Government for providing such services in India. Government will designate an appropriate agency for entering into such an agreement with the Service provider located within and outside India. The users of any group G,B or C taking such services from Service Providers. are also responsible to provide plain text when demanded.

There are literally thousands of websites and mobile app services that use encryption. Are they supposed to enter an agreement with the Government of India?

The policy further goes on to add in a separate section

Registration: All vendors of encryption products shall register their products with the designated agency of the Government. While seeking registration, the vendors shall submit working copies of the encryption software / hardware to the Government along with professional quality documentation, test suites and execution platform environments. The vendors shall work with the designated Government Agencies in security evaluation of their encryption products. Complete confidentiality will be maintained in respect of information shared by the vendors with designated agency. The vendors shall renew their registration as and when their products are upgraded. Mass use products like SSL / TLS are exempted from registration.

This is just ridiculous. What an encryption product is is not clear. SSL/TLS are classfied as products (they are not). Such vendors shall submit working copies of software / hardware to government along with professional quality documentation, test suites and execution platforms? So some small startup somewhere buys a piece of helpful hardware to help with the encryption, it now has to buy another one just to deposit it with Government of India? Assuming the GOI is capable enough to accept such software/hardware, the startup will now need to provide detailed documentation. Assuming one gets past that, anyone who has the slightest confidence that the government has the necessary resources and skills to deal with thousands of such submissions may please raise their hand. For a government claiming to be pro-startups, its hurting them hard.

Please note: SSL/TLS are exempted from registration, not from the onerous requirement of storing 90 days of plain text, encrypted data and the session keys

The document further goes on to state

The Government will notify the list of registered encryption products from time to time, without taking responsibility for security claims made by the vendors.

So even after this, each user is left to sift through a list of "registered encryption products" to figure out if their preferred web site or mobile app may be used in India. Every traveler to India will need to do the same as well when in India.

Lack of a legal framework

The document adds

Government reserves the right to take appropriate action as per Law of the country for any violation of this Policy.

Ummm right. Under what IPC will you charge a user for deleting a Whatsapp message? Where is the legal framework which tells the citizens of the country what maximum jail time or penalties they might face if found in violation? What if the phone got stolen? Does that make his penalty any less? Where is the legal framework for prosecuting violations of this policy? Or the clarity if one does exist.

Promotion of R&D in cryptography

With a view to promote the R&D into cryptography the document states

Research and Development programs will be initiated for the development of indigenous algorithms and manufacture of indigenous products for Encryption, hashing and other cryptographic functions. These will be carried out by Public and Private Sector / Government Agencies and Academia. Continuous intensified R&D activities in the niche areas of technical analysis and evaluation of Encryption products will be strengthened.

Nice sounding. But the entire tenor of the document is meant to make encryption so onerous that people will run a thousand miles away from it. This document essentially screams "DO NOT ENCRYPT". In such an environment one wonders why the government actually needs to promote indigenous algorithms and products when it intent seems to be strongly focused on completely destroying such a market (and the IT, BPO and Knowledge Processing industries as stated later).

Algorithms to be used

In the annexure the policy prescribes the algorithms to be used.

Symmetric Cryptographic/Encryption products with AES, Triple DES and RC4 encryption algorithms and key sizes up to 256 bits are prescribed by the Government for use for protecting information by stakeholders.

One wonders how much the government even hopes to support encryption when one of the three suggested algorithms has been explicitly proscribed by the Internet Engineering Task Force (IETF) under RFC 7465 from using it within TLS clients. Clearly the government seems to be unaware of what is happening on the security front and is suggesting algorithms that the Internet widely now believes to be insecure. The least they could have done was at least drop RC4. While not inconvenient, this does make one wonder how well the government is conversant with the happenings in the IT industry.

Disingenious Objectives

The document in its preamble lays down its vision, mission, and objectives. And yet after reading the rest of the document, one wonders if the government even remotely understood that its policy is almost entirely inconsistent with what it set out to do. Thus either it was not entirely honest in setting out its goals properly, or it did a rather shoddy job by actually undermining these very goals. Let us take a look at the goals themselves.

Vision

To enable information security environment and secure transactions in Cyber Space for individuals, businesses, Government including nationally critical information systems and networks.

Mission

To provide confidentiality of information in cyber space for individuals, protection of sensitive or proprietary information for individuals & businesses, ensuring continuing reliability and integrity of nationally critical information systems and networks.

Objectives

To synchronize with the emerging global digital economy / network society and use of Encryption for ensuring the Security / confidentiality of data and to protect privacy in information and communication infrastructure without unduly affecting public safety and National Security.

To encourage wider usage of Digital Signature by all entities including Government for trusted communication, transactions and authentication.

To encourage the adoption of information security best practices by all entities and Stakeholders in the Government, public & private sector and citizens that are consistent with industry practice.

I don't think a further discussion is really required as to the incogruity between the goals as set out and the implementation directives.

Implementation difficulties with this policy

Lets just run through the number of issues this creates.

What does it mean to store plain text information. Does it need to be stored as a plain text? Many mobiles and computers have fully encrypted file systems or fully encrypted databases. Will storing plain text information on such file systems or databases be allowed ? If not, will organisations and end users need to change to unencrypted file systems thus substantially compromising their security?
Continuing from above, many users (including yours truly) use full disk encryption as a mechanism to guard against accidental data leakage in case of mobile/laptop theft, loss etc. Will end users have to give up on that security? And will theft or loss of a mobile / laptop now mean someone else can easily access all the data?
If a service provider is registered, does it absolve the end user citizen from maintaining encryption logs? It does not seem so.
It seems a new market is being unintentionally created for transparently logging plain text data, encrypted text and keys used for encryption every time encryption is used for storage or transmission. This could be done at various levels
1. End users - Clearly end users cannot hope to maintain a log of all this stuff. They are more likely to give up using computers and start using and the pure mobile handsets (non smartphones) thus reversing any attempts at increasing digitisation.
2. Applications - There are again literally thousands if not hundreds of thousands of desktop and mobile applications. Does each one need to be modified now? Again exceptionally unlikely
3. Operating systems - Perhaps Windows and Android and iOS could be enhanced. But will they? And really, why should they? And that will only cover those scenarios where such encryption is done using system level libraries. Rest of the apps are still on their own.
Citizens will not be able to use web services and mobile apps that are not compliant. I mean if I edit and save an email 20 times before sending it from a mail service which stores all emails as encrypted, would an extreme interpretation mean web mail client has to save 20 triplets of plain text, encrypted text and key used for encryption? Or at least plain text and encrypted pairs? Unless the internet application or mobile application does it, no end user will be able to use it for fear of being termed a criminal
There's a deep conflict with good practices and internationals standards. As an example PCI standards. They have a substantial focus on encryption including not storing data in plain text and using random keys and securely encrypting any key information that must be stored. A credit card processor could not find a way to obey the law of the land and the mandatory requirements of secure processing as laid down by the industry. There's no way to run that business within the territorial boundaries of India.
Keys are often stored in Hardware Security Modules (HSMs) for storing them securely. No HSM manufacturer worth his salt will ever condescend to create an HSM that complies with the directives in this policy simply because it is incredibly stupid to do so. Thus enterprises would not be able to use HSMs. Even normal key storage itself would need to be extraordinarily insecure to be consistent with this policy.
I think the general extrapolation of the last two points will suggest that software engineers will need to throw out of the window all known best practices for ensuring data security and privacy, and make sure data is fully open or at least open enough to be just one step away from disclosure as and when required (which also makes it far far more easily accessible to hackers)
Assuming one does log the data, how does one ensure its availability. What happens in case of disk crash? Do mobiles now need to have RAID 5 storage?
What happens with cloud based systems like google chrome? There is little if any storage on the local systems. Do such OS's now require to be modified to cater to the encryption policy?

Political implications

There are some other serious implications in terms of required professional confidentiality (eg. with doctors and lawyers), having secure protected communications with say a spouse, whistleblowing activities, communication with journalists, political dissidence et. al. These are all communications that must be protected within any self respecting democracy.

In addition there is the very basic fundamental right citizens have to privacy.

For a government which claims to promote digitisation, it is setting out to impose a huge cost on the same. Is the policy consistent with the ambitions?

This is an explosive can of worms that threatens civil rights and encourage movement to a orwelian state.

Basically this policy document justs seems like a really bad dream. However this being a technical blog, this entire issue is not explored or commented upon further.

Commercial implications

As mentioned earlier this policy draft does not live up to its stated goals. As also further pointed out this draft basically destroys any notion of secure data storage and encourages rejecting decades of improvements in data storage best practices. But most importantly this policy if implemented (which it should not ever get there hopefully) will headline India for having a regulatory framework which totally compromises data management best practices.

This will make it hard for businesses to outsource data to Indian companies because their data will now be stored in plain text along with the necessary encryption keys. This could be disastrous for the IT, BPO and all Knowledge Processing industries.

EDIT:

The government has since published the addendum to the policy following the media uproar. PROPOSED ADDENDUM TO THE DRAFT ENCRYPTION POLICY

Since it substantially impacts this post, am documenting my thoughts regarding the same separately as follows.

What is the definition of "The mass use encryption products, which are currently being used in web applications, social media sites, and social media applications"? This is such a large category that it is impossible to imagine those who authored the original document did not think of this class of services. So I suspect this was essentially a knee jerk reaction to the uproar rather than having been thought of in the first place. Also does gmail qualify for the exemption above? Lets presume for a moment it does, does my webmail client as offered by my web host provider qualify? It is unlikely to. So some webmail clients are exempt and some aren't. Does telegram, another app similar to whatsapp qualify? The above definition does not make it clear. If I wrote a similar app for my family, Will it? If not, why not?

In short this clarification makes the whole thing even more murky.
I presume by SSL/TSL based products it means corresponding web sites. Why call them products? And there are tons of other apps which may not qualify. eg. a health care management system, or a logistics routing application. One might argue that if they are password based, then they should also be exempt. Assuming all password based services are exempt then what is the whole policy for? What non password based services need such a disturbing policy around it? And if all password based services are not exempt, then all the issues I've pointed to are still relevant. Thus leading to a scenario where a large number of internet based services and mobile apps having to deal with a real ton of issues to sort through.
What about outsourced IT, BPO and other backoffice services to India ? Do they qualify for exemption. Many of them will not meet the addendum requirement for exemption. And yet they will deal with very important sensitive data. Will the data owners continue to allow their Indian outsourcing partners to continue dealing with their data?

This whole addendum sounds like a knee jerk reaction to a really deficient policy in the first place. Seems to me this policy was a very very bad idea to begin with, something that the government simply did not think through. And now it is trying to put up a face saver using an addendum. Sadly that only deals with the issues raised in the popular media. It still does not address many of the implementation issues I have referred to. Perhaps the policy makers simply have very little clue about what is it that they are attempting to regulate. And leaving us very very confused, and very very scared.

Have we misunderstood DVCS and git / hg ?

2013-10-04T04:00:00+05:30

Yesterday github was at least partially inaccessible due to a DDOS attack.

We are working to mitigate a large scale DDoS attack.
— GitHub Status (@githubstatus) October 2, 2013

And there were some interesting tweets appearing. I just show a few that appeared at the top of my twitter search results. But there were many many more.

So github goes down and apparently collaboration becomes unpossible? Guess the ‘D’ in DVCS went unnoticed by many.
— Christopher Darroch (@chrisdarroch) October 3, 2013

If you can't work when GitHub is down, you are misunderstanding the D in DVCS.
— ¯\_(ツ)_/¯ (@SeanTAllen) August 31, 2013

How many times must github go down before people grok DVCS?
— Robert Pitts (@rbxbx) August 15, 2013

Some clown tried to DDoS @github? I hope they know Git is a DVCS.
— Postmodern (@postmodern_mod3) March 23, 2013

Perhaps the most hard hitting of them all

If Github being down stops your entire development process, you're doing DVCS wrong.
— Greg Hurlman (@ghurlman) August 15, 2013

There is clearly a mismatch between perceived expectation of DVCS workflows and our actual experiences. I did start wondering, was it our expectations that were inflated, or were we (as a community) not using git correctly.

Frankly all the work a developer does by himself/herself is not affected. One can continue to commit, branch, merge etc. at will. What is impacted is the collaboration with other developers. So the "D" in the "DVCS" has at least partially lived up to its promise. Since each developer can still do a lot of work even as he/she is distributed.

The event allows us to reflect on whether we could use git a bit differently, and perhaps suggest some opportunities for improvement. However, I think the major issue is perhaps our expectation of the D in DVCS is inflated. Some of the reasons are related to the state of technology. Some others are inherent in the workflow. Here's why

A truly distributed (federated) workflow may not require github at all. However it will require all repositories to be able to communicate with each other in a peer to peer fashion. That would mean it would require us to open up our development workstations to be accessible to other development workstations either directly or via some protocol like bittorrent. Is it really practical in the age of don't expose your development machine, private IP addresses, proxy servers, firewalls etc.? I suspect not. We are not yet ready for a peer to peer communication over the internet between our development nodes. To the extent that developer nodes can communicate with each other, some amount of collaboration remains feasible even if the central github/origin repo goes down.
Assuming these nodes can communicate with each other, we don't ship different versions of software from different development nodes. We typically have a reference version that is ready to test, ship etc. We need to be able to point to a branch or a tag and say thats the current state of my software under development. Granted that in this model, the reference node may not be able to reflect the current state of software development in realtime. In the sense, some changes and commits on the development nodes still haven't made it to the reference node (often called the origin). But the organisation as a whole treats the development nodes as work in progress and the origin as the current state of (partially) completed work. If we work in a fully federated model where will we have the reference version? If we annoint one dev node as the reference node - then that node is in effect the equivalent of github, and should that node be inaccessible, many collaborative activities will stop. And if we do not annoint one particular node as the first amongst equals, there are things we need to figure out from an overall business perspective, how to make sense out of a bunch of disparate nodes, and compute the current state of software as a function of all the collective development nodes.
While I cannot readily imagine a good solution to the issue I just described, let us imagine, we did find a good solution. We would have an issue of different views of history. Different developers would see history differently. They would see the software being developed with a differently ordered commits. Plus given n-to-n communication, the complexity of doing rebases and merges would shoot up drastically. Are we sure we have understood what kind of complexity it would require us to deal with even if the technology to do so was feasible? I suspect not.

An analogous situation would be where sales staff synced up their notebooks with a central database in the morning, and went off to do their tasks independently during the day in remote areas without any connectivity, came back in the evening and synced up their notebooks once again with the central database. This is a similar scenario to how we use git. Clearly, issues such as resolving conflicts would remain, yet the offline model allows the sales staff to do "most" of their work except for the coordination that might be required but not feasible due to lack of connectivity.

Is there a way to use git better than how we've been using it so far? Perhaps yes. Perhaps we could have multiple central databases (using the analogy above) in a master slave model or multi master model. Now I dont think we have the ready technology with DVCS to use the multi master model where different masters can readily synchronise with each other without any manual intervention (I could be wrong, but I make that assumption). In which case a master slave model would work better. So perhaps we could have a master/slave arrangement between multiple "origins". And the master would near instantaneously push changes out to the slaves. These would be hosted by different git providers. And if the master was to be not accessible, we could always promote another slave to become the master, and have the original master re-join the cluster as a slave when it is back online. This might require a bit of tooling but does not seem like a very difficult task.

Perhaps, I haven't fully imagined how people would be doing it "right" when they design their workflows in a manner a github downtime does not impact them at all. And would be keen to learn or get answers. But at least for the moment, I think we need to temper our expectations. Based on the real constraints. It does not mean distributed VCS has failed, but instead has lived up to promises that it reasonably could. And the rest was just us letting our expectations go a little further.

Partially applied functions and decorators in python

2013-10-03T19:00:00+05:30

This is not a quick tutorial. It will tell you all you need to know about decorators, but slowly build ups the necessary theoretical and technical background in terms of partially applied functions, closures and lexical scoping before eventually discussing decorators.

Partial application of functions

As per the wikipedia article on Partial Application,

In computer science, partial application (or partial function application) refers to the process of fixing a number of arguments to a function, producing another function of smaller arity. Given a function f : (X × Y × Z) → N , we might fix (or 'bind') the first argument, producing a function of type partial(f) : (Y × Z) → N . Evaluation of this function might be represented as f_partial(2, 3). Note that the result of partial function application in this case is a function that takes two arguments.

Thus if you have a function f with arguments x₁, x₂, ..x_n, a partial application technique would allow you to

Create a partially applied function f_partial specifying only a subset of the arguments x₁, x₂, ..x_n. Assuming them to be the first two, it would be f_partial = f(x₁, x₂)
Use the partially applied function with the remainder of the arguments to get the same result as one would have got from the original function. Continuing with the above assumption that would allow you to do f_partial(x₃, x₄, ..., x_n)

Partial application in python

Let us start with a simple function add

def add(a,b) :
    return a + b

print("Adding 2 and 3 results in {}".format(add(2,3)))

Adding 2 and 3 results in 5

Let us look at two different ways of partial application in python

Partial application using nested functions

def add(a) :
    def add_a_to(b) :
        return a + b
    return add_a_to

print("Adding 2 and 3 results in {}".format(add(2)(3)))
add2 = add(2)
print("Calling add with 2 alone returns {}, a {} with the name '{}'".format(
    add2, type(add2), add2.__name__))

Adding 2 and 3 results in 5
Calling add with 2 alone results in <function add_a_to at 0x33eb848> which is a <type 'function'> with the name 'add_a_to'

Note the following :

In this implementation we changed the calling syntax from add(2,3) to add(2)(3)
This is because we now make two function calls. The first call is add(2) which results in a function being returned called add_a_to . We then call add_a_to(3). However since we just chain the function calls together, it is succinctly written as add(2)(3)
The implementation technique of nesting one function inside another is often referred to as nested functions. It requires structural modification of the function definition itself.

Partial application using functools.partial

This technique does not require us to rewrite the function differently. It does allow us to first pass any arbitrary subset of arguments (eg. the second argument ie, 3 above), create a partially applied function using functools.partial and then use that to subsequently invoke the function with the remainder of the arguments. For reference see functools.partial in python documentation

from functools import partial

def add(a, b):
    return a + b

added_b_as_3 = partial(add, b=3)
print("add partially applied with 2nd argument as 3 results in {}".format(
    added_b_as_3))
print("Adding 2 to 3 using partial application results in {}".format(
    added_b_as_3(2)))
print("Partial applications using functools.partial can be inlined, " +
    "though unlikely to be used so.")
print("2 + 3, ie. partial(add,b=3)(2) = {}".format(partial(add,b=3)(2)))

add partially applied with 2nd argument as 3 results in <functools.partial object at 0x32c8470>
Adding 2 to 3 using partial application results in 5
Partial applications using functools.partial can be inlined, though unlikely to be used so.
2 + 3, ie. partial(add,b=3)(2) = 5

Why is partial application useful ?

Partial application helps us implement spatial and temporal separation.

Spatial separation : Partial argument lists can be specified at different places in the code. This might be convenient from just a structuring perspective or perhaps from the perspective of lexical scoping.
Temporal separation: Partial argument lists can be partially applied at different times in the execution of the code. This is convenient say when some arguments are to be included in the code itself, some are to be partially applied only at the program startup configuration time and some others might be required at runtime.

A tour of closures and lexical scope

Typical usage of decorators leverages the capabilities afforded by closures and lexical scope. As a result let us take a quick side tour of these.

According to the wikipedia page on Closure

In programming languages, a closure (also lexical closure or function closure) is a function or reference to a function together with a referencing environment—a table storing a reference to each of the non-local variables (also called free variables or upvalues) of that function. A closure—unlike a plain function pointer—allows a function to access those non-local variables even when invoked outside its immediate lexical scope.

Also on the page of Scope : Lexical vs. dynamic scoping we can find the words

In lexical scoping (or lexical scope; also called static scoping or static scope), if a variable name's scope is a certain function, then its scope is the program text of the function definition: within that text, the variable name exists, and is bound to the variable's value, but outside that text, the variable name does not exist

Let us explore this in the context of python programs

var1 = 3
var2 = 4

def foo() :
    var2 = 5
    print("Value of var2 is {}".format(var2))
    print("Global var1={}, var2={}".format(globals()["var1"], globals()["var2"]))
    print("Locals are {}".format(locals()))
    var1 = 9
    print("Value of var1 is {} and var2 is {}".format(var1, var2))
    print("Global var1={}, var2={}".format(globals()["var1"], globals()["var2"]))
    print("Locals are {}".format(locals()))

foo()
print("Outside the function, value of var1 is {} and var2 is {}".format(var1, var2))

Value of var2 is 5
Global var1=3, var2=4
Locals are {'var2': 5}
Value of var1 is 9 and var2 is 5
Global var1=3, var2=4
Locals are {'var1': 9, 'var2': 5}
Outside the function, value of var1 is 3 and var2 is 4

I would like to highlight the following

Variables declared in the global scope are accessible to a function explicitly via the globals() function
Variables declared within the function scope are accessible to a function either implicitly or explicitly via the locals() function
A global variable redeclared within the function scope results in two variables, one in the global and another in local scope. However the local binding shadows the global binding.
When a function declaration concludes, its scope is over and the local bindings in that function cease to be relevant

Now let us take a look at another program which uses nested functions and will require us to revisit the last point I make above.

var = 5

def outer(arg):
    var = arg
    def inner() :
        print("The value of var is {}".format(var))
    print("local value of var is {}. Now being incremented".format(var))
    var = arg + 1
    return inner

print("The value of the global var is {}".format(var))
nested1 = outer(6)
nested2 = outer(99)
print("The free variables carried by nested1 from outer scope are {}".format(
    zip(nested1.__code__.co_freevars, (c.cell_contents for c in nested1.__closure__))))
print("The free variables carried by nested2 from outer scope are {}".format(
    zip(nested2.__code__.co_freevars, (c.cell_contents for c in nested2.__closure__))))
nested1()
nested2()
print("The value of the global var still is {}".format(var))

The value of the global var is 5
local value of var is 6. Now being incremented
local value of var is 99. Now being incremented
The free variables carried by nested1 from outer scope are [('var', 7)]
The free variables carried by nested2 from outer scope are [('var', 100)]
The value of var is 7
The value of var is 100
The value of the global var still is 5

Now note the following :

var is created in the global scope, its value is set to 5 and remains 5 throughout the program
There is another var created within the scope of the outer function which shadows the global var within the lexical scope of outer.
The local value of var is initialised to the value passed as a parameter to outer. That is the value visible to inner() at declaration time
The local value of var is incremented after the inner function has been declared.
In general the local value of var declared is no longer valid once the function declaration of outer ends, since thats where its lexical scope ends.
However if a nested function is declared and returned from outer (in this case inner), then such a function continues to carry the variables available to it at declaration time. This is exactly what closures are, ie. the function inner closes over the scope of the function outer.
The variable value in the closure is not a snapshot of the value at the point in time the function was declared. It will reflect any changes after the inner function was declared but before the outer function got over
The variable values are passed to the nested function via its _closure_ attribute (variable names via _code_.co_freevars)

To summarise the necessary learnings :

Nested functions help us implement partial application of functions
Any variables declared in the lexical scope of the outer nested functions are available to the inner nested functions even after the outer function declaration is over.
A new local scope gets created whenever a outer nested function is called. Any returned inner functions, as a result multiple outer function invocations, get independent copies of the local scope of each execution of the outer function.

How is partial application useful ?

There are a number of uses of partial applications. Some of these are

Creating specialised functions from general functions

Many a times functions we can easily associate with are but specialised versions of more general functions. Writing more general functions allows is a better way to follow the Do not repeat yourself (DRY) principle. Specialising them can allow for easier readibility when using them, since they can be more intuitive to understand at call site.

A simple generic function might be declared as follows

def exponent_of(base, exponent) :
    return base ** exponent

print("3 to the power 4 is {}".format(
    exponent_of(3,4)))

3 to the power 4 is 81

However a more specialised function might make for easier reading. While you could just as easily use functools.partial, the example below shows partial application using nested functions

def nth_power(exponent) :
    def exponent_of(base) :
        return base ** exponent
    return exponent_of

square = nth_power(2)
cube = nth_power(3)

print("The square of 3 is {}".format(square(3)))
print("The cube of 4 is {}".format(cube(4)))

The square of 3 is 9
The cube of 4 is 64

Temporal separation of function arguments

In some cases, a function may need many arguments, some of which are specified and available at different points in time than others. A usual way of dealing with this is to carry forward all the available arguments to the eventual call site. Sometimes those that are specified early on are rarely if ever modified later. It becomes inconvenient to carry around all these variables in the namespace. An alternative is to create a partially applied function and carry forward only the partially applied function to the eventual call site. eg.

# Conventional method

def query_database(userid, password, query) :
    # do query
    # return results

def bar(userid, password):
    return query_database(userid, password)

def foo(userid, password) :
    return bar(userid, password)

def main(userid, password) :
    # .. lot of code here .. eventually reaching
    foo(userid, password)


# Alternative method
def get_query_agent(userid, password)
    def do_query(query) :
        # do query
        # return results
    return do_query

def bar(querying_func):
    return func(querying_func)

def foo(querying_func) :
    return bar(querying_func)

def main(userid, password) :
    query_agent = get_query_agent(userid, password)
    # .. much further down the line
    foo(query_agent)

Use closures instead of classes

It is not uncommon to see classes with only one significant public method. In many such scenarios, the class constructor is used to specify the arguments of the method, and the method itself is used to perform the desired behaviour. This gives us temporal separation (object construction and method execution) and encapsulation of the arguments. It so happens the same can be done through using closures as well. (Note: although the example below is quite similar to the one above they serve to describe different intents)

# traditional method

class Connection(object):
    def __init__(userid, password):
        self.userid = self.userid
        self.password = self.password

    def execute(sql):
        # execute SQL using userid, password.
        # return results

c = Connection("myuserid", "mypassword")
c.execute("select 'x' from dual;")

# Using closures

def get_connection(userid, password) :
    def execute(sql):
        # execute SQL using userid, password & sql
        # return results
    return execute

c = get_connection("myuserid", "mypassword")
c("select 'x' from dual;")

Aside: Currying vs. Partial Applications

As per the wikipedia article on Currying,

In mathematics and computer science, currying is the technique of transforming a function that takes multiple arguments (or a tuple of arguments) in such a way that it can be called as a chain of functions, each with a single argument (partial application).

It further goes on to define it as follows

Given a function f of type f : (X × Y) → Z , currying it makes a function curry(f) : X → (Y → Z) . That is, curry(f) takes an argument of type X and returns a function of type Y → Z .

With a further view to disambiguate the term from partial applications, it goes on to state that

Currying and partial function application are often conflated. One of the significant differences between the two is that a call to a partially applied function returns the result right away, not another function down the currying chain; this distinction can be illustrated clearly for functions whose arity is greater than two.

Given a function of type f : (X × Y × Z) → N , currying produces curry(f) : X → (Y → (Z → N)) . That is, while an evaluation of the first function might be represented as f(1, 2, 3), evaluation of the curried function would be represented as f_curried(1)(2)(3), applying each argument in turn to a single-argument function returned by the previous invocation. Note that after calling f_curried(1), we are left with a function that takes a single argument and returns another function, not a function that takes two arguments.

Currying thus is just a special case of partial application, where we partially apply the arguments, one argument at a time, in the order in which they are declared. In general currying (especially as defined above) is not a useful notion to attempt to use in python. However you might find it getting used in context of python, and more often than not, it might mean partial application. How would currying actually look like, well for the curious, here's a function that can curry another

def add(a,b,c) :
    return a +  b + c

def curry(func) :
    args = []
    if func.__code__.co_argcount == 0 :
        return func()
    def wrap(arg) :
        args.append(arg)
        if len(args) == func.__code__.co_argcount :
            return func(*args)
        else :
            return wrap
    return wrap

curried_add = curry(add)
print(curried_add(3)(4)(5))

Decorators

Decorators were introduced as a syntactic addition to python language in PEP-318. It is often considered that the name decorators comes from the gang of four book on Design Patterns. The PEP in fact clarifies it is not so (and acknowledges that the name has drawn its share of criticism and perhaps a better name will come up eventually - it did not).

The name 'decorator' probably owes more to its use in the compiler area -- a syntax tree is walked and annotated

I've heard it being discussed that decorators are just nested or partially applied functions. While that is accurate, it misses an important distinction. Although both use the same technique, partial application is usually used to split a task into two. Decorators on the other hand are used to wrap one or more add-on tasks around a core task. So there are many many scenarios where you would use partially applied functions, but you would not want to use decorators.

Let us once again start with the simple add function which we shall subsequently decorate

def add(a, b):
    return a + b

print("Adding 2 and 3 results in {}".format(add(2,3)))

# Note the following.
# we shall contrast this later when function wrapping kicks in
import inspect
print("The specification for add is name={} args={}".format(
                            add.__name__, inspect.getargspec(add)))

Adding 2 and 3 results in 5
The specification for add is name=add args=ArgSpec(args=['a', 'b'], varargs=None, keywords=None, defaults=None)

We shall now decorate this function with a tracer. A function which can print entry and exit to the desired function

def trace(func) :
    def wrapper(*args, **kwargs) :
        print("Entering {} with arguments {} {}".format(
                                        func.__name__, args, kwargs))
        ret = func(*args, **kwargs)
        print("Leaving {} with result {}".format(func.__name__, ret))
        return ret
    return wrapper

@trace  # <--- This is how a decorator is applied
def add(a, b) :
    return a + b

print("Adding 2 and 3 results in {}".format(add(2,3)))

import inspect
print("The specification for add is name={} args={}".format(
                            add.__name__, inspect.getargspec(add)))

Entering add with arguments (2, 3) {}
Leaving add with result 5
Adding 2 and 3 results in 5
The specification for add is name=wrapper args=ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)

Note that applying the trace function as a decorator using "@trace" did not just create a new function. It did create a new wrapped function, but also replaced the function in the namespace for the name "add" with the new wrapped function. The original function add is no longer available to be directly accessed from the namespace.

Decorators are just the outer function in a nested function. The decorated function is essentially a closure which retains access to the variables as registered by the decorator in its local namespace and the rules of how such variables are lexically scoped but accessible by closures continue to apply.

The function name is now changed if you were to inspect it. So add._name_ now returns wrapper and not add. That can be corrected by using the @wraps decorator from the functools module.

from functools import wraps

def trace(func) :
    @wraps(func) # <-- Applying the wraps decorator helps in the wrapped function retaining the original name
    def wrapper(*args, **kwargs) :
        print("Entering {} with arguments {} {}".format(
                                        func.__name__, args, kwargs))
        ret = func(*args, **kwargs)
        print("Leaving {} with result {}".format(func.__name__, ret))
        return ret
    return wrapper

@trace
def add(a, b) :
    return a + b

print("Adding 2 and 3 results in {}".format(add(2,3)))

import inspect
print("The specification for add is name={} args={}".format(
                               add.__name__, inspect.getargspec(add)))

Entering add with arguments (2, 3) {}
Leaving add with result 5
Adding 2 and 3 results in 5
The specification for add is name=add args=ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)

Here thankfully the name of the wrapped function is now add should anyone care to inspect the function. The argument specification still remains different. And it should if trace has to be a generic function which can trace all functions, not just those which have two parameters.

The takeaway is that because decorators implement functions in place, some information about the functions can get lost to function inspectors. The same would not happen if you chose to explicitly wrap a function without using the decorator syntax as in the following. There are other effects as well eg. you can no longer easily unit test the original undecorated function. If the decorator introduces side effects or say changes the return value you have to deal with it in your unit tests.

The code below once again demonstrates the scenario where trace is used as a wrapper, without actually using it as a decorator

def trace(func) :
    def wrapper(*args, **kwargs) :
        print("Entering {} with arguments {} {}".format(
                                        func.__name__, args, kwargs))
        ret = func(*args, **kwargs)
        print("Leaving {} with result {}".format(func.__name__, ret))
        return ret
    return wrapper

def add(a, b) :
    return a + b

# Not traced
print("Adding 2 and 3 results in {}".format(add(2,3)))

# traced add
print("Adding 4 and 5 results in {}".format(trace(add)(4,5)))

import inspect
print("The specification for add is name={} args={}".format(
                            add.__name__, inspect.getargspec(add)))
print("The specification for trace is name={} args={}".format(
                            trace.__name__, inspect.getargspec(trace)))

Adding 2 and 3 results in 5
Entering add with arguments (4, 5) {}
Leaving add with result 9
Adding 4 and 5 results in 9
The specification for add is name=add args=ArgSpec(args=['a', 'b'], varargs=None, keywords=None, defaults=None)
The specification for trace is name=trace args=ArgSpec(args=['func'], varargs=None, keywords=None, defaults=None)

Decorators with arguments

Once one starts using decorators, pretty soon one wants to parameterise their behaviour. For example we might want to decide whether to log the entry/exit arguments and return values at the point where we apply the trace decorator to each function. I have seen people struggle with decorators when it gets to using decorators with arguments. But if one understands the basics of how decorators work, it is actually very easy to figure it out.

Part of the confusion stems from what exactly is a decorator. For the purposes of this discussion a decorator is a function which takes exactly one argument, a function, and returns exactly one another, which is also a function. If you want to parameterise a decorator, the way (using a term used by the PEP) is to create a decomaker. A decomaker is a function which takes a bunch of arguments and return a decorator.

Let us see the parameterised implementation of trace below. Under the definitions we just laid down, trace is no longer a decorator. It is a decomaker. ie. it takes a bunch of arguments and returns a decorator. That decorator in turn decorates the method add.

For the remainder of this post, I shall use decorator as a generic placeholder for both decorators and decomakers (as generally most writings in python do). But hopefully the context should allow you to disambiguate as necessary.

from functools import wraps

def trace(trace_arguments = False) :
    def decorator(func) :
        @wraps(func) # <-- how the wrapped function retains the original name
        def wrapper(*args, **kwargs) :
            if trace_arguments :
                print("Entering {} with arguments {} {}".format(
                                                func.__name__, args, kwargs))
            else :
                print("Entering {}".format(func.__name__))
            ret = func(*args, **kwargs)
            if trace_arguments :
                print("Leaving {} with result {}".format(func.__name__, ret))
            else :
                print("Leaving {}".format(func.__name__))
            return ret
        return wrapper
    return decorator

@trace(True)
def add(a, b) :
    return a + b

@trace()
def subtract(a, b) :
    return a - b

print("3 plus 5 is {}".format(add(3,5)))
print("5 minus 3 is {}".format(subtract(5,3)))

Entering add with arguments (3, 5) {}
Leaving add with result 8
3 plus 5 is 8
Entering subtract
Leaving subtract
5 minus 3 is 2

Composing decorators

Unsurprisingly decorators can be composed. After all they are just functions which accept one function and return another function, so composing should be trivial. This is often popularly referred to as chaining.

Here's a example

from functools import wraps

def trace(trace_arguments = False) :
    def decorator(func) :
        @wraps(func)
        def wrapper(*args, **kwargs) :
            if trace_arguments :
                print("Entering {} with arguments {} {}".format(
                                                func.__name__, args, kwargs))
            else :
                print("Entering {}".format(func.__name__))
            ret = func(*args, **kwargs)
            if trace_arguments :
                print("Leaving {} with result {}".format(func.__name__, ret))
            else :
                print("Leaving {}".format(func.__name__))
            return ret
        return wrapper
    return decorator

# using a global for brevity
function_calls = 0

def track(func) :
    @wraps(func)
    def wrapper(*args, **kwargs):
        global function_calls
        function_calls = function_calls + 1
        return func(*args, **kwargs)
    return wrapper


@track
@trace(True)
def add(a, b) :
    return a + b

print("Function calls so far {}".format(function_calls))
print("3 + 5 = {}".format(add(3,5)))
print("2 + 7 = {}".format(add(2,7)))
print("Function calls so far {}".format(function_calls))

Function calls so far 0
Entering add with arguments (3, 5) {}
Leaving add with result 8
3 + 5 = 8
Entering add with arguments (2, 7) {}
Leaving add with result 9
2 + 7 = 9
Function calls so far 2

Use of function decorators

Function decorators are frequently used and are quite helpful. I doubt if I can create a comprehensive set of use cases, but will list some of the ones where I've observed decorators are useful.

Applying aspects to functions

Decorators are frequently used as an implementation technique for some facets of Aspect Oriented Programming. The trace decorator (or decomaker) I described above implements a tracing aspect. You could imagine other decorators which for example update global counters (to keep track of how many web requests have been served), authorisation decorators for implementing ACLs, or memoization ie. to cache results computed earlier and serve them if the function was invoked with the same arguments again. eg. functools.lru_cache is a memoization decorator.

Autoregistration of function handlers

Many web frameworks allow you to configure the url route around its handler function using a decorator as below.

@route("/show")
def show(request) :
    # process the request and return the generated HTML

In this case there is a global routes repository list which contains the routes, and their handler functions. In the above example "/show" is the url_prefix that if matched should result in the show handler function being called. The routes engine is matches an incoming request to the routes in the list, and if a match is found, then the corresponding handler function is called giving it the request as an argument. Without decorators, you would need a separate place where all the routes and their handlers could be configured (sometimes called wiring up). With decorators like above, this configuration can be done at the site of each function declaration. (The part where the runtime discovers all such functions and calls them once in order to actually perform the registration is beyond the scope of this post). We could use decorators similarly to register event listeners etc.

Markers, metadata and housekeeping

Sometimes we want to mark a function as having a specific characteristic so that it can be processed differently at runtime. eg. celery marks each task handler using @app.task decorator. One might want to assign more metadata to the function. Decorators can also be used for basic housekeeping (eg. the wraps decorator seen earlier). Python itself requires us to use @staticmethod and @classmethod decorators to mark appropriate methods on a class.

Class decorators

Class decorators were introduced in Python 2.6 using PEP-3129. A class decorator is a function which gets a python class as an input. It performs the necessary decoration activities on the class and returns the decorated class. Much of what class decorators achieve can also be achieved through metaprogramming, and more often than not, being able to write good class decorator requires a decent understanding of the class meta model.

The following example demonstrates how a class decorator can be used to substitute a class constructor to print a statement and then eventually call the original class constructor.

def trace(class_):
    original_init = class_.__init__
    def __init__(self, *args, **kws):
        print("Instantiating an object of class {}".format(class_.__name__))
        original_init(self, *args, **kws)
    class_.__init__ = __init__
    return class_

@trace
class Foo(object):
    def __init__(self, value):
        self.value = value

foo = Foo(5)
print("The value of foo is {}".format(foo.value))

Instantiating an object of class Foo
The value of foo is 5

A screencast overview of python enums as introduced by PEP 435

2013-09-28T14:00:00+05:30

Earlier in the week, I made my first screencast to give an overview of PEP 435 which introduces enums to the python standard library.

The code below is the set of python notes I had prepared as I made the screencast.

# What are enumerations ?
# A series of immutable discrete arbitrary values
# of some type
# may or may not have a semantic meaning

# Why need them as a separate construct ?

# You can represent them as constants with values eg. ints
# But what does GREEN * 2 mean ?
# And if GREEN = 2, when you print / log GREEN, you expect to see GREEN not 2
# And sometimes you want to treat all the values as a set eg. days of week

# PEP 435 is the formal python document introducing enums
# Implemented in 3.4. You will need a python 3.4 runtime

# How are enumerations constructed ?

from enum import Enum

# There are two ways to construct enums. The first is a functional API. eg.

Direction = Enum('Direction','North East South West')

# Direction is an enumeration
# It has 4 members

list(Direction)

# Each member has a name and an associated value
# This is more apparent when you render the __repr__ of the member

list(Direction)[0]

# in this case the values were automatically provided.
# When values are automatically generated they are integers starting from 1

# You could also pass the member name and value as 2 tuples
StrDirection = Enum ('StrDirection',(("North", "N"),("East","E"), ("West", "W"), ("South", "S")))
list(StrDirection)

# Or as a dict
AngleDirection = Enum ('AngleDirection',{"North": 90,"East":0, "West": 270, "South": 180})
list(AngleDirection)

# Note: while Direction is an iterable in definition order,
#       you can't pick up the zero'th element eg. Direction[0]
#       ie. you cannot index by position
# However Direction.North and Direction["North"] are both valid

Direction.North
Direction["North"]

# Curiously a enum can also be looked up using a value by using the call operator

Direction(1)

# A key property of an enum is that its members are hashable

# Internal structure.
# Every Enum type has a __members__ attribute of the type OrderedDict

Direction.__members__
list(Direction.__members__.keys())
list(Direction.__members__.values())
list(Direction.__members__.items())

# enums are compared by identity and can be checked for equality
# however you cannot perform ordered comparisons

#   Aliases are not returned when iterating
# While two members cannot share the same name, they can share the same value (aliases)
# There is only one member that is shared in case of aliases

print(list(name for name, member in Colour.__members__.items() if member.name != name))

# An alternative way of defining an enum is a class
# Enum classes can have instance and classmethods

class Timezone:
    def __init__(self,offset,dst) :
        self.offset = offset
        self.dst = dst
    def earlier_than(self, other) :
        # ignoring dst
        return self.offset < other.offset
    def __eq__(self,other) :
        return self.offset == other.offset and self.dst == other.dst
    def __str__(self) :
        return "TZ({},{})".format(self.offset, self.dst)
    def __repr__(self) :
        return "TZ({},{})".format(self.offset, self.dst)

class Timezones(Enum):
    IST = Timezone(330, False)
    GMT = Timezone(0, False)
    UTC = Timezone(0, False)
    EST = Timezone(-300, False)
    @classmethod
    def all_zones(cls):
        #   Aliases are not returned when iterating
        return list(n.name for n in cls)
    @classmethod
    def aliases(cls):
        return [(name, member.name) for name, member in
                cls.__members__.items() if  member.name != name]
    def offset(self) :
        return  self.value.offset


for tz in Timezones :
    print(tz)

Timezones.all_zones()
Timezones.IST.offset()

list(Timezones)
list(Timezones.__members__.keys())
list(Timezones.__members__.values())
id(Timezones.GMT)
id(Timezones.UTC)

Timezones.IST == Timezones.GMT

Timezones.GMT == Timezones.IST

Timezone.UTC < Timezone.IST

# An enum which does not have any members can be subclassed. One which has cannot be

# So far the values per se have not been important (except to distinguish between distinct / aliased members)
# However in some cases it is important, and for that a special case Enum ie. IntEnum is defined

from enum import IntEnum

class Education(IntEnum):
    Kindergarten = 1
    Primary = 2
    Secondary = 3
    HigherSecondary = 4
    Graduate = 5 # elsewhere eg. US, this would be under graduate
    PostGraduate = 6 # and this would be called graduate
    Doctorate = 7

# This is an example where the value (and its ordering particularly is considered important)
# Now operators relevant to the semantics of the mixed in type (in this case int) can be used

Education.Secondary > Education.Primary

# Note that this opens up semantic issues of equivalence across enum types. eg.

class Colour(IntEnum):
    Red = 1
    Green = 2
    Blue = 3

Education.Secondary == Colour.Blue

# Thus you can choose to bring in any value types consistent with their appropriate semantics
# Note that this could lead to unexpected results eg.

list(range(Education.Graduate))

# However IntEnum is just a special case of being able to mix in values of any types ie.

class IntEnum(int, Enum) :
    pass

# So use value types that are semantically consistent

"Exploring Java8 Lambdas. Part 1"

2013-02-26T09:00:00+05:30

Lambda functions and closures are coming to Java 8. This has been a project that has been in progress for a while and recently I took the opportunity to take it for a test drive.

Information :

Some of the useful documentation for the project may be found at :

Project Lambda Home page.
State of the Lambda : A old paper (Dec 2011) describing the then status of lambda implementation. While the paper continues to be quite relevant, it may not describe a whole lot in terms of specifics of how to use the features.
Lambda FAQ : This is a must read FAQ. In many ways it covers many of the same topics already covered in the state of the lambda, but describes the same in a greater detail and in a Q&A format which makes it easy to consume the information one small answer at a time. I refer to it frequently in this post.
lambda-dev mailing list

Motivation :

Let us understand the motivation and goals of Java 8 lambdas. This is particularly important since we often take a look at a new piece of software and attempt to map it to something we already know. That may or may not be the appropriate thing to do in a given context. I focus on the motivation so we can understand what the authors set out to do, which will help us better evaluate the implementation and how well it addresses its goals

The page Why are lambda expressions being added to Java? describes the primary motivation

Lambda expressions (and closures) are a popular feature of many modern programming languages. Amongst the different reasons for this, the most pressing one for the Java platform is that they make it easier to distribute processing of collections over multiple threads. Currently, lists and sets are typically processed by client code obtaining an iterator from the collection, then using that to iterate over its elements and process each in turn. If the processing of different elements is to proceed in parallel, it is the responsibility of the client code, not the collection, to organise this.

In Java 8, the intention is instead to provide collections with methods that will take functions and uses them, each in different ways, to process their elements. The advantage that this change brings is that collections can now organise their own iteration internally, transferring responsibility for parallelisation from client code into library code.

On the page Are lambda expressions objects? we come across the following quote

To understand the situation, it is useful to know that there are both short-term goals and a longer-term perspective for the implementation in Java 8. The short-term goals are to to support internal iteration of collections, in the interests of efficiently utilising increasingly parallel hardware. The longer-term perspective is to steer Java in a direction that supports a more functional style of programming. Only the short-term goals are being pursued at present, but the designers are being careful to avoid compromising the future of functional programming in Java, which might in the future include fully-fledged function types such as are found in languages such as Haskell and Scala.

A useful line is to be found in one of the posts in the lambda-dev mailing list referred to above ie. A peek past lambda

Lambda is the down-payment on this evolution, but it is far from the end of the story. The rest of the story isn't written yet, but preserving our options are a key aspect of how we evaluate the decisions we make here.)

Whither Java Collections Framework

As one would soon realise, much of the newer features being added are not being added directly to the classes you currently know as the Java Collections Framework. There's a new interface Stream and helper class Streams thats at the center of these changes. The rationale for Stream is explained as follows in Where is the Java Collections Framework going?

Analysis of the usage of the Java collections shows one pattern to be very common, in which bulk operations draw from a data source (array or collection), then repeatedly apply transformation operations like filtering and mapping to the data, often finally summarising it in a single operation such as summing numeric elements. Current use of this pattern requires the creation of temporary collections to hold the intermediate results of these transformations. However, this style of processing can be recast to a pipeline using the well-known “Pipes and Filters” pattern, with significant resulting advantages: elimination of intermediate variables, reduction of intermediate storage, lazy evaluation, and more flexible and composable operation definitions. Moreover, if each operation in the pipeline is defined appropriately, the pipeline as a whole can often be automatically parallelised (split up for parallel execution on multicore processors). The role of pipes, the connectors between pipeline operations, is taken in the Java 8 libraries by implementations of the Stream interface; examining this interface will clarify how pipelines work.

Another page Why are Stream operations not defined directly on Collection? documents the rationale behind why many of the stream methods are not defined directly on the Collection interface.

Implementation used in this post

I have used the b78 version of the Java™ Platform, Standard Edition 8 Early Access with Lambda Support from here. It could happen that some of the api methods I refer to continue to evolve. The JDK api docs for this particular version can be found here

Typical pipeline structure.

Let us explore the typical pipeline structure. For a given collection, convert it into a stream, perform a set of transformations on the same and finally accumulate the results into a Collector. We shall look at the details of a collector later, but for the moment we can imagine the return value of executing a pipeline is an object that is returned by the collector.

List<String> result =  Arrays.asList("Larry", "Moe", "Curly")
                              .stream()
                              .map(s -> "Hello " + s)
                              .collect(Collectors.toList());

// result will be a List<String> containing "Hello Larry", "Hello Moe" and "Hello Curly"

Let us evaluate the steps in the pipeline

Arrays.asList("Larry", "Moe", "Curly"): instantiates a list with three strings viz. "Larry", "Moe" and "Curly".
.stream(): Converts the list into a stream
.map(s -> "Hello " + s): Performs a map operation which prepends "Hello " to each item in the stream
.collect(Collectors.toList()): Defines a new collector which will collect the results of the pipeline into a list and invokes collect to collect the results into it (and eventually returning the collected list).

As you can observe, the type of the original collection is not preserved during the transformation. In fact you should be able to take a collection and invoke .stream() on it, whereby you effectively lose the type of the original collection itself. All the transformations are defined on streams and eventually you can convert the result Stream into your desired collection type by supplying the appropriate Collector. This leads me to imagine that lambda based transformations are really stream transformers with converters at both ends - one to convert a collection into a stream and eventually to collect the stream back into a collection of a desired type. (One could of course choose to have the end result continue to be a Stream that one can use subsequently). The stream allows one to account for considerations related to efficiency, parallelisation,and even collection agnosticity. The collections at the ends provide a concrete implementation best suited for other tasks outside the pipeline.

Streams

Clearly streams are amongst the most important types to be used in lambda based transformations. So it makes sense to take a look at the two associated classes. The interface java.util.stream.Stream and the helper java.util.stream.Streams.

Let us first take a look at Streams (javadoc). This is a helper class and has lot of supporting operations for creating streams. (note: .boxed() as used below converts a stream of primitive ints into a stream of Integers)

eg. some of the methods (for a full listing see the javadocs) are

Streams.intRange(start, stop, step) will generate a stream of primitive ints. (step if not provided is 1)
Streams.longRange(start, stop, step) will generate a stream of primitive longs. (step if not provided is 1)
Streams.concat(stream1, stream2) will concatenate two streams

IntStream str1 = Streams.intRange(1,10);
IntStream str2 = Streams.intRange(21,30);
Stream<Integer> joined = Streams.concat(str1.boxed(), str2.boxed());
System.out.println(joined.collect(Collectors.toList()));

// Expected output : [1, 2, 3, 4, 5, 6, 7, 8, 9, 21, 22, 23, 24, 25, 26, 27, 28, 29]

Streams.zip(stream1, stream2, function) will return a stream based on a function being applied on consecutive pairs across stream1 and stream2

Stream<Integer> ints = Streams.intRange(0,5).boxed();
Stream<String> strs = Arrays.asList("foo", "bar", "baz", "qux").stream();
System.out.println(Streams.zip(ints,strs, (i, s) -> i.toString() + s).collect(Collectors.toList()));

// Expected output: ["0foo", "1bar", "2baz", "3qux"]

Note: As mentioned earlier, any collection can be converted into a stream by invoking .stream() on it.

Stream

Now that we've see how to create, concatenate or merge streams, let us explore some of the more useful operations that can be performed on the stream. I won't be dwelling much on these transformations in this post. I intend to come back to them in a future post.

(javadoc)

Method	Description
allMatch	true if all elements of the stream match the predicate
anyMatch	true if any element of the stream match the predicate
filter	return a stream of subset of the elements matching the predicate
findAny	return an element of the stream matching the predicate
findFirst	return the first element of the stream matching the predicate
flatMap	Return a stream where each input element is transformed into 0 or more values
flatMap	Return a stream where each input element is transformed into a stream
forEach	Perform an operation on each element of the stream (usually for side effects)
limit	Return a stream with no more than the first maxSize elements of this stream
map	Transform the stream into another containing the results after applying the function mapper on each element of the stream
max	Return the maximum element of this stream based on the supplied comparator
min	Return the minimum element of this stream based on the supplied comparator
peek	Return the same stream even as the elements are also provided to the consumer
reduce	Reduce the stream to a single value, performing the reducer operation on each element along with the accumulated value (accumulated value starting with the identity).
sorted	Sort the stream based on natural order or supplied comparator
subStream	Return a stream after discarding the first startingOffset elements (and those after optionally provided endingOffset elements)
toArray	Convert stream to array

The result of performing the operations above can result in zero or 1 element (eg. findFirst) from the stream, a boolean result (eg. allMatch), no result whatsoever (eg. forEach), a reduced value (eg. reduce) or simply, another stream (eg. map, filter, flatMap, limit, subStream etc.). When a stream is returned, similar transformations could be applied on it again leading to a pipeline of transformations.

Functions

If you looked at the javadocs for the methods above, you will have seen that many of them require some kind of functions as predicates or transformers. These are similar to the .map(s -> "Hello " + s) invocation we saw earlier. And if you are used to the earlier versions of Java, they no doubt look very compact.

Let us for a moment imagine we are working with Java 7 or before and imagine how such a functional transformation could get passed to another method as an input. A completely hypothetical equivalent using classic java would be

Stream<String> strings;
strings.map(
    new MapperFunction<String,String>() {
        public String map(String str) {
            return "Hello" + str;
        }
    }
);

Contrast this with the compact declaration that java 8 Stream.map method accepts. You can quickly note the following :

There is no anonymous class instantiation required (MapperFunction above)
There is no function declaration (public String map above)
There are no type declarations (the two String declarations in public String map(String str) above)
return is implicit.

Also note that instead of traditional class being a parameter, we passed what for all practical purposes seems like a function. But, can java accept functions as arguments ? And if we do not pass the signature for the function how does it infer it?

For answers, we need to look at Functional Interfaces.

Functional Interfaces

Functional interfaces is a very interesting (and to the extent of my awareness possibly unique) introduction to the java 8 lexicon. For details we can refer to What is a functional interface?

a functional interface is defined as any interface that has exactly one abstract method. (The qualification is necessary because an interface has non-abstract methods inherited from Object and may also have non-abstract default methods). This is why functional interfaces used to be called Single Abstract Method (SAM) interfaces, a term that is still sometimes seen.

So functional interface can inherit non abstract methods. It also can have non-abstract default methods. But it can have exactly one abstract method.

Aside : Java 8 introduces the ability to define default implementations in interfaces. The treatment of the same is beyond the scope of this post. But you could read up on What are default methods? and the subsequent four pages if interested.

So, what was getting passed to map() earlier was not a function, but in java 8 terms, an implementation of a functional interface. If you look at the various parameters being passed to the Stream methods discussed above, you should find that many of the arguments are functional interfaces.

Java 8 also introduces a substantial degree of syntactic sugar to succinctly declare the implementation of a functional interface. In the case above it was s -> "Hello " + s

Lets look at how the compiler can make sense out of all this. First the argument provided to a map method is a Function<T,R> interface. If you look at the signature of this interface you will find not one, but two methods. Thankfully one of them called compose has a default implementation provided. Thus there is only one abstract method apply which takes a T and returns a R.

Given that the requirement of a functional interface has been met (since there is a single abstract method - apply), we can look at the signature and say that since the Stream<T> is of type Stream<String>, the input argument to the apply method will be of type String. Now the compiler can treat the function supplied as an argument to the map method, as the body of apply method of an instance of the Function<T,R> interface. The input argument to the apply method ie. T is a String. What about the output?

Here's where we meet another interesting aspect introduced by Java 8. ie. type inference using target types. The ultimate result of the pipeline above was a List<String>. Which was a result of a .collect(Collectors.toList()) operation. So the input to the collect should've been a Stream<String>. Since the output of the map method call has to be a Stream<String>, it means the output of the apply method must be a String. That fits nicely with our expectations that since the input is a String, "Hello " + s must also be a String. So all is well and the compilation can go ahead. You can read more about this at What is the type of a lambda expression?

Collectors

Another interesting introduction to how java 8 terminates the processing pipelines is collectors. To briefly recap, pipelines can start with a collection with a .stream() method on in which converts it into a Stream. Many operations on the stream eg. map, flatMap etc. can be chained on the same which will continuously transform the stream into another stream and yet another stream and so on. At some stage you could invoke a method such as findFirst, reduce or something similar which will return just one value. eg. the following will return a single value 109 because of the last reduce invocation.

Arrays.asList(1,2,3,4,5,6)
      .stream()
      .map(n -> n * n) // square the number
      .map(n -> n + 3) // add 3 to the value
      .reduce(0, (i, j) -> i + j)); // add the number to the accumulator

However in many other cases you might prefer a collection as the result. This is where collectors are useful.

A collector has the capabilities necessary to

instantiate the necessary object of the target class. eg. it could be a List (or some other collection you choose)
take each element from the Stream and appropriately modify the target object (eg. add to the List)
take two instances of the target object and combine them into a single instance. (this is related to parallelisation - if multiple lists were constructed in parallel, this helps combine all of them into one).

To recap, I will repeat the code block I showed earlier

List<String> result =  Arrays.asList("Larry", "Moe", "Curly")
                              .stream()
                              .map(s -> "Hello " + s)
                              .collect(Collectors.toList());

// result will be a List<String> containing "Hello Larry", "Hello Moe" and "Hello Curly"

In this example, I used Collectors.toList() which returned an appropriately useful collector. In most cases one can use similar functions to create an appropriate collector, or create a collector using instances of three functional interfaces for the three capabilities listed in the prior paragraph. However should you want to carefully craft your collectors in a peculiar way, you could write your collector and use the same as well. It is the collector which eventually returns the result of the pipeline, eg. in the case above, it is a List<Integer>

Pending topics :

Hopefully I have been able to provide a reasonable feel of the capabilities of Java 8 Lambdas and how they can be used. However there are many topics that I haven't been able to cover in this post and intend to cover them in later posts. Some of these are :

Many more examples of how to use stream transformations
Primitives vs. objects. There are a many methods and classes which help work on the primitives alone.
Closures
New classes like Optional
Aspects related to parallelisation
Hand crafting some of your own classes

Conference Report - Hasgeek jsFoo Pune 2012

2012-01-21T18:22:36+05:30

Just finished attending jsFoo Pune 2012 organised by HasGeek. It was an interesting and a well spent day. And I've learnt if one wants to blog about a conference, it is best done immediately post the event else much gets lost with a recollection as weak as mine.

The conference was spread over 3 tracks. So the best I can do is talk about the talks I attended.

The first talk I attended was Node.js, HTML5 and Phonegap for high performant content site app by Prasoon Kumar. I learnt some stuff about phonegap and espresso. Good starting point, but hardly got to see any actual code.

Next one was Synchronized models using Backbone, Sockets and Node by Ruben Stolk. He started up a demo and suggested that people in the room connect to it. It basically had a canvas with a image and amongst many things, one could turn the light on the light-stand on and off .. and any actions propagated to the backend and then to all the connected clients. Similarly with post-it kind of notes that one added on the canvas. Was a good example of using backbone for defining models both on the client and the server and using socket.io to stream events between the two. He went through a fair amount of code too and it was very understandable. I came back with a distinct impression that the model maintenance and ability to synchronise the model changes between a client and server and then between the server and all connected clients (thus in effect across all connected clients) was pretty cool. However I still felt there needed a fair degree of wiring up of the event related code, though I am not sure about it and need to experiment with it more.

The next one was How to apply BDD and TDD practices, using Jasmine library? by Anil Tarte. Anil and I have been colleagues in the past and I have a high respect for his skills. In the brief period he was able to convey how to use jasmine to write BDD Test cases for the client side code. His server was over web sockets, and he was essentially intercepting calls into specific methods that eventually communicated with the backend. He also mentioned it would be possible to intercept the over the wire HTTP traffic instead of javascript functions, though it would create more difficulties if one was not able to precisely control the server's outputs. I quite frankly had not imagined that BDD could be used so effectively and especially localised strictly to the client. So it was an insightful talk, that definitely will have me thinking about javascript BDD the next time. However he did seem rushed and under pressure and perhaps the talk lengths could be extended from the 45 mins to 55 mins or so.

Next was Building real-time web applications ... (Introduction to Websockets / Socket.IO) by Aditya Y. He talked a bit about evolution of websockets, various libraries that were developed along the way and the current state of websockets support. There was an interesting demo at the end where changes in one browser session were almost being reflected in real time into another browser session via websockets connections back to the server. One of the important points noted was that Websockets continues to ride over port 80 so can work across many firewalls, and while its initial handshake is HTTP like, the subsequent traffic is essentially TCP like over that HTTP connection.

A thoroughly enjoyable talk was Advanced JavaScript Techniques by Rajasekharan Vengalil. He looked like he had enough stuff to talk about for the whole day, and delved into some of the OO related aspects of Javascript and the additions to ECMAScript 5. He spent some good amount of time in explaining how prototype based languages work. There are just so many intricacies to the language and so many really strange behaviours (like silently ignoring assignments etc.) that if I hated the language I would've just continuously muttered WTFs and if I happened to passionately like the language, I just might've facepalm'ed my way through the talk. Thankfully my outlook towards javascript is quite neutral, so I just enjoyed it . A lot.

Node.js Patterns and How we build ActiveNode was the next talk by Sreekanth Vadagiri. He talked about some of his experiences with node.js, his preference for using CoffeeScript rather than JS (despite it being harder to debug), many of the patterns he liked, some of the libraries he used and deployment matters. He was just as candid about some of the gotchas as well. A useful insight into the node.js system.

The last one of the day was Amplify your stack by Sunil Pai. He talked about a lot of libraries that could be used for client side development and deployment, about using javascript for templating, ensuring rigorous unit test coverage at all stages. Gave me the feeling there's a lot I simply do not know about whats happening on the client side JS related libraries and frameworks. Couple of remarks I recollect - "You (the JS developer) own the browser" and "When you have a strong test case coverage, you can CODE BOLDLY".

Summary: There's a lot I need to learn about whats happening on the client side. Thats true about the server side as well. And the day helped me understand just a little bit better what I don't know. Yet I got the distinct feeling of discomfort with node.js. Not with the tool. But with the assumptions that seem to go with its usage. There was poor articulation about what kind of use cases it is good for. Except that it is really good for high thruput/no. of client connections. Either there was a misplaced understanding of it being the only good way to get this kind of thruput or there was inability to clearly articulate what other benefits developers could expect out of using node.js in situations where thruput/concurrent connections is not particularly important for them. Perhaps I was just at the wrong places when someone was offering a more insightful articulation of this. But I really did not hear it.

On the whole, I enjoyed the sessions, and this was a day very well spent. Hats off to Kiran and his team for organising a good event. Makes me look forward to the next one they organise.

Why OSGi? Or why not using it makes your JVM runtime unsafe.

2012-01-21T18:22:36+05:30

Not sure how long ago I started using OSGi. Perhaps it was 12 months ago or then perhaps 18. And yet I still find it painful using OSGi especially every time I bring in a foreign set of jars into the ecosystem. And yet I continue to be a dogged proponent. Here’s why.

First let us understand one of the many problems OSGi solves. Let us imagine your java application has exactly three classes. One is the class you wrote called “My.java” bundled in a jar called “my.jar”. Another is a class called “Uses.java” whose api and features are leveraged by “My.java” and is in a jar called “uses.jar”. Finally there is yet another class “Transitive.java” which is leveraged by “Uses.java” and lies in a jar called “transitive.jar”.

Ideally when you run your application you should have all the three jars available in your classpath. Since you did not write “Uses.java”, how would you know about its transitive dependency on “transitive.jar” ? If you are fortunate, the build tool used to create “uses.jar” would have created a corresponding “uses.pom” or similar file which would allow your build tool to discover the transitive dependencies and assemble them all together. But what if that wasn’t true?

Turns out these situations are far more frequent than one would imagine. When I started using OSGi, I imagined such cases to be rather unlikely. And yet I have found a large number of situations. Sometimes the unrecorded transitive dependencies are in a profusion of xml libraries, sometimes on alternate logging tools, sometimes even on packages from a different jvm like bea and at least in one case even on dalvik.

So how do you know the libraries you are using have all their transitive dependencies recorded (and in any typical java application today there are literally tens of such jars)? Turns out you don’t.

The OSGi way :

OSGi requires you to record the metadata for the jar within the jar itself. Thus in the case above, “my.jar” would document the fact that it depended upon packages from “uses.jar”. In turn, the metadata on the “uses.jar” would record the fact it exports a certain set of packages (which “my.jar” uses) and also requires (imports) other packages which incidentally also happens to be exported by “transitive.jar”. The OSGi runtime will make sure all the dependencies are appropriately resolved. Thus in this situation if you attempted to run the application without “transitive.jar”, OSGi would flag that off as an error since some of the package imports would not get satisified and your application will not start until you add the third jar into the classpath.

The JVM runtime by default is unsafe

Java is often considered safe given its static typing. But that is just half the story. As the example above shows, its runtime is unsafe. And if you are a library author, you are in turn contributing your share by carrying forward that unsafeness into your user’s domain. The compiler makes sure you have compiled the code correctly to the interface of a given type. But the runtime does not guarantee the presence of that type. At a code level even if you think you’ve proved your code works to the extent you could, quite frankly the poor runtime makes such a proof quite illusionary. Since you have no guarantee your code will work even if it compiled successfully. And if you want to test such errors, how on earth will you even test such conditions ?

Library Vendors : Please make your jars OSGi compatible

For every hour you save by not making your jars OSGi compatible, you are requiring your “n” users who are interested in ensuring a stronger safety of their system to spend “n” hours ie. one each. Believe me, I’ve spent enormous amounts of time doing this as a user. Thats what gives OSGi a bad rap. Thats what makes people say it is complex.

OSGi is really trying to make your application and java runtimes in general safer (There are many other capabilities, but I’ve focused on only one). Do use it when you publish your jars. And do encourage others to do so.

Scala needs terraces

2012-01-11T05:22:42+05:30

Terrace (noun) : each of a series of flat areas made on a slope, used for cultivation, or a flight of wide, shallow steps providing standing room for spectators in a stadium

Recently there was a useful discussion triggered off by the post True Scala complexity by Yang Zhang. Much has been debated about it in the comments on the post and other forums, most of it both articulate and constructive.

Martin Odersky proposed an idea on the ycombinator news thread as follows (next paragraph)

So, I believe here is what we need do: Truly advanced, and dangerously powerful, features such as implicit conversions and higher-kinded types will in the future be enabled only under a special compiler flag. The flag will come with documentation that if you enable it, you take the responsibility. Even with the flag disabled, Scala will be a more powerful language than any of the alternatives I can think of. And enabling the flag to do advanced stuff is in any case much easier than hacking your own compiler.

The basis of the idea wasn't new. Martin himself had earlier suggested Scala Levels, though the suggested enforcement of two levels via a compiler flag was. While I thought it was a useful idea, strangely enough I found many (especially in the twitter streams I follow) less than enthusiastic. I think there exists a perspective where one can look at this idea with more enthusiasm, and this post details that.

Learning scala is non trivial. For a person who is well versed with Java programming and with little else, this will require learning at least the following (I am listing only a few items)

Learning traits and objects (and unlearning statics). There's a lot of stuff here including multiple inheritance, member overriding, whether the members should be defs or vals etc. which takes some times to get used to.
Learning to deal with immutability and occasionally lazy evaluation
Learning collections constructs such as for comprehensions or, map, flatMap, filter etc.
Understanding the capabilities that scala offers in terms of generics including being able to define co/contravariant collections, and then learning to use these capabilities effectively
Being able to understand and leverage many aspects of type theory

Typically when a person makes a sustained effort driven by passion, he can cover a lot of ground very quickly. Thus some might be able to deal with the scala learning curve quickly. While this ability to learn could be achievable for many especially smaller companies, it is likely to be quite difficult for many others. Especially when the team sizes are large and/or developer capabilities might be modest, this is a real issue. One could form an analogy with a Cheetah and an Elephant. Some animals will be able to run fast individually, and others will trod slowly as a herd. One could wish elephants could run as fast as cheetahs but thats unhelpful, since the jungle is defined, and one best deal with it the way one finds it.

An ideal way to introduce scala would be to start with small groups of 3-5. But thats not practical in all cases. Besides for large teams, that means, most will need to wait for a much longer time before being able to use scala and profit from its substantial benefits. So lets say an organisation attempts to introduce scala in a modest team of say 20. Of which 2 are passionate, self driven souls, who tend to work much harder the rest to be able to progress much faster. Because learning to use scala the way scala wished to be used, is not a sprint but is a marathon (or at least a long distance jog), this creates issues. Soon enough the 2 are likely to run far ahead of the rest of the team and will start writing code which the others can't grok. Even as some of the remaining may actually be struggling hard but are likely to be losing enthusiasm simply because they can't find themselves being able to keep pace. An even bigger risk is the 2 continuing to make rapid progress even as they realise so much more remains to be learnt, even as the remainder are unable to start leveraging scala, since they still think there's a lot of distance yet to cover and they become far more focused on catching up rather than leveraging what they have learnt. So while learning proceeds, actual development with direct economic benefit stalls.

Lets say hypothetically this team decided that they would only focus on reducing java boiler plate (which is a big deal) and continue to use scala in a primarily OO paradigm (since that is what they are used to) and defer learning remainder of scala by at least six months. There's likely tons of benefits to be had by shifting to scala even with this limited scope. Yet all it takes to destroy harmony is one very enthusiastic developer. Who starts inserting (as yet) alien notions such as writing highly functional code, or implementing advanced type theory concepts (eg. scalaz) or starting to use a largely un-understood symbol soup (scalaz or the dispatch periodic table). This would make a mockery of the phased learning and force every one else to catch up (even if just to be able to read and maintain somebody else's code).

We've long known hills are not conducive to agriculture. If these hills are our learning curves, fields could be considered to be directly impactful deployment of our skills. Agriculture generally happens flatlands and plateaus but not hillslopes. And yet we've learnt to tame the tall hills using terraces. Instead of climbing for a long time, climb a little & grow a little. If teams can learn for a month, and deploy these skills for the next 6-9 months with the cycle repeating itself, it could help in the following ways :

Organisations do not need to invest in long training times where the opportunity cost of lost development time is one large fixed cost.
Intra team disparity is likely to be contained since the deployment period will allow many to catch up with those in the lead
Instead of scala being viewed as one big leap, it could be viewed as a incremental leaps with managed effort. I saw a remark which said people anyways deal with complexity - its called OO. And Java programmers deal with OO. So they should be able to come to terms with scala quickly. But there is a difference. Java wasn't as big a learning curve leap over C++ as Scala is over Java. And as importantly, a learning curve one has triumphed over already, somehow seems far less onerous than an identically sized one, one still needs to deal with.

In short learning scala needs terraces. Terraces, where some developers will not run far ahead of the pack during the learning stages. Terraces where developers won't have to feel too scared of alien'ish code suddenly appearing on their monitors. Terraces where developers will learn a few things and then deploy these skills over the next few months. These terraces could be made to work in different ways. It could just be a honour system using a set of levels identical or similar to the levels Martin Odersky proposed (though it would be extremely hard to ensure the same without tooling support). Or they could implemented using (as yet unwritten) tool like a scala-lint which will flag off advanced usages as warnings. Or as compiler switches the way Martin proposed.

There could be other ways to look at it too. Scala could be considered as a sharp weapon. Lethal in the hands of the trained. Self damaging in the hands of the untrained. So compiler switches could simply be a way to graduate from using sticks, to wooden swords, to blunt steel swords to the really sharp ones.

I saw a remark which said a compiler switch would divide the programmers. I seem to think such steps which help create terraces will divide the learning challenges. I also saw another remark which said, implementing compiler switches would play into the hands of languages like Kotlin or Ceylon or Fantom. I believe exactly to the contrary. The lower terraces can compete with Kotlin or Ceylon, and the upper terraces can provide a growth path that many other languages will not have. Not being able to plan a phased and an economically practical learning curve for large teams will make the case for other languages stronger.