So some parts of the AWS EC2 specifically related to EBS were non responsive or down yesterday. This was of course also supposed to be judgement day after skynet became self aware. Links between facts and fiction aside, the number of sites that got impacted was just quite large and at least some (not most) started wondering if cloud indeed is the way to go.

I think the cloud just became stronger. Even as many services go impacted, some didn’t. As an example Netflix was able to continue with its services. See Slide 33 in a recently conducted presentation of its architecture. I am certain there were many others who also were able to ride the storm. Since they had not just planned for redundancy of equipments but the architecture had accounted for disaster recovery (redundancy of locations). And many others who got impacted and put their customers through undeserved anguish still perhaps learnt a lot. A lot that they could’ve done differently.

What is interesting about “a lot that they could’ve done differently” is that cloud infrastructure as a service opens up a few more options to ensure high availability. Amazon AWS apparently has 5 regions and the US East region which got affected has at least four availability zones. It was one of that availability zone which got substantially impacted. Though in fairness the impact seems to have spread into other availability zones, something that shouldn’t ideally have happened. I am sure there probably were a lot of things AWS probably learnt and perhaps would do differently. But the same applies now to AWS customers as well. Some may choose to lose faith in the cloud. But many others might choose to realise that cloud infrastructures have the potential to offer so much more redundancy and options to high availability. Its just that one needs to realise that cloud data centers are not infallible, and after being aware of all the redundancy options, one just needs to design the right way to leverage them.

And how does one leverage them ? AWS has multiple availability zones. An application should ideally leverage at least two. If you read the Netflix presentation I referred to, Netflix apparently uses three. Do not assume the servers will not go down. Assume it is possible that at least one availability zone could go down. Make sure you have the systems to quickly activate, systems in the alternative availability zone. For that you will need to find ways to keep data current across availability zones. Also find ways to ensure you have the ability to quickly switch to and fro between availability zones. More advanced options could include concurrently active systems across availability zones or those spread across AWS regions or even between AWS and other vendors.

There’s little if any learnings here that are specific to AWS. They are indeed applicable in general to cloud based infrastructure as a service providers. But unless you are a part of a Fortune 500 or equivalent it is very unlikely that your internal infrastructure will offer as many options for redundancy as at least some of the larger infrastructure as a service providers could. Which is why I believe even if some may choose to switch back to the seeming comfort of private infrastructure, many currently using private infrastructure are likely to look at today’s events and realise that the public cloud actually offers so many better options for building highly available systems. The issues are not yet fully resolved. And many customers still perhaps are not being served. And while one hopes not, it could so happen that there could be further disruptions. But I find them really aspects of dealing with a learning curve as one transitions across class and scale of infrastructures. Ignoring the short term pain, and looking at it a little bit in the longer term, whichever way one really looks at it, I think the cloud just became stronger.

The file cia-1996.xml (links back to rublylearning.org) is the data from the CIA World Factbook of 1996 in XML format. It has details about 260 countries across five continents. Your challenge, should you choose to accept it, is to uncover the following details buried within this file:

• What is the population of the country with the most people? Yes, we know it’s China, but just how many people lived there in 1996?
• What are the five countries with the highest inflation rates, and what were those rates in 1996?
• What are the six continents in the file and which countries belong to which continent? Can you also produce them in alphabetical order?

1
2
3
4
5
6
7
8
9
10

import zipfile
from lxml import objectify
from operator import itemgetter
import itertools

# Unzip file, extract xml, convert to object    
zip = zipfile.ZipFile("../cia-1996.zip")
xmlfile = zip.open("cia-1996.xml")
root = objectify.fromstring("".join(line for line in xmlfile.readlines()))
zip.close()

1
2
3
4
5
6
7
8
9

# Country with highest population
maxp = 0
maxc = ''
for country in root.country :
    if int(country.get('population',0)) >= maxp :
        maxc, maxp =country.get('name'),int(country.get('population',0))
print (maxc,maxp)

# Output is : ('China', 1210004956)

1
2
3
4
5
6
7
8

# Top 5 countries with highest inflation rates
inflation_country_tuples = []
for country in root.country :
    inflation_country_tuples.append((float(country.get('inflation',0.0)), country.get('name')))
inflation_country_tuples = sorted(inflation_country_tuples, key=itemgetter(0), reverse=True)
print inflation_country_tuples[0:5]

# Output : ((244.0, 'Belarus'), (94.0, 'Turkey'), (85.0, 'Azerbaijan'), (83.299999999999997, 'Malawi'), (71.299999999999997, 'Yemen'))

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# Countries by continent
continent_country_tuples = []
for country in root.country :
    continent_country_tuples.append((country.get('continent',''), country.get('name')))
continent_country_tuples = sorted(continent_country_tuples)
current_continent = None
countries_of_continent = None
continent_grouped_countries = []
for continent, country in continent_country_tuples :
    if continent != current_continent :
        if current_continent :
            continent_grouped_countries.append((current_continent, countries_of_continent))
        countries_of_continent = []
        current_continent = continent
    countries_of_continent.append(country)
continent_grouped_countries.append((current_continent, countries_of_continent))
print continent_grouped_countries

# Output : too long to include here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Country with highest population
print reduce(lambda (maxc,maxp), c :
             (c.get('name'),int(c.get('population',0)))
                if int(c.get('population',0)) >= maxp else
                    (maxc,maxp),
             root.country,('',0))

# Top 5 countries with highest inflation rates
print tuple(itertools.islice(
            sorted((
                (float(country.get('inflation',0.0)), country.get('name'))
                        for country in root.country),
                key=itemgetter(0),reverse=True),
                5))

# Countries by continent
print tuple((continent,tuple(country[1] for country in countries))
        for continent, countries in itertools.groupby(
            sorted((country.get('continent',''), country.get('name'))
                    for country in root.country),itemgetter(0)))

1
2
3
4
5
6
7
8
9
10

# Find the country with the maximum population
print reduce(lambda country_pop_max, country_pop_next : country_pop_next if country_pop_next[1] > country_pop_max[1] else country_pop_max,
        map(lambda country : (country.get('name'), int(country.get('population', 0))),root.country),('',0))

print tuple(itertools.islice(sorted(
            map(lambda country : (country.get('name'), float(country.get('inflation', 0.0))),root.country),
            key=itemgetter(1),reverse=True),5))

print map(lambda (continent, continent_country_tuples) : (continent, map(lambda (continent, country) : country, continent_country_tuples)),
    itertools.groupby(sorted(map(lambda country : (country.get('continent'), country.get('name')),root.country)),itemgetter(0)))

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

print reduce(
        # Comparator to decide the country with the maximum population
        lambda country_pop_max, country_pop_next : country_pop_next if country_pop_next[1] > country_pop_max[1] else country_pop_max,
        # get a sequence of tuples of (country name, country population)    
        map(lambda country : (country.get('name'), int(country.get('population', 0))),root.country),
        # initial seed
        ('',0))

print tuple(
        # take the top 5 items
        itertools.islice(
            # sort the list
            sorted(
                # get a sequence of tuples of (country name, inflation)   
                map(lambda country : (country.get('name'), float(country.get('inflation', 0.0))),root.country),
                # sorting to be done using the second element of the tuple
                key=itemgetter(1),
                # sort to be done in the descending order
                reverse=True),
            # count of elements to be sliced
            5))

print map(
    # function to flatten the continent_country_tuple tuple into a country tuple      
    lambda (continent, continent_country_tuples) :
        # first element of the tuple is continent
        (continent,
        # as the second element, return only the country from the continent country tuple to form a tuple of countries                                        
        map(lambda (continent, country) : country, continent_country_tuples)),
    # group the continent country tuples by continent
    itertools.groupby(
        # sort the continent country tuples
        sorted(
            # extract a continent country tuple from the country
            map(lambda country : (country.get('continent'), country.get('name')),root.country)),
        # this is the function to extract the key to sort by
        itemgetter(0)))

The other day I was referred to a blog post jQuery is a Monad. That is an interesting post which if you have any interest in Monads should read. My first thought was that jQuery was another implementation of Fluent Interface and it just did not strike me as a monadic construct.

There are blog posts which take an essential thought or a message and by putting together various arguments bring it to some logical conclusion. This post is not one of them. It just describes the results of my thought process which started off contesting the assertion, to accepting it and then realising that either way, it really didn’t matter a whole lot. I don’t claim it to be sufficiently accurate except to state that it is accurate only to the best of my belief and understanding. Pointers to any inaccuracies or alternate interpretations in the comments shall be gratefully received. And yes, this post in the end reaches a somewhat boring conclusiion.

Having got that little expectation management out of the way, lets get back to monads, jQuery and Object Orientation. Writing a tutorial on Monads is a rite of passage for many. Thats something I’ve never done. And I don’t wish to, but I cannot escape the fact that I’m going to have to introduce monads to those who are as challenged in their understanding as I am.

What are Monads ?

I’ll give you the loose idea. Whenever a function completes any computation, the results of the computations are often returned as the return value of the function. But every once in a while you need a container around that return value. A container to store the state of a computation, sometimes with additional metadata or historical annotations for use later. At its very basis, a monad is a container for storing computations and being the container to be carried between functions. Think of it as a shipping container if you will, which moves materials between factories - the factories being various functions which operate on the contents of that container.

But it takes more for a monad to be a monad than just being a container. Given any content, the monad should be able to construct itself by wrapping itself around the content. Thats what the source factory invokes when it fills up the container with goods. Some prefer to call it a type constructor - the type here being the container, or the monad (which may internally contain goods of various types, but I get ahead of myself).

In addition once the container reaches the destination factory, unlike typical container, this one does not just allow its contents to be unloaded. Nay, that would make it too simple. In this case, the container has to be taken to a factory where it allows a robotic arm of the factory to plug into a receptacle it provides, which the factory can use to extract the underlying values, often one at a time. The factory further processes these goods, which again come out at the other end of the factory as a different set of containers. And it is quite likely that the type of the goods in the new containers could’ve changed.

That in essence is a monad. Easy, right ? I’m sure not and I’m now only going to make it a bit harder.

The wikipedia, definition of monad Monad)

I’m just going to let you read the definition as provided by wikipedia.

A monad is a construction that, given an underlying type system, embeds a corresponding type system (called the monadic type system) into it (that is, each monadic type acts as the underlying type). This monadic type system preserves all significant aspects of the underlying type system, while adding features particular to the monad. The usual formulation of a monad for programming is known as a Kleisli triple, and has the following components:

1. A type construction that defines, for every underlying type, how to obtain a corresponding monadic type. In Haskell’s notation, the name of the monad represents the type constructor. If M is the name of the monad and t is a data type, then “M t” is the corresponding type in the monad.
2. A unit function that maps a value in an underlying type to a value in the corresponding monadic type. The result is the “simplest” value in the corresponding type that completely preserves the original value (simplicity being understood appropriately to the monad). In Haskell, this function is called return due to the way it is used in the do-notation described later. The unit function has the polymorphic type t→M t.
3. A binding operation of polymorphic type (M t)→(t→M u)→(M u), which Haskell represents by the infix operator >>=. Its first argument is a value in a monadic type, its second argument is a function that maps from the underlying type of the first argument to another monadic type, and its result is in that other monadic type. The binding operation can be understood as having four stages:

   1. The monad-related structure on the first argument is “pierced” to expose any number of values in the underlying type t.
   2. The given function is applied to all of those values to obtain values of type (M u).
   3. The monad-related structure on those values is also pierced, exposing values of type u.
   4. Finally, the monad-related structure is reassembled over all of the results, giving a single value of type (M u).

In object-oriented programming terms, the type construction would correspond to the declaration of the monadic type, the unit function takes the role of a constructor method, and the binding operation contains the logic necessary to execute its registered callbacks (the monadic functions).

In practical terms, a monad (seen as special result values carried throughout the pipeline) stores function results and side-effect representations. This allows side effects to be propagated through the return values of functions without breaking the pure functional model.

The bind operation

So coming back to the container analogy, the container is the monad, the goods are the underlying types, and the factory is a function which takes a underlying type and spews out another monad which contains the newly manufactured goods. There is a difference to be noted, the container doesn’t go into the factory, its the factory thats plugged into the monad through the bind operation, which triggers the processing. Also note that the output of each such factory is not just goods, but containers of such goods it manufactures.

There, you can now imagine a series of containers and factories, or monads and functions which are sequenced to assemble a pipeline to produce the desired goods / computations.

Before we go on to how this works in OO, I would like to spend some more time on the function thats provided to the binding operation. This function accepts the individual contents of the incoming container and returns a container with the newly manufactured goods. A somewhat similar function is used when using map operations on containers eg. Lists. So if I had a list of integers, and I wrote a function which doubled their values, then the map operation would be written in python as :

1
2
3
4
5

def twice(val): return str(2 * val)

print map(twice,(1,2,3,4,5))

#Expected output is :['2', '4', '6', '8', '10']

This is a good example where the container (tuple) is unbundled inside the map operation, each constituent value of the list is passed to the twice function, and all the results of the twice function are reassembled into yet another container (list). The type of the constituents also changed from ints to str (string) along the way. To be a strictly monadic construct, the twice function should’ve returned not just the double of the value, but a double contained in some other container (either a tuple or a list), and the map function should’ve extract the individual strings from each of the individual tuples/lists and constructed a final tuple / list out of the same.

While unlikely to be confused as so, let me restate for clarity, the map function is not the monad. The tuple and the lists are the monads here. The map function is an example of the contextual capabilities that monads expect from their environment (eg. via and around the bind operator in haskell). And twice is the operation thats performed on the monad. Again to restate, the monadic chain is like a set of factories that spew out containers, which allow other factories to plug into them and in turn extract the contents and then spew out even more containers.

Monads and Object Orientation

So how does this work in an object oriented context ?

Lets take rules 1 and 2 of a monad. If I was to declare a class which had a constructor which took in a value and then wrapped it, then I would satisfy these two rules for being then be able to suggest that the class is a monad. However the rule 3 gets a little bit more interesting. Object oriented languages have the ”.” operator which is somewhat analogous to the do block which can chain operations. So if I was to write o.foo() that would be equivalent of suggesting that I invoke the method foo() on the object which being a member method of the class has access to all the object internals and thus is able to access the wrapped value and do the necessary computations. Now if foo() were to return any object again of a class which satisfies these very rules, then I would be able to say that this class along with its member function foo() is a reasonable object oriented analogy of the monad. And I would be able to start chaining the methods as in o.foo().bar().baz()

Whoa! let me restate that again. If a class, has a constructor which takes a value and wraps that value, and has a number of additional member functions which each operate on these underlying values and return an instance of either the same object or another object of a class which follows the same rules, then I can say that that particular class is a monad. Incidentally thats exactly what fluent interfaces do, except that they do not have any specific expectation of wrapping. And a large number of classes may incidentally fit this description. And if jQuery is a monad, so are all of them.

Well, we did relax a few constraints along the way. First of all the functions are not stand alone functions. They are member methods which have direct access to the underlying wrapped value. Secondly they themselves don’t return a monad around an individual item in a collection. They return a monad around the entire collection of values. Thus the complexities and capabilities of the do block and the bind function are substantially simplified when using the “.” operator. And finally going back to the container analogy, the class defines and consists of both the containers and the factories. Seems like the threshold for stating a particular class is a monad is actually quite low. Turns out we reached a boring end. And it seems we are not much wiser at least in terms of any specific conclusions or insights. But every once in a while sometime the journey is more exciting than its end. For me this did seem like one.

Why are objects simple and monads complex ?

At least for me the above was true. Turns out objects collate related data and functions together into one class. Also the do block and bind operator on an object is very simple. So for many especially coming from the OO school, objects are well understood. On the other hand understanding the requirements of monadic constructs takes quite some time. So there’s a lot of gray cells that need to be exercised to start understanding what a monad is and how the various requirements for a monadic construct can be satisfied. And when mapping between monads, and objects which can also be seen to be monads, a lot of that complexity is either partially waived (eg. functions returning monads, the infrastructure around bind operators unbundling and bundling monads) or just simplified (functions and underlying values being colocated in the same class, the “.” operator being a far more simplified version of do block). But after some substantial headaches, it does start to seem that perhaps, just perhaps, monads aren’t so complicated after all :).

So is jQuery a monad ? I believe one could choose to be very pedantic and point out minor issues with that assertion. Or one could accept that the intent of monadic sequences are well represented in jQuery chains and accept it as one. I started with the former and ended at the latter position. Would I express that jQuery is a monad ? To the monadically challenged - No. That obfuscates far more than it enlightens. To them I would say jQuery is a fluent interface which allows continuous chaining of operations on an underlying set of dom objects. And is there a big “Aha moment” when one realises that jQuery is a monad ? I couldn’t find one. While the journey of trying to understand monads and correlate it with monads was very exciting, at least the OO practitioner in me is unlikely to have missed much had I not known that. But I got to understand monads better - and thats well worth the time, and all the headspinning.

A long long time ago, I worked on a fairly complex piece of design. And like any well behaved designer, I broke it down into a number of abstractions that made it manageable. I gave the abstractions funny sounding names. And before long I found those abstractions finding their way into the user interface. Abstractions which made no sense to the end user.

That experience taught me a good lesson. In attempting to deal with complexity, I was attempting to come up with the appropriate abstractions in a very bottom up way. So, strategic closure (of the Open Closed Principle), dependency inversion principle, et. al. all found their way into these abstractions I was modeling. These abstractions represented themselves via the various programmatic interfaces in the software design and their roles. At the same time there was a different perspective of the software. The way the end users would’ve preferred to see that software. The abstractions the way the end users saw the system were sometimes different than the underlying abstractions I modeled.

In a good design, the two abstractions should line up, and if there is an inconsistency, there is probably an issue with the design. Certainly this could be an issue in some cases. However in many cases the underlying backend might be built to offer far more capabilities than what are being exposed through the early version of the user interfaces. Probably there are multiple intents for which the software is being designed, and the particular user interface on table is just one of them. Probably, the underlying complexity of the back end is way too high which requires a very different nature of bottom up abstractions, which are different from the top down ones. Frankly, the reason doesn’t matter. Even after so many years, I look back and am comfortable with the thought that there needed to be two abstractions - one top down and one bottom up, one front end and one backend.

Fast forward many years. Another example helped me further attain some insight into the matter. We had a bunch of mobile smartphones floating around with the traditional PC UI abstractions being carried over into their design. Along came an iPhone - which rethought the interface the way users would’ve preferred to see the interface. Now iPhone was built on a traditional operating system. So it had the same abstractions that these operating systems have in the backend. However it decided to change some of the front end abstractions - in tune with the target market and the device / form factor peculiarities. Did the iPhone change the backend abstractions baked into its software - most likely not. However it changed the frontend abstractions, just enough to make the experience really simple and easy for its users.

As an engineer, I have lived with the regret of allowing backend abstractions to leak into the front end. But I have learnt something along the way.

• Don’t let the inapplicable frontend abstractions leak into the backend. This is especially true for most reasonably complex software. Strictly top down design can lead to a lot of brittleness in the long run, requiring very substantial surgery eventually.

• Don’t let the inapplicable backend abstractions leak into the frontend. In most cases this is a usability nightmare. ‘nuff said.

• Realise that you have to simultaneously service independent expectations ie. robustness of usability and robustness of modeling. Work with both the abstractions to mould each of them well: Work your frontend abstractions well enough to mould them to reflect the use cases of the software in the most usable manner. Work with the backend abstractions to let them emerge from the problem space you are trying to resolve at the backend. Now make sure you work with both of them to map them into each in a reasonably smooth fashion. This is easier said than done. But doing it is whats necessary.

PS: I am not referring to what is conventionally referred to as leaky abstractions, where the implementation leaks into an abstraction. I am referring to a set of frontend and backend abstractions leaking into each other. However if all the backend abstractions were to be treated as an implementation, then this would be a case of leaky abstractions as well.

I am disappointed with many of the newer languages which I earlier thought showed great promise of making programming easier, quicker, and more robust. And it boils down to one thing. Simplicity in learning. Having gone through substantial amounts of programming in C, C++, Java and Python, my quest for the “next” programming language remains unfulfilled.

Why ?

Programming should be simple. And it should be accessible. And when I mean accessible I mean people with IQ of approximately 100 should be able to write programs. I am disappointed that many of the trends seem to raise either the minimum IQ or the training time required to gain competency. And while that helps a community of the super brilliant, it does not make a substantial difference to programming in general. It remains esoteric and does not stoop to touch everybody.

What ?

So what are the features of my preferred programming language :

• JRE support: Should run on the JRE with Java interop. Thats the dominant well engineered platform that runs across all classes of desktops, servers and devices. Additional support for CLR is a bonus but not mandatory.

• Simplicity: Should be fairly simple to read, learn and understand. Python is a good example. PHP is a great example (at a simpler class of problems). C++ and Scala not good examples.

• Multi paradigm : Should support both OO and FP constructs. eg. Scala and Python. Half hearted support to functional programming as with python discouraged. Ditto with passionate support for objects with second class treatment for functions as in Java/Ruby.

• Multi core compatible : Should have good constructs for leveraging multiple cores eg. erlang, scala, clojure and many others.

• Type inferencing : (and I am a python programmer :) ). Good type inferencing coupled with on the fly non intrusive/disruptive compilations as with say the eclipse on the fly compiler or play framework. Three cheers for Scala. One reason I prefer type inferencing to dynamic typing is the much superior performance even while maintaining brevity and removing boilerplate.

• Constructs that are natural to humans not mathematics : This is actually a sub point to Simplicity. The constructs should be consistent with the normal average non mathematically trained brains. 2 + 3 is much simpler to understand than (+ 2 3). Python rocks. Lisp / Clojure or for that matter brainfuck dont.

• Closures and code blocks : Love ruby for this.

Is this a pipe dream ?

For the moment seems so. Do you know of a language which helps meet these requirements ?

And to be very clear (because there is a substantial risk of the same) - this is no flame bait or an opportunity to trigger language wars. It is meant to highlight two things

1. There is no ideal language out there, and

2. When designing languages - make them simple to learn and use. ie. for a given problem statement a good language is one which requires the minimum talent or training to solve the problem

This was a presentation I recently conducted to an audience of programmers / architects primarily in the financial services domain.

Every now and then I come across a blog post which talks about outsourcing and soon enough a commentary or discussion on Indian IT and a whole host of associated parameters comes up. Soon enough some of them start attracting a number of views and comments. And more often than not the comment stream starts attracting far more extreme views which do sometimes leave me recoiled.

No, I am not going to list them, link to them or comment on their content specifically. After some thought I decided that simply would shift the focus unnecessarily. Instead I thought given my reasonably long experience both in and outside India, it would be helpful offer my perspective on Indian IT within the context of global IT and Indian economy.

Indian IT and the role of exports

Indian IT (along with ITES and Gems & Jewellery) is a one of the stranger segments of the Indian economy. Unlike the rest of Indian economy, it is heavily export focused. The various numbers reflecting the state of Indian IT agree on at least one factor - at least 2/3rd of its revenues are based on exports.

Let us first look at some of the perceptions about Indian IT that I would like to comment upon.

You get what you pay for : In general thats a very reasonable statement. But it breaks down when you realise one Indian engineer + 1 H1B visa + 1 long flight results in tripling (at least) of remuneration without either the H1B visa or the long flight adding anything to the capability or the quality of the output of the engineer (well it could add a chip on the shoulder). Part of the difference is due to the difference in nominal exchange rates (which are driven by demand and supply of goods to/from the respective countries) and the exchange rates based on purchasing power parity. Based on purchasing power parity Indian salaries in the IT space do not underperform the rest of the world substantially. In fact when compared on purchasing power parity Indian Economy GDP more than triples and it ranks as the fourth largest economy in the world next only to the US, China and Japan.

Indian engineers are offering their services too cheaply. : Quite to the contrary exactly the opposite is true. The Indian IT engineer is too handsomely remunerated compared to the non IT engineers in India. I believe the high salaries in Indian IT is a problem. Many Indian innovations are today happening in areas outside of IT - primarily in the areas of making products and services affordable to the millions surviving on the fringe of the poverty line. And if recent trends in automobiles and telecom sectors are indicative, India is actually proving, that it is the rest of the world which is way too expensive, by offering products and services at price points which are unfathomable elsewhere. My reading of the empirical evidence leads me to believe that Indian IT could underperform other sectors of the Indian economy in terms of both innovation and quality. And a lot of it is due to the fact that the export revenues offer cushy jobs without the really hard work it takes to compete within the Indian economy. Quite frankly this one stereotype must be deposited very quickly where it belongs. In the Trash Can. So let me repeat - Indian engineers are and Indian software is too expensive and containment in its cost growth is most urgently required. While containing salary growth might be useful, investing in ability to create high quality software upfront and eliminating the defect fixing cycles post the initial release will help bring the cost down.

Indian IT offers poor quality software (The alternative version is outsourcing leads to drop in quality) : This is too hard a statement to comment upon, given its utter and gross generalisation. I am not aware of specific quality benchmarks which could be used to assert or deny such claims, though there is a fair amount of empirical evidence which could be used to either assert or deny these claims. India like any other country has a range of software quality in different companies and products which span the entire spectrum from superb to utter crap. I tend to agree with the statement that in many cases Indian software output does leave a lot to be desired in terms of quality. I also believe that there is a curious dynamic at play here. It is well known that in software one can demand and expect any two of three parameters to be fulfilled - viz. Cost, Time and Quality. Any efforts to increase quality can in some contexts run into the business issue of the customer clearly prioritising cost and time to market. And remember two thirds of the revenues (and presumably the customers) of Indian IT are from overseas. I have often struggled hard on the quality aspects, and have generally found that it requires a very strong support not just from the engineers but from the business stakeholders to offer sustained high quality.

Indian engineers are not as skilled / Indian engineers are far superior to the rest of the world : This is an interesting stereotype which is obviously wrong at either of the two extremes at which it is observed. Some believe the Indian engineers are extremely competent and productive but fail to realise that these engineers are from the extreme top end of a very competitive educational system. Bring down the comparison to some reasonable way of comparing an average Indian and Non Indian IT programmer and perhaps the comparison may not be so rosy for India. I think India does need to work much harder to strengthen its capabilities of programmers at the middle and lower ends. And sometimes I blame the fact that too many projects being transferred to India has resulted into so much demand for programmers that a programmer with a 10 percentile performance can still make a wonderful living by just changing his jobs every two years. This needs to change. But I am afraid as long as India continues to bill the rest of the world by the hour and not by capability and quality, this shall continue to be an uphill battle even as we shall continue to see islands of excellence.

Double whammy

The double whammy I refer to in the title of this post stems from the fact that due to heavy reliance on exports, Indian IT has been substantially based on the priorities dictated by her customers. Thus Indian IT is largely today what its customers asked it to be. And most of the customers are non Indian. The double whammy is the fact that Indian IT gets criticised for becoming exactly what its customers asked it to be - fast, low cost and high quality so long as the high quality doesn’t interfere with the fast or the low cost :). To be fair I am aware of the same criteria getting applied within many non Indian IT companies as well - so its not an Indian innovation. To be fair I am also aware of some situations where the lack of focus on quality is not driven by customer prioritisation. Is there a way out of the double whammy. Frankly I can’t see one easily. Yet this post would perhaps go some way in enriching the reader’s perspective on some of these factors.

The role of the population

There is one aspect about India which separates her from many other IT exporting countries. Her population. Not is it only really large (the second largest in the world), but it is growing fast and expected to continuously grow younger over the next two decades (dream demographics for an economist). Which means India stands unique in her ability to deploy masses. Which also means the problems which are easier to solve by deploying masses are more likely to find their way to the Indian shores. Combined with a high growth rate it also contributes to most Indian programmers getting promoted to management ranks (whatever that means) fairly soon. As we shall see in the next section this is an important factor in the Indian context.

Improving capability and quality

Independent of the validity of stereotypes which I questioned above I think it is a dead certainty that Indian IT can focus on improving both its capability and quality. It is not infrequent for me to feel demotivated after meeting some college students or fresh graduates and realising their priorities are really driven around what is the quickest way to make maximum money. This leads to unhealthy focus on building skills with high resume value. (Yeah that may sound funny to some of you, but its not uncommon). And given the high growth as I referred to in the earlier section - the role model for IT is - start as a programmer, change jobs every 2-3 years, become a team lead in 3 years, a manager in 6. And if you are particularly technically inclined you can become an architect guiding many projects and helping support many pre-sales efforts. What I haven’t seen Indian IT getting criticised for frequently enough, is the fact that so few of her members contribute to open source. While dzone and reddit may attract a large number of readers from India, a disproportionately small proportion of the people writing the posts stay in India. And most good programmers have moved on to becoming a Tech Lead or a Project Manager driving the average sustained technology experience lower and lower. So much for the “IT superpower” marketing that the hype manufacturing machinery creates internally. In a recent meeting with around 20 plus people in the room, I was one of the only 2-3 persons who believed India is not an IT superpower.

Change

I would like this to change. I would like this to change substantially. But I do not see the economic incentives in place for that to happen. Yet. However there is this strong feeling that something will indeed happen to make things change for the better. This is clearly something Indian IT will need to grapple with in the months and years to come. However there are at least a few factors which will lend themselves positively towards strengthening Indian IT.

Agile. Movement to agile requires a continuous focus on quality that cannot be wished away as easily. Projects that genuinely adopt agile methodologies will be implicitly driven toward being able to offer higher and sustained high quality.

Saas. As more software development shifts from intra enterprise development (where it is a little easier to contain impacts of poor quality) into Saas, there will be implicit pressures from the customers on the quality front. Another positive influence of Saas is that given the higher sharing of software across a much larger user base, the demand for number of developers would come down. Even as it may influence the demand for Indian IT itself negatively, I think it would help drive Indian IT into focusing more on capability than deploying masses.

Internal Growth. While much of the historic growth of Indian IT came from outsourcing, I anticipate the Indian industry to start driving its growth even more soon (given the really high growth engine it finds itself in). In such scenarios, the portfolio of software assignments will include a higher number of strategic and critical projects at extremely challenging cost parameters. This portfolio readjustment will help influence the quality positively.

I’ve attempted to highlight that Indian IT as exists today is (partially) a function of her customer’s expectations. And while there are some unfair stereotypes about it, there are clearly some things it can clearly improve upon. It is with some trepidation I write this since attempting to deal with perceptions in a generalised / stereotyped scenarios can be quite risky. So allow me to end with the disclaimer. This is a documentation of my understanding - YMMV.

In Functional Programming with Python – Part 1, I focused on providing an overview. In this post I shall focus on the core python language constructs supporting functional programming. If you are experienced pythonista, you may choose to skip this post (and wait for the next post in this series :) )

Sequences in python are not immutable :

When using sequences in python the thing to be noted is that sequences are not immutable. This provides you with the following options.

a) Use immutable sequence types : This is only possible by defining different types for sequences than the ones built into the language. b) Ignore all methods on the sequences which modify them c) Waive functional programming tenet of immutability

My preferred option when explicitly focusing on writing functional code is to use b)

Sequence types in python

The most commonly used sequence types in python are :

• Tuple : Immutable, Ordered, Indexed collection represented as (val1, val2)

• List : Ordered collection represented as [val1, val2]

• Dictionary : A key indexed collection represented as { key1: val1, key2 : val2 }

• Set : An unordered collection allowing no duplicates. Represented as set([val1,val2])

• str and unicode : While primarily meant to serv as ascii and unicode strings, these data structures also act as sequences of characters.Represented as “abcd” or ‘abcd’

Of the above only tuples are immutable.

There are other sequences used less often as well. An example is frozenset which is an immutable set.

Simple iteration The for construct allows you to loop through a sequence. eg.

1
2
3

one_to_five = [1,2,3,4,5]
for num in one_to_five :
    print num

Iterators on dictionaries

Unlike tuples, lists and sets where iterators essentially traverse through the sequence constituents, there are a number of different iterators on dictionaries. These are :

1
2
3
4
5
6
7
8
9

d = {1:"One", 2: "Two", 3:"Three"}
# keys
for val in d : print val
# an alternative for keys
for val in d.keys() : print val
# values
for val in d.values() : print val
# key, value tuples
for key,val in d.items() : print key, val

Slices

Slices allow creation of a subset on a sequence. They take the syntax [start:stop:step] with the caveat that a negative value for start or stop indicates an index from the end of the sequence measured backwards, whereas a negative step indicates a step in the reverse direction. The following should quickly indicate the use of slices

1
2
3
4
5
6
7
8
9
10

seq=[0,1,2,3,4,5,6,7]

#first 3
print seq[:3]
# last 3
print seq[-3:]
# 2nd to second last
print seq[1:-1]
# reverse the sequence
print seq[::-1]

The iterator protocol Since python is an object oriented language as well, it provides strong support for allowing iteration over an object internals. Any object in python can behave like a sequence by providing the following :

a. It must implement the iter() method which in turn supports the iterator protocol b. The iterator protocol requires the returned object, an iterator, to support the following ie. an iter() method returning self. and a next() method which returns the next element in the sequence, or raise a StopIteration in case the end of iteration is reached.

_I’ve tested iterators which do not themselves have an iter() method and it still works, but I still do not give in to the temptation of not defining the next() method alone since that would be inconsistent with the documented python specifications _

Let us examine a simple class indicating a range. Note that this is just for demonstration since python itself has better constructs for a range.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# An iterator object to sequentially offer the next number
# in a sequence. Raises StopIteration on reaching the end
class RangeIterator(object):
    def __init__(self,begin,end):
        self._next = begin - 1
        self.end = end
    def next(self):
        self._next += 1
        if self._next < self.end :
            return self._next
        else :
            raise StopIteration

# A class which allows sequential iteration over a range of
# values (begin inclusive, end exclusive)
class Range(object):
    def __init__(self,begin,end):
        self.begin = begin
        self.end = end
    def __iter__(self):
        return RangeIterator(self.begin, self.end)

oneToFive = Range(1,5)

In the above example, iter() on Range returns an instance of a RangeIterator.

The way this capability is used is as follows :

1
2
3
4
5

for number in oneToFive :
    print 'Next number is : %s' % number

print 'Is 2 in oneToFive', 2 in oneToFive
print 'Is 9 in oneToFive', 9 in oneToFive

The output you get with it is :

1
2
3
4
5
6

Next number is : 1
Next number is : 2
Next number is : 3
Next number is : 4
Is 2 in oneToFive True
Is 9 in oneToFive False

Generators

In the above situation, the state necessary for the iteration (ie.begin and end attributes) need to be stored. This was stored as a class member. Python also provides a function based construct called a generator. In case of a generator, the function should just do a yield instead of a return. Python implicitly provides a next() method which resumes where the last yield left off and implicitly raises a StopIteration when the function completes. So representing the above class as a function,

1
2
3
4
5
6
7
8
9
10
11

def my_range(begin, end):
    current = begin
    while current < end :
        yield current
        current += 1

for number in my_range(1,5) :
    print 'Next number is : %s' % number

print 'Is 2 in oneToFive', 2 in my_range(1,5)
print 'Is 9 in oneToFive', 9 in my_range(1,5)

I do not document the results since these are identical to the earlier code using a RangeIterator.

Note: I used my_range() and not range() since there already exists another range() already provided by python

An interesting point to realise is that in both the above situations, the next element to be returned in the sequence was being computed dynamically. To put it in the terminology better consistent with functional programming, it was being lazily evaluated. While python has no mechanism of lazy evaluation of functions, the ability of functions or objects to lazily evaluate the return values are sufficiently adequate to get the benefits of lazy evaluation at least from the perspective of cpu utilisation only on demand, and minimal memory utilisation.

List comprehensions

List comprehensions are one of the most powerful functional programming constructs in python. To quote from python documentation (even as I leave out a very important part of the full quote .. to be covered later),

Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses. The result will be a list resulting from evaluating the expression in the context of the for and if clauses which follow it. If the expression would evaluate to a tuple, it must be parenthesized.

A sample list comprehension is one which returns a sequence of even numbers between 0 and 9 :

1
2
3
4

# A list comprehension that returns even numbers between 0 and 9
even_comprehension = (num for num in range(10) if num % 2 == 0)
print type(even_comprehension)
print tuple(even_comprehension)

The output one gets on running the code above is

1
2

type 'generator'
(0, 2, 4, 6, 8)

Note that the list comprehension returns a generator which then returns a sequence containing 0, 2, 4, 6 and 8.

The for and if statements can be deeply nested.

lambda

Lambdas are anonymous functions with a constraint. Their body can only be a single expression which is also the return value of the function. I will demonstrate their usage in the next section.

map, filter and reduce

Three of the functional programming constructs which probably have aroused substantial discussions within the python community are map, filter and reduce.

map takes a sequence, applies a function of each of its value and returns a sequence of the result of the function. Thus if one were to use map with a function which computes the square on a sequence, the result would be a sequence of the squares. Thus

1
2

def square(num): return num * num
print tuple(map(square,range(5)))

results into

1

(0, 1, 4, 9, 16)

I mentioned earlier I will demonstrate usage of a lambda. In this case I could use a lambda by defining the square function anonymously in place as follows :

1

print tuple(map(lambda num : num * num,range(5)))

filter takes a predicate and returns only the elements in the sequence for whom the predicate evaluates to true.

Thus

1

print filter(lambda x : x % 2 == 0, range(5))

results in the following output

1

[0, 2, 4]

Finally the most controversial of them all - reduce. Starting with an initial value, reduce reduces the sequence down to a single value by applying a function on each of the elements in the sequence along with the current manifestation of the reduced value. Thus if I wanted to compute the sum of squares of numbers 0 through 4, I could use the following

1

print reduce(lambda reduced,num : reduced + (num * num), range(5), 0)

In the above example, 0 as the right most argument is the initial value. range(5) is the sequence of numbers from 0 thru 4. Finally the anonymous lambda takes two parameters - the first is always the current value of the reduced value (or initial value in case it is being invoked for the very first time) and the second parameter is the element in the sequence. The return value of the function is the value which will get passed to the subsequent invocation of the reduce function with the next element in the sequence as the new reduced value. The reduced value as returned finally by the function is then the returned value from reduce

The functional programming folks are likely to find this an extremely natural expression. Yet reduce resulted in a substantial debate within the python community. With the result that reduce is now being removed from python 3.0. (Strictly speaking it is being removed from python core but will be just an import statement away as a part of the functools package). See The fate of reduce() in Python 3000. Why so controversial ? Simply because the usage of map, filter, reduce above could be rewritten as

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#map
seq = []
for num in range(5) :
    seq = seq + [num * num]
print seq

#filter
seq = []
for num in range(5) :
    if num % 2 == 0 :
        seq = seq + [num]
print seq

#reduce
total = 0
for num in range(5) :
    total = total + (num * num)
print total

Just look at the two blocks of code and figure out which is easier to read (especially look at reduce). One of the areas where a large number of pythonistas and lispists are likely to disagree is the tradeoffs between brevity and easy readability. Python code is often english like (well, at least to the extent that a programming language can be) and some pythonistas do not like the terse syntax of lisp thats hard to follow.

I earlier mentioned that I left out a part of the description of list comprehensions from python documentation. Here’s that part of the quote.

List comprehensions provide a concise way to create lists without resorting to use of map(), filter() and/or lambda. The resulting list definition tends often to be clearer than lists built using those constructs.

Having said that I do believe many including myself will continue to use map, filter, reduce

Other helpful functions in python core that are helpful for functional programming are :

• all : Returns True if all elements in a sequence are true

• any : Returns True if any element in a sequence is true

• enumerate : Returns a sequence containing tuples of element index and element

• max : Returns the maximum value in a sequence

• min : Returns the minimum value in a sequence

• range : Return a sequence for given start, end and step values

• sorted : Returns a sorted form of the sequence. It is possible to specify comparator functions, or key value for sorting, or change direction of sort

• xrange : Same as range except it generates the next sequence element only on demand (lazy evaluation) thus helping conserve memory or work with infinite sequences.

We’ve seen many of the constructs that are typically useful for functional programming. I left out one big part - the itertools package. This is not a part of python core (its a package which is available with a default python installation). Its a large library and substantially helps functional programming. That along with some more sample python programs shall be the focus of my next blog post in the series. At this point in time I anticipate at least a few more parts after that to focus on a) Immutability b) Concurrency and c) Sample usage.

Hope you found this useful and keep the feedback coming so that I can factor it into the subsequent posts.

Lately there has been a substantial increase in interest and activity in Functional Programming. Functional Programming is sufficiently different from the conventional mainstream programming style called Imperative Programming to warrant some discussion on what it is, before we delve into the specifics of how it can be used in Python.

What is Functional Programming?

To quote from Wikipedia,

In computer science, functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data. It emphasizes the application of functions, in contrast to the imperative programming style, which emphasizes changes in state.

For beginners, one of the most fluent starter pages I would recommend for the history and specifics of functional programming is Functional Programming For The Rest of Us. This is a must read article which provides the reader with a good overview without getting too much into the nitty gritties of functional programming.

Since a detailed discussion on functional programming (henceforth referred to FP) is beyond the scope of this post, I will just briefly summarise the most critical elements of FP.

• Functions as the basic building blocks : Unsurprisingly FP requires the construction and usage of functions as the basic building units. In the simplest terms ”int add(int x, int y) { return x + y; }” is a simple addition function written in ‘C’. It takes two parameters x and y, adds them, and returns the result. This is a rather obvious and simple case but I stated it since I would like to refer back to it subsequently in this post.

• Functional Programming prefers functions without side effects : The add function above is a good example of a function without side effects. A function is said to be without side effects if the only changes it makes are those that are manifested in the return values. In other words such a function cannot change any global variables, write to the console, update the database etc. A fairly related term is referential transparency. A function is said to be referentially transparent if its invocation can be substituted by the return value in a program without impacting the program in any other way.

• Immutability : Pure functional programming often requires you to deal with immutable data structures. Thus the value of any variable is not open to modification (Thus they are called values and not variables). This aspect complements the functions without side effects. Thus the way most changes to state are implemented are not by modifying an object in place (which is how imperative programming deals with it) but by cloning the data structure with some of the values getting modified and the modified data structure being returned by the function. Java programmers are aware of the immutability of the String instances wherein any modifications to the string result in a new String instance being created. Imagine the same happening to all the datatypes across the program.

Benefits of Functional Programming :

Some of the nice benefits (I am tempted to say side effects) of functional programming are :

• Superior ability to deal with concurrency (multi threading) : Threaded programs are nasty to write. And nastier to debug. In an imperative environment, you not only have to deal with data structures being modified in place by some other parts of the program, in a threaded environment such modifications can happen using peer threads, even as your current thread whose logic you are focusing on is attempting to exercise that logic. Its an extremely unpredictable environment which has resulted in a number of how-to’s for safe threaded programming using constructs such as locks, mutexes etc. Functional programming deals with the issue far more elegantly. Instead of controlling and managing unpredictability, it takes it out completely. Because a data structure once constructed will not be modified and because the source of the modifications can be clearly located to the function which instantiated the datastructure, the unpredictability of data changing right under you is gone. This can be a little expensive to manage and FP does sometimes come up with some compromises (or cool features depending on how you view it) such as Software Transactional Memory but a discussion on that is completely beyond the scope of this post.

• Easier testing and debugging : Because modifications to data are contained and because a function communicates with the context outside it only via its return values, testing and debugging become far easier. You essentially need to focus on testing each function individually. Similarly during debugging you need to be able to quickly locate the function likely to have the problem, after which you can easily focus on the function to be able to quickly resolve the issue. Mocking out functions can also help testing each function in isolation. In general because of fewer side effects, testing under functional programming is often a lot easier, and the importance of having to do “integration” testing and “module” testing is lesser since testing functions in isolation is likely to identify most issues, far more than in typical imperative programming.

Why Python?

Python is not the best functional programming language. But it was not meant to be. Python is a multi paradigm language. Want to write good old ‘C’ style procedural code? Python will do it for you. C++/Java style object oriented code? Python is at your service as well. Functional Programming ? As this series of posts is about to demonstrate - Python can do a decent job at it as well. Python is probably the most productive language I have worked with (across a variety of different types of programming requirements). Add to that the fact that python is a language thats extremely easy to learn, suffers from excellent readability, has fairly good web frameworks such as django, has excellent mathematical and statistical libraries such as numpy, and cool network oriented frameworks such as twisted. Python may not be the right choice if you want to write 100% FP. But if you want to learn more of FP or use FP techniques along with other paradigms Python’s capabilities are screaming to be heard.

Sample Program :

I debated whether I should introduce various elements of functional programming using python in detail and then put it all together in a sample program all in future blog posts of this series, or whether I should start with a sample program which cover various aspects of function programming in this post and then explain various aspects in much more detail in future posts. For better or for worse, I have chosen the latter option. That means I shall be explaining one sample program and shall leave it to future posts in this series to get into greater details.

The sample program I have chosen is that of a simple calculator. A typical calculator supports simple unary or binary mathematical operators and performs floating point operations. Without much ado we now get into the sample program.

Immutable Data :

1
2
3
4
5

from collections import namedtuple

Context = namedtuple('Context','stack, current, op')
def default_context():
    return Context([],0.0,None)

Python is not particularly strong at immutable data. However one of the data structures, a tuple is immutable. A namedtuple is another data structure which supports both tuple like access through indices or through named elements in the tuple. For the calculator I shall need a Context which contains a stack for storing any incomplete operations, an attribute current reflecting the current value being shown on the screen and an op which might reflect a pending operation which is typically required for binary operators where the second value still needs to be provided. While namedtuple is a reasonable construct for simple tuple like objects, it would be helpful to have immutable objects as well - but thats to be covered in a future post.

Simple Functions

1
2
3
4
5
6

def add(x,y): return x + y
def sub(x,y): return x - y
def mult(x,y): return x * y
def div(x,y): return x/ y
def reverse_sign(x): return -1 * x
def pow(x,y): return x ** y

There’s not much to describe here. The functions should be self explanatory. For purpose of emphasis I would like to note that in the above code, “add” is now an entry in the namespace which is a reference to a function. This reference can be passed around, assigned to other entries. Thus the code below should work (though it does not form a part of the calculator program). This is to demonstrate how python treats attributes and functions virtually identically consistent with the Uniform access principle.

1
2
3
4

otheradd = add
add = sub
assert otheradd(7,3) == 10
assert add(7,3) == 4

Currying Currying is a treatment afforded in functional programming which allows a function of n parameters to be treated as a sequence of n sequential functions each of one parameter.

1
2
3

from functools import partial

square = partial(pow,y=2)

Here partial is a function reference. square now refers to another function with its y parameter value being anchored to 2

Invoking functions dynamically

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

unary_functions = {'!' : reverse_sign, '@' : square }

def handle_unary_op(ctx,x):
    return ctx._replace(current = unary_functions[x](ctx.current), op = None)

binary_functions = {'+' : add, '-' : sub, '*' : mult, '/' : div}

def handle_binary_op(ctx,x):
    return ctx._replace(op = binary_functions[x])

def handle_float(ctx,x):
    if not ctx.op :
        return ctx._replace(current = x)
    else :
        return ctx._replace(current = ctx.op(ctx.current,x), op = None)

Note that I created a unary_functions dict (or dictionary or hashmap) where the key is the character which represents the function and the value is the reference to the function.

Also note that in the handle_unary_op_function, I invoke ctx.replace method. On a named tuple it creates another tuple based on the existing namedtuple data, but with some of the values modified as specified in the keyword paramters passed to replace. After looking up the appropriate unary function ie. unary_functions[x], I also invoke it on the current value ie. unary_functionsx. I also defined another dict for binary operators. The handle_binary_op_ method reflects how the op in the context is set to the appropriate binary function that should be triggered after the subsequent value is known.

Finally the handle_float function either sets the current value to the incoming value or in case the current operator is already set it applies the binary operator to the current value and the incoming value and replaces current with the computed value.

Additional Code When I wrote the calculator program, I wrote the functionality to introduce braces. However that functionality is not particularly important in this explanation. So it is being listed here for completeness.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def start_brace(ctx):
    newstack = ctx.stack
    newstack.append((ctx.current,ctx.op))
    return ctx._replace(
                stack = newstack,
                current = 0.0, op = None)

def end_brace(ctx):
    stack = ctx.stack
    current = ctx.current
    oldcurrent, oldop = stack.pop()

    oldctx = Context(stack,oldcurrent,oldop)
    return process_key(oldctx,current)

tokens = { '(': start_brace, ')' : end_brace}

def handle_tokens(ctx,x):
    return tokens[x](ctx)

Processing one key I must confess I started off using key to represent the keystrokes, but along the way the key can also represent a complete floating point number (not just a single keystroke). Thus the key parameter can refer to a single character operator or a sequence of characters representing a floating point number

1
2
3
4
5
6
7
8
9
10
11
12
13

function_groups = {
    tuple(unary_functions.keys()) : handle_unary_op,
    tuple(binary_functions.keys()) : handle_binary_op,
    tuple(tokens.keys()) : handle_tokens
}

def process_key(ctx,key):
    if isinstance(key,(types.FloatType,types.IntType, types.LongType)) :
        return handle_float(ctx,key)
    elif isinstance(key,(types.StringType)) :
        for function_class in function_groups :
            if key in function_class : return function_groups[function_class](ctx,key)
    return ctx

In this case I set up a dictionary where the key is a tuple of all the keys representing a particular class of a function. Note that the .keys() method is a method which returns a list of all the keys in a dictionary. However since list is mutable, it cannot get used as a key into the overall hashmap, hence I convert it into a tuple.

The process_key function takes the incoming key, passes it handle_float if it is a number, or treats it as an operator. If it is the latter it searches for it in all the keys of each operator groups, and if it finds a match, it locates the corresponding handler function from the map and invokes it. Finally in case no match is found it ignores the key.

Processing a sequence of keys

1
2

def process_keys(keys):
    return reduce(lambda ctx,key : process_key(ctx,key), keys, default_context())

Here you see a reduce function being invoked. This belongs to the family of map and filter functions which are used extensively in functional programming. I shall attempt to briefly explain it here, but this family of functions in addition to a number of others will again be dealt with in a future blog post.

To interpret the usage read the above reduce statement right to left. Thus we start with a default context, and for each key in the sequence of keys, we invoke a lambda (thats like an anonymous function), which calls process key with the context and the key. Note that the first parameter to the lambda is either the initial value (the default context) or the return value of the last process_key (which is also a context) and the key is each key in the keys sequence injected sequentially.

To further make it easy I re-represent the same function below differently which is much more readable and easier to understand. This shows one more strength of python. Because of its focus on readability, it actually can be used to write functional programs are much more readable by a large mass of programmers than most of the functional programming languages themselves (readability being subjectively interpreted by me as what is most natural for english or similar language speaking people).

1
2
3
4
5

def process_keys(keys):
    ctx = default_context()
    for key in keys :
        ctx = process_key(ctx,key)
    return ctx

Usage As a mechanism to conduct some rudimentary tests on the code written so far, the following code is introduced. Here you can get an overall feel of the program.

1
2
3
4
5
6

if __name__ == "__main__" :
    assert process_keys((2,'+', 3)).current == 5
    assert process_keys((2, '!', '+', 5)).current == 3
    assert process_keys((2, '@')).current == 4
    assert process_keys((2,'+',3,'*',5)).current == 25
    assert process_keys((2,'+','(',3,'*',5,')')).current == 17

Some more slightly advanced Functional Programming

Finally to tickle your interest even more, here’s a slightly more advanced usage of functional programming constructs. Here I shall add all numbers between 1 through 10.

1
2
3
4
5
6
7
8
9
10

from itertools import chain

def plus_num_seq(n):
    count = 1
    while count <= n :
        yield '+', count
        count += 1

keys = list(chain.from_iterable(plus_num_seq(10)))[1:]
assert process_keys(keys) == 55

The plus_num_seq is a generator. Note the usage of the yield statement. Thus it will continuously generate tuples with the first element of the tuple being the ‘+’ character and the second being the number with the number varying from values 1 through n. The chain.from_iterable flattens the generated list (it thus has 20 items for n = 10, each alternate one being the ‘+’ character starting with the first items). Since we do not need the very first ‘+’ character, I removed it using the [1:] slice operator.

Just like reduce this style of code is quite typical of functional programming. Thats something I shall detail upon much more in future posts.

Hope you enjoyed the post. Keep the feedback coming so I can better structure the subsequent posts based on the feedback.

Note: The full source for the calculator calculator.py can be accessed here.

Update: I much later also conducted a presentation on the same topic at Pycon India 2010. The slides to the topic can be found at the bottom of this page (direct link: talk.html)