/var/log/mind

Background :

Roy Fielding recently wrote : REST APIs must be hypertext-driven. It is an excellent writeup which actually focuses on What REST is NOT and is written in the context of SocialSite Web APIs which are an implementation of the OpenSocial Restful Protocol. If you are interested in understanding what REST is in any substantial way I highly recommend Fielding’s post, and if you can spare some time do read up the comments and also Section 5 of his dissertation or perhaps the whole dissertation.

The essence of the post is found in the statement :

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

I agree with it entirely, and that is not what makes me uncomfortable. Its just that the rules of figuring out what is compliant with REST are a little porous, and the whole requirement of statelessness I find a little orthogonal to the driving principles of REST design.

Do semantics decide what is RESTful ?

Roy essentially states he has no difficulties in people following non REST approaches, but suggests that if something is called or classified as RESTful, then it must adhere to what he has described as characteristics of REST architecture, and if it doesn’t, thats allright, just don’t call it RESTful. I think this is a perfectly fair stance. But it does beget a question, what do I call an architecture style that is strongly inspired by REST, fairly close to REST but does not actually meet all the expectations that Roy laid out. But an even more important question - apparently there is some amount of semantic jugglery that one can conduct to make an architecture seem RESTful. Does it in such a situation become RESTful ?

I think REST as a architecture style is great, it really simplifies things in many ways, but (oh! the horror of it) I cannot seem to agree with it entirely. However that is my individual perception and opinion and not the topic of this post. The topic is the fact that I feel I am modeling something that does not meet the REST requirements in spirit but yet can argue my way through it in letter. If anyone of you gets the same feeling or has figured out a way out of it do post a comment below.

Setting the context. Quotes from Roy.

A few quotes from Roy before we get into the details. As per the post, clearly REST wasn’t intended to be a RPC substitute.

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

and from section 5.1.3 of the thesis on the matter of statelessness,

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

and from section 5.2.1.1 of the thesis, with regards to what a resource is :

The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. “today’s weather in Los Angeles”), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author’s hypertext reference must fit within the definition of a resource. A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time.

A sample application and REST like API :

Let us take a simple example - an ATM machine. A simple use case, user enters bank atm card, machine prompts for pin, user enters pin, machine shows menu, user selects make card payment, machine asks for card number and amount, user enters card number (destination account) and amount, machine transfers funds. Simple enough ?

Now lets build this using a restful (or not so restful) API.
Note: Purely for writing and reading simplicity I shall represent all HTTP requests using GET semantics.

1. User enters card : Client level activity. No API required here.
2. Machine prompts for pin : Client level activity. No API required here
3. User enters pin (and presses a OK / Submit button) :
Here’s where we need to send the following data to the server - card number and pin. As per REST, we should always access a resource on a server. So here’s how I am going to design the api for that

http://atm.bank.com/session?cardnumber=123456&pin=1234

This will initialise a new session, and the response shall return me a new session. The response xml shall be :

<session id="999" state="valid" authenticated="true" menuurl="http://atm.bank.com/menu/999"/>

4. Client requests menu :
The response was hypertext and had the URI for the menu which the client shall now invoke. Note that in the example below, 999 is the session identifier, and it would be possible to pass 999 as a parameter rather than as a segment in the URI itself (take your pick).

http://atm.bank.com/menu/999

Here’s a sample menu that might be expected

<menu>
  <action name="Withdraw" url="http://atm.bank.com/withdraw/999"/>
  <action name="Deposit" url="http://atm.bank.com/deposit/999"/>
  <action name="Make Card Payment" url="http://atm.bank.com/cardpayment/999"/>
</menu>

5. The user selects card payment.

http://atm.bank.com/cardpayment/999

The system detects that the required fields are not supplied responds with the necessary information that the client needs to supply

<cardpayment id="" status="not-initialised" target="http://atm.bank.com/cardpayment/999">
  <field="destinationaccount" name="Card Number" type="Integer" currentvalue=""/>
  <field="amount" name="Amount" type="Decimal" currentvalue=""/>
</cardpayment>

So the user now enters the two fields and submits the data

http://atm.bank.com/cardpayment/999?destinationaccount=98765&amount=111.22

The system verifies that all is OK (ACL, valid destination account, adequate balance exists etc.) and performs the transfer successfully

<cardpayment id="777" status="done" menuurl="http://atm.bank.com/menu/999">

Analysis :

Either you might have found the above API allright, or you might have one or more of the following issues :
1. The session id is a state tracking identifier. REST is stateless. Clearly this is not RESTful
2. The card payment was clearly a call to a remote procedure which could otherwise be represented as : transferMoney(sessionId, toAccount, amount)

One of the early things I learnt was that when designing transaction processing systems, many user requests result into some activities that need to be triggered on the server side (as in the transfer funds to credit card above). Intuitively it is a procedure, and because one is in the client server world it is a remote procedure. So how can it be RESTful ? Again, from what I have learnt, its easy - just convert the verb into a noun and the remote procedure is now represented as a resource. How so ? Simple - instead of transferMoney procedure, I now have a cardpayment resource which I activate. What else changed, the return format of the procedure (oops resource) is now a hypertext document which contains necessary state information about the transfer and hyperlink to the other associated resources. Voila ! We are RESTful. But is it still RPC ? The essence of RPC has not gone away.

But isn’t this a stateful architecture. By all means yes. Most certainly it is. But here’s how I am going to argue my way out of it - I actually created a resource called session, and subsequently was simply using it (I could use it by making a reference to it as a segment in the URI as I did above, or by passing in the id as a parameter, or by passing in a addressable URI for the session itself as a parameter eg. http://atm.mybank.com/session/999). This session has associated with it on the server side the card number and account number which I can use readily. Stateful ? Yes. But it is still consistent with the overall Resource Oriented approach of REST where the session is but one instance of that generic thing called a resource. Thats semantic skullduggery you might argue. Quite right .. But I couldn’t find a way to blow a hole in that argument.

However there is a small hitch in the above argument. The REST dissertation explicitly excludes the session from being treated as a resource (see the last line of Roy’s quote related to statelessness above). I still haven’t quite gotten over the asymmetry of treatment to session data and other resources. I cannot quite understand why stateless = RESTful and stateful = RESTless. It is not as if one ends up specifying all the resource data in all the cases. For all the associated objects (eg the Account object corresponding to the ATM Card number in the case above), one simply passes in the resource URI / identifier, and the server side then goes fetches the additional data as might be required to complete the transaction. So if one passes an invalid card number, the service will ensure that the appropriate validations will reject the transaction. So why can one not treat the session as a resource and have it being treated symmetrically with all the other resources. Why single it out ?

The need for statefulness :

Here’s one reason why I would really like to have statefulness - security. I do not want to keep on transferring again and again with each API call the ATM card number and the PIN, since these do not change frequently (if ever). I would like to transfer the same as few times as possible (notwithstanding any additional encryption or obfuscation I build on top of them). There have been cases where sniffing data over the wire has been suspected, and I would like to minimise the likelihood of a breach by minimising the number of times critical data is shipped over the wire.

Here’s another reason why I like statefulness - efficiency. In the above example, the (source) account number was never mentioned or passed, but yet I am sufficiently aware, it is actually required for every single non trivial activity. So why not just load it when the session is initialised, stuff it into the session and not worry about having to retrieve it from the database.

Clearly it is very easy to overuse and abuse statefulness, and I try to keep state data to the absolute absolute minimum, but completely statelessness is something I have yet to feel comfortable with.

Finally - and this is an argument more in principle than in practice, the need for statelessness is context specific. If I have a hefty server, and expect only a handful of active sessions at any point in time, it is possible to argue that stateless architecture simply may not be required at all in such a particular context. But to then imagine that the otherwise RESTful architecture is now not so makes me restless.

Final words :

In this particular post I am going to assume that those with more knowledge of REST and RESTfulness than the limited amount I carry, are going to blow holes in my semantic jugglery, and maybe prove that there is no amount of semantic jugglery which will make the architecture RESTful. That does beget the first question I asked - If an architecture style is largely inspired by REST but makes a few deviations what should one call it ? Maybe RESTlike or RESTspired but you might come up with a better word. I am certain I am going to be far more concerned about whats required in the given context and what is most appropriate for the users and the sponsors rather than whether the APIs are pure REST APIs. How about you ?

Update: Modified the title to make it a little shorter.

This post talks about applying Open Closed Principle, Liskov’s Substitution Principle, Dependency Inversion Principle and Interface Segregation Principle in Python, coming from a Java programming background.

Background :
A few days ago I blogged about Commentary on Python from a Java programming perspective. In that post I avoided getting into the specific details with code snippets etc since I wanted to focus on how it feels.

One of the observations I made was that I thought coming from a Java background, that background helped me from a class design perspective. I ran into a couple of posts from the ObjectMentor blog, namely The Open-Closed Principle for Languages with Open Classes, and The Liskov Substitution Principle for “Duck-Typed” Languages. The gentleman behind ObjectMentor is Robert Martin, who wrote a number of articles in the mid 90s related to these design principles. (Links to PDF : Open Closed Principle, Liskov Substitution Principle, Dependency Inversion Principle and Interface Segregation Principle)

Having had applied these principles a countless number of times,in C++ and Java, I thought it would be an interesting exercise to document how class design changes even when the same underlying design principles are applied to a dynamic language (in this case - Python). The remainder of this post summarises the design principles and the examples that Robert Martin talked about in his articles, and how these get implemented perhaps a little differently when used in static typed (Java) and dynamically typed (Python) languages. His original source snippets in the C++ language can be found in the articles I have hyperlinked to earlier.

Open Closed Principle (OCP)

Software entities (classes, modules,functions etc.) should be open for extension but closed for modification

What this basically means is that the code you write should be in a manner where it does not need to be modified when you need to extend it - the design of the code should allow for the extension to be made by new code being added, not old code being modified.

In the example below Circle and Square are two Shapes which can be rendered. The design essentially sets out with an objective that it should be easy to add newer shapes (say triangles) without modifying existing code. Lets straight away get into the Java version of the code.

public class Painter {

    public static void main(String[] args) {
        // We sould like the shapes to be drawn in the 
        // order of the shape types as found in this list
        List<Class> classOrder = new LinkedList<Class>();
        classOrder.add(Square.class);
        classOrder.add(Circle.class);

        // The shapes
        List<Shape> shapes = new LinkedList<Shape>();
        shapes.add(new Circle(1,1,5));
        shapes.add(new Circle(3,3,7));
        shapes.add(new Square(2,4,3));
        shapes.add(new Square(4,2,4));

        Collections.sort(shapes, new ShapeComparator(classOrder));

        for (Shape shape : shapes)
        {
            shape.draw();
        }
    }
}

// This declaration defines the contract across shapes
public interface Shape {
    public void draw();
}

public class ShapeComparator implements Comparator<Shape> {
    private List<Class> orderedClasses;
    public ShapeComparator(List<Class> orderedClasses) {
        this.orderedClasses = orderedClasses;
    }

    @Override
    public int compare(Shape s1, Shape s2) {
        return this.orderedClasses.indexOf(s1.getClass()) -
            this.orderedClasses.indexOf(s2.getClass()) ;
    }
}

public class Circle implements Shape {
    private int x;
    private int y;
    private int radius;

    public Circle(int x, int y, int radius) {
        super();
        this.x = x;
        this.y = y;
        this.radius = radius;
    }

    @Override
    public void draw() {
        System.out.println("Drawing Circle at (" +
                x + "," + y + ") with radius " + radius);
    }
}

public class Square implements Shape {
    private int x;
    private int y;
    private int width;

    public Square(int x, int y, int width) {
        super();
        this.x = x;
        this.y = y;
        this.width = width;
    }

    @Override
    public void draw() {
        System.out.println(
            "Drawing Square at (" + x + "," + y +
            ") with width " + width);
    }
}

The Painter class here defines the order in which the various shapes need to be rendered (classOrder), creates a list of all the shapes (shapes), sorts the shapes list using classOrder and then renders all the shapes.

The ShapeComparator is a class which implements the comparator interface for Shapes (public class ShapeComparator implements Comparator) and implements the compare method which compares two shape instances and decides the relative order between them.

The remainder of the code should be rather self explanatory.

So where is OCP being applied here ? For that you have to imagine a new shape, say a triangle now being introduced into the mix. In this case, Triangle would be a new class which would implement Shape. No existing line of code will change. However the Painter main method will now add the new class in the classOrder list in the appropriate place, and if you want to modify the order in which the types should be rendered, just modify their corresponding class placement in the classOrder list. The essential thing to be noted is that Shape, Circle, Square and ShapeComparator are all “open for extension but closed for modification.”

So what does the corresponding python code look like ?

# Look ma ! No Shape class
class Circle(object):
    def __init__(self,x,y,radius):
        self.x = x 
        self.y = y 
        self.radius = radius
    def draw(self):
        print "Drawing Circle at (%s,%s) with radius %s" % \ 
                (self.x, self.y,self.radius)

class Square(object):
    def __init__(self,x,y,width):
        self.x = x 
        self.y = y 
        self.width = width
    def draw(self):
        print "Drawing Square at (%s,%s) with width %s" % \ 
            (self.x, self.y,self.width)

if __name__ == "__main__":
    order = [ Circle , Square] # the order in which to draw the shapes
    shapes = [Circle(1,1,5), Circle(3,3,7), \
                    Square(2,4,3), Square(4,2,4)] 
    for shape in sorted(shapes,
            # Comparison function is embedded inline using a lambda
            lambda s1,s2 : order.index(type(s1)) \
                     - order.index(type(s2))): 
        shape.draw()

What learnings can we get from this ?

The first most apparent difference is - No Shape Class. Where did it go ? Well, static typed languages use polymorphism as a powerful mechanism of extensibility. In other words, in many cases the extensions are likely to be newer derived types. Thus design the rest of your code to work on the base type and introduce the newer derived types later as required without having to necessarily change existing code. However static languages primarily depend upon inheritance as the vehicle for delivering polymorphism. Dynamic languages on the other hand depend upon duck typing. Duck typing supports polymorphism without using inheritance. In this context you need the same set of relevant methods to be implemented in each of the extension classes. The role of the abstract base class or interface as the one which specifies the contract / api has been made redundant. You can still choose to define a base class / interface if you want to, but you no longer have to.

Another thing to be noted is the way the comparator is implemented. While java required us to create a new one method class implementing a required interface, and then required us to instantiate the same and then trigger its functionality, python allowed us to implement it inline as a lambda. In a very different way this is OCP being applied (okay okay .. for those insisting on theoretical correctness thats not true .. but close enough) inside the language design itself. A function is an object in python, and a lambda is a special kind (not a subtype) of a function, and there are capabilities built into python to easily manipulate function objects / lambdas. The sorter method functionality was extended to allow custom comparators by specifying a particular evaluation expression as lambda. In java we actually had to implement the inheritance hierarchy ourselves before being able to leverage the extensibility in the list class for custom comparisons.

In the context of OCP, the learning is that dynamic type languages allow you to build extensibility by leveraging duck typing instead of type inheritance.

Dependency Inversion Principle (DIP)

• High level modules should not depend upon low level modules. Both should depend upon abstractions.
• Abstraction should not depend upon details, details should depend upon abstractions

To understand the principle, the reference point should be structured and modular programming scenarios prior to OO. If I wanted to book a tour, the tour booking function would in turn call functions to book a flight, book a car and book a hotel. All four function implementations would be in the parlance of the definition - “details”. And anytime you had to change the way any of them worked it would require a relatively high amount of effort to do the changes. If you applied DIP in such a scenario, you would perhaps have a function say TripBooker which would in turn call methods on other classes which implement the methods in an interface / abstract class to book a flight, car and a hotel. The actual detailed booking logic would now be implemented in another class. Thus TripBooker now depends upon an abstraction which specifies the contract, and it becomes much easier to change or plugin the concrete implementations.

In the following example which is largely similar to (but not identical to) the example in Robert Martin’s article, we are having a Lamp with a button. All the class names from the example are retained. These are the two specific details - Lamp and ButtonImpl. The lamp has capabilities to turn on and off, but one would like to to move the exact mechanism of turning on and off (which might be hardware specific) into the detail but treat the ability to turn on and off as an abstraction (ButtonClient). Meanwhile buttons can have a variety of possible implementations one of which is ButtonImpl, which all share the essential characteristic that a button has one state with two possible values, can toggle between the states and have a reference to a ButtonClient to which they can communicate with.

Thus the model instead of having two details Lamp and ButtonImpl, with the details communicating with each other directly, one has the following design :

• Detail Lamp implements (depends upon) abstraction ButtonClient
• Detail ButtonImpl implements (depends upon) abstraction Button
• Abstraction Button has a reference to (depends upon) abstraction ButtonClient

Here’s the code in Java.

public class ButtonHappy {
    public static void main(String[] args) {
        Button button = new ButtonImpl(new Lamp());
        button.toggle();
        button.toggle();
        button.toggle();
    }   
}

public interface ButtonClient {
    public void on();
    public void off();
}

public class Lamp implements ButtonClient {
    @Override
    public void off() {
        System.out.println("Lamp turned off");
    }   

    @Override
    public void on() {
        System.out.println("Lamp turned on");
    }   
}

public abstract class Button {
    private ButtonClient client;
    public Button(ButtonClient client){
        this.client = client;
    }   
    public void toggle(){
        boolean newstatus = ! getStatus();
        if (newstatus) client.on();
        else client.off();
        setStatus(newstatus);
    }   
    public abstract boolean getStatus();
    public abstract void setStatus(boolean status);
}

public class ButtonImpl extends Button {
    private boolean status;
    
    public ButtonImpl(ButtonClient client) {
        super(client);
    }   

    @Override
    public boolean getStatus() {
        return status;
    }   

    @Override
    public void setStatus(boolean status) {
        this.status = status;
    }   
}

There’s probably no additional explanation required, so here’s the equivalent implementation in python.

class Lamp(object):
    def on(self):
        print 'Lamp turned on'
    def off(self):
        print 'Lamp turned off'

class Button(object):
    def __init__(self,client):
        self.client = client
    def toggle(self):
        status = self.get_status()
        if self.status : self.client.on()
        else: self.client.off()
        self.set_status(not status)

class ButtonImpl(Button):
    def __init__(self,client):
        self.status = False
        super(ButtonImpl,self).__init__(client)
    def get_status(self):
        return self.status
    def set_status(self,status):
        self.status = status

if __name__ == "__main__":
    btn = ButtonImpl(Lamp())
    btn.toggle()


    btn.toggle()
    btn.toggle()

We can again see here that one class that is missing but is no longer being missed ( ) is ButtonClient and the reason is the same as in OCP - Duck Typing. In terms of the definition of DIP itself - abstractions no longer necessarily have to be implemented. They exist implicitly in the detail classes but are no longer explicitly documented.

Liskov Substitution Principle (LSP)

What is wanted here is something like the following substitution property:
If for each object o1of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1 is substituted for o2 then S is a subtype of T.

Actually LSP is more of a test of whether two classes qualify to share an inheritance or is-a relationship, rather than a prescription for the design itself. In simple terms it sets out a requirement that if you define a new derived class, it should be possible to substitute an instance of a base class in a program (and though it doesn’t state it, an instance of a peer derived class ie a class in the same inheritance hierarchy) with an instance of a derived class without introducing any negative or unexpected side effects whatsoever. A rather simple example would be if we were to attempt to define a derived class of java.lang.String (say ConstrainedString) which now had an additional constraint (max characters - say 20 for a particular instance). To support this constraint a new runtime exception MaxLengthExceededException would need to be defined. It would have to be a runtime exception as you do not have the ability to add to the checked exceptions thrown by the java.lang.String class in ConstrainedString. Now programs that concatenated strings have no notion of having to react to a String overflow situation and this would create undesirable side effects in the program. Thus using LSP one could conclude that it would be incorrect to implement ConstrainedString as a derived class of java.lang.String.

I could not think of a good way of showing LSP in action in code since it is a test of candidate relationship and does not have any structural manifestation itself. However it is still a valid test to be applied in Dynamic Languages as well with one change as recommended in the post The Liskov Substitution Principle for “Duck-Typed” Languages .

What is wanted here is something like the following substitution property:
If for each object o1of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1 is substituted for o2 then S is a subtype of substitutable for T.

Duck typing introduces looser coupling than inheritance but a coupling that has no static checks whatsoever. So the application of LSP in a designer’s mind is a little more interesting. Because the modified LSP is now a rather obvious statement which in loose term says “If A can be substituted by B without any side effects then A is substitutable by B”, on the face of things, it is no longer such a useful principle. But if you explore under the covers, the real interesting part has now shifted from “B is a subtype of A” to “If A can be substituted by B”. In static type languages, the places where A was being used was clearly known and easily searchable or navigable into from IDEs. Because this is now so much more difficult even though you are no longer explicitly attempting to apply LSP, you are on your toes the time far more often to avoid the situations that LSP was setup to warn you about.

Interface Segregation Principle (ISP) :

Clients should not be forced to depend upon interfaces that they do not use

This is a little anticlimactic. The good news here is that thanks to duck typing, the client is actually now making the choice of what interface they choose to use. This is one principle that no longer needs to be explicitly applied. No more discussion required here.

Summary :
As we have seen duck typing does imply some changes to your class design. While the first three design principles continue to be relevant, their relevance is now a little different in your design process. It is important to be aware of these changes to adjust your design models in a more appropriate and idiomatic way. One would typically create lesser complex class hierarchies, especially with all the interfaces / pure abstract classes now no longer mandatory. Not only type information but even some of your abstractions are now less explicit.

After having worked with Java (and earlier C++) for a number of years, I have been working with Python for the last few months. Since I came to Python from Java, I thought it might be useful to share my experiences, which might be of interest to many programmers. This is not intended to be a language or feature comparison or something that contrasts the various advantages or disadvantages, but is intended to reflect on the softer aspects of how it feels to write Python programs. Hence I have refrained from including code snippets and other programming constructs in this post, but if you believe I am unclear in some of the comments I am making, do let me know and I shall be glad to update the post or write a new one as required.

Programming is Easier and Enjoyable

I think the most dominant impression from the last few months is that python does make programming feel a lot more easier and often more enjoyable. The feeling is not very different between riding a bicycle without gears then riding one with gears. In the latter case one just feels one can cover a lot more distance much more easily though any physicist will tell you the actual effort is not particularly different. It just feels like one has a much bigger toolbox (ie a wider assortment of tools) to work with and therefore the task seems simpler. Why do I think that way ? I believe the following features of python do help (in no particular order) :

• Concise Coding style : The code typically is much more concise, with much lesser verbosity
• Dynamic typing : You really do not need to worry about declaring data types and making sure the inheritance hierarchies especially for all the interfaces and implementations well laid out. The various objects do not even need to be in the same inheritance hierarchy - so long as they can respond to the method, you can call it. This is a double edge sword, but that doesn’t take away the fact that programming under dynamic types environment does seem a lot easier.
• Easier runtime reflection : Java seems to have all the reflection capabilities but I think these are just way too painful to use as compared to python. In python the entire set of constructs (classes, sequences etc.) are available for easy reflection. In case you need to use metaprogramming constructs, python really rocks.
• More built in language capabilities : Items such a list comprehensions, ability to deal with functions as first class objects etc. give you a broader vocabulary to work with.
• Clean indentation requirement : It took me about 2-3 days to get over it but, it seems that python code is much easier to read since if you do not indent it correctly it will be rejected.

Need to spend time on understanding how to write code in Python

One of the statements I have heard in different contexts is that Java programmers when they start using python write it as if they are writing a java program using python syntax (unpythonic in python parlance). This actually took me a fair amount of time to understand. I did look at a lot of other python code to attempt to understand this in greater detail. I realised that python offers many more capabilities than java and that as a person who had written java code earlier, I was able to be immediately productive using the constructs in python which mapped onto the equivalent constructs in java easily. However it did take a long time to be able to start using the other constructs. One of the exercises that did help was to get away from the problem at hand and try to work on something entirely different (especially a program which had a strong algorithms element to it with complex data structures), and slowly review each line with how it might be better done in python.I realised it is easy to start coding in python but it takes some effort and time to start using the entire python toolbox effectively. While I reviewed other programs written in python, I did think there was one area where the prior exposure to java helped. Class design. I am not sure if this was an issue with the programs I read and thus there was an issue with the sample references I used. However while these helped me understand how to write code in python differently, I have a strong feeling that the programs I wrote were much stronger in terms of class design. There are many situations especially given the fact that python supports both function oriented programming and object oriented programming, where one wonders what is a better way to design the logic in a particular context. I thought it generally made sense to use proper class based design and use the functional constructs in situations which started getting a little loopy or algorithmic.

No Compile Cycle

Another thing I really enjoy about python is - no compile cycle (its implicit). So while in the middle of my editor, I could exit to the shell prompt, run the source immediately without trying to deal with an ant script in between which compiles java code and then recycles the web application. This has boosted my net productivity quite a bit.

Productivity

So are the statements that one can be 5 to 10 times more productive in python supported by my experience. While I haven’t gotten through the full life-cycle yet, prima facie I do believe 5 times does seem like a good number. However I would introduce the following caveats. It will not happen for your first project .. more like your second project onwards, primarily since it does require a lot of effort to start understand how to write pythonic code and that does take away a lot of those benefits initially. Secondly it does not factor in refactoring. Point refactoring is much much easier in python, but bulk refactoring is much much more difficult since automatic refactoring is tough. Thus when I am refactoring, I sometimes feel I was able to get it done faster in python, but in many other cases I thought I ended up taking a lot more time.

Switched to python completely ?

Not really. But I feel happy I have more options available to me. One thing that really sucks is the poor refactoring capabilities. This is unlikely to be an issue with python alone and is likely to be an issue with all dynamically typed languages. However if you want to use dynamic languages, be prepared to spend some more effort during refactoring since many of the automatic refactoring capabilities you may have gotten used to, may not be available. The other issue is performance. Java wins hands down on performance by a big margin. If I have to ever get back to another project where performance was particularly important and the primary constraint in performance was not network or disk IO but was likely to be the CPU, and it would be acceptable to substantially reduce development productivity in order to get the performance gains, I shall be found to be writing Java code for certain. The next project I work on, I am likely to “first” evaluate whether python fits the bill and switch to other languages if I do not find it appropriate enough.

Interesting post in the Agile Journal, “Software Testing in an Agile Environment”. Great article and a reflection on how the software testing function would be influenced in an agile environment. IMHO, the author does a nice job, but perhaps could have gone a little bit further in terms of exploring how the development and testing functions could get merged within the same set of people, and this post attempts to do the same. Some comments I have are as follows.

But, for the QA professional an Agile approach causes discomfort - In the ideal world they would have a ‘finished’ product to verify against a finished specification. To be asked to validate a moving target against a changing backdrop is counterintuitive

This is no different than developers adapting themselves from a situation where they would’ve expected a set of clean requirements and/or design specifications. To be asked to develop a moving target against changing backdrop is counter-intuitive as well.

For some, the role of QA is now questionable citing Test Driven Development (TDD) as the key to testing. But, what is most important is that QA is directly involved in the agile scrums all the way through, to be an integral part of the team designing the tests, at the same time as the requirements and solutions evolve.

Absolutely.

1. “You only need to unit test - TDD testing is sufficient”

For the vast majority of commercial developments this simply isn’t true. Even strong proponents of agile development recognize the need for their armory to include a range of testing techniques.

Here the assumption is that TDD is in some way meant to constrain itself to unit testing. While most examples of TDD do show unit testing, I haven’t actually come across an opinion which says TDD is limited to unit testing. In fact this is one of the most important aspect to be understood if one needs to place QA in the right perspective in TDD environments.

TDD stands for test driven development. These tests could be white box or black box, unit or integration. Once it sinks in that TDD encompasses all forms of testing, it is easier to work out the workflow for the delivery teams. The most obvious implication is that the QA function needs to work very closely with development in deciding the tests of all types up front (prior to writing code). Given the agile proclivity for executable code and executable tests as the specification for code and unit tests, the same would apply to functional and acceptance tests as well. One issue here is usage of Record and Playback tools. I suspect these can no longer be used in a TDD environment simply because these assume the application under test is completely ready when writing the tests. (Record and Playback tools can be used as supporting tools additionally .. but they cannot meet the expectations of a Test Driven Development workflow). Coming back to executable tests whether these be coded in jUnit or HttpUnit or whatever one’s choice of tools, these can still be coded upfront and checked in before the code gets written to ensure that the code that eventually comes out meets the QA expectations.

Therein to me lies the biggest adaptations that testing functions shall need to adapt to. For most small to medium projects, in a TDD environment, the members (who perform both development and testing roles) write the tests and the code thus obviating the need of a separate testing department. Agile Environments will drive development and testing skills to be merged within the same set of people. While this increases the learning curve, it also substantially increases productivity since the entire communication stream between development team and testing team is largely eliminated, the occasional blame game between the two now just needs to get resolved within one persons mind.

However there are situations where I think classification of members into developers and testers might be called for eg. where a particular product needs to be tested on a v. wide range of hardware / software platforms independently, where writing the executable tests calls for a high level of capability and skill that is of a specialised nature etc. For such teams, their rhythm will now need to adapt. Instead of developers writing code and delivering it to testers, testers shall write tests and deliver these to developers who shall then need to write code to pass them. I believe this is the most important leap that the author of the article wasn’t able to make.

3. “Developers write the tests using open-source tools, so we no longer need testers, or automation tools”

Professional testers fulfill a different and equally valid role from their development colleagues.

Heres the tradeoff . Separating developers and testers has benefits of increased specialisation but levies high costs in terms of communication overheads, resolution of mismatched understandings, and crossing departmental boundaries in some cases. Merging the responsibility into one team member (development + testing) helps reduce all this costs even though the same person now needs to be trained in terms of having both the skillsets. Importantly, it clearly identifies where the buck stops (and eliminates the occasional ping pong between development and testing) resulting in developers verifying their own code with a much higher level of vigour resulting in a higher quality. I would submit there would be a large number of projects out there (but certainly not all) where merging these functions and removing the separation between developers and testers makes sense.

Often, TDD projects have at least as much test code as application code and, therefore, are themselves software applications. This test code needs to be maintained for the life of the target application.

All the more reason why development and testing responsibilities could get merged.

7. “Developers have adequate testing skills”

If testing was easy everybody would do it and we’d deliver perfect code every time. Sadly, many organizations that invest in increasing a developer’s coding skills and provide them with the latest integrated development toolsets, fail to see the need to develop the equivalent testing skills for their QA team, or provide them with the tools to do the job either effectively or efficiently.

Even better option - train team members to do both development and testing and equip them with the appropriate tools.

An independent testing team serves as an objective third-party, able to see “the big picture”, to validate the functionality and quality of the deliverable. While developers tend towards proving the system functions as required, a good tester will be detached enough to ask “what happens if…?” When you include business user testing as well, you are more likely to have a system that is fit for purpose.

Why would you want to have developers who cannot see the big picture. This is a training issue not a separation of function issue. Moreover I have always found it a bit strange that one can trust one person to write the code but not believe in him to be able to think through various combinations. I would argue that it is more cost effective to have the members simultaneously trained in “how to write code …” and “what happens if …” thinking.

10. “Developers and Testers - are like oil and water”

Since the dawn of time there has often been a “them and us” tension between developers and testers. This is usually a healthy symbiotic relationship which, when working correctly, provides a mutually beneficial relationship between the two groups resulting in a higher quality deliverable for the customer.

I would submit that this has been a high cost separation. Probably at least half (and perhaps many more) of the projects out there would benefit from these responsibilities getting merged in terms of increased productivity and quality. A small project should only have customers (or their representatives or proxies if not available directly) for specifying what is needed and conducting acceptance testing, and members (who have abilities to develop and test) who service these needs. Larger projects may choose to have a more classified team (domain experts, developers, tool builders, graphic designers, testers etc.), however such classification does and will make them lesser agile (as in the english word, not the methodology) to some extent.