/var/log/mind

I must confess back in the mid 90s I was an avid usenet user. Over the years I have watched the overall development community online interaction landscape changing (mailing lists, web based forums, blogs, wikis, blog aggregators, voting systems etc.). While each change seemed a little questionable at times, these have brought in a tremendous vibrancy in the developer interactions even though I still feel like composing a Usenet GaGa song (just like Freddie Mercury did the Radio Gaga thing). While I wondered as each change was introduced, what it implied, I never felt strongly enough about it to speak loudly. Till Now.

Take a look at stackoverflow. Its seems to be getting a tremendous developer community traction. I wish it the very best and believe it has been successful in many ways in being able to attract so much traffic. But as I move around it, I just feel a little odd. I also feel a little old perhaps. While the following may seem to be a rant it is not. I end the post with a suggestion which I suspect might be helpful to people who might be having similar opinions.

1. Content gets deprioritised : Here is a list of things that appears in a bigger font than the main content / discussion itself.
   • Vote Up Arrow
   • Vote Down Arrow
   • Number of votes received
   • Asked When
   • How many people answered it
   • How many votes it received
   • How soon it was answered
   • When was it last updated
   • Advertisement blurb text

   Don’t get me wrong. Its not like I am against this information. But so much metadata about the question / answers and displayed so boldly ? I want to come into the site review some questions quickly, review their answers, maybe post a few questions or answers myself, and want to go back on do my work. So much additional information is quite distracting and imho leads to much more inefficient browsing sessions.

2. Voting is too much in your face : I think voting is useful. Voting allows a democratic mechanism of content prioritisation. But this is way too much in the face. There’s no way I can focus on the content without getting distracted by the votes per answer. One of the limitations of voting is that there is no way to understand why someone liked or disliked your answer. Good old usenet or other non voting channels of communication forced everyone to chip in descriptively which made for a much richer content.
3. Am I in an election ? Since voting has been made so prominent, try hard as you can it is difficult to get away from trying to find out how many votes your earlier responses got, and how many points you have accumulated. I suspect it also starts driving your behaviour somewhat (more on that later) in a narcissistic pattern which starts shifting your focus from communicating ideas to pitching self (its a subtle difference but a strong one nevertheless). I don’t mind pitching at all, in fact I do it in multiple places including in my blog. But pitching self can be too distracting when carried forward to every question and answer.
4. Am I a pin up board ? Whats this thing with badges ? Thirty different types of them to be precise. This number is probably ten times what might be useful (Novice, Seasoned, Guru .. or something similar). Why am I even getting these ? And why does how many badges I have need to be carried around wherever my user id is shown. Again, these toys are taking us away from content.
5. Where is the spirit ? The thing I find lacking the most is the “spirit” of discussion. Now I don’t particularly encourage the use of expletives or do it myself, but I can be exceptionally strong at times in my communication. Some discussions on usenet, online forums, blogs etc. get quite interesting not because of the content alone, but also because of the personalities behind the same. I do not generally see any strong language (not that I particularly regret it per se but its an indication of the lack of spirit). I would qualify most discussions I have browsed as quite good on content but extremely tepid in spirit. This could be due to the contributors keeping an eye on the number of votes their answers are getting (almost feels like a extremely busy place of worship at times instead of a noisy bazaar that I personally would so much want to look forward to).

So should we do away with all this stuff ? Absolutely Not. All I am requesting is that we maintain a sense of balance in prioritising the different aspects of online communication. While there are other of ways in which StackOverflow is superior, I am sure one of the sites it could look as a reference in the context of the points I mentioned above is reddit.com.

Reddit comments page has voting based prioritisation, karma points. But these are never in your face and they don’t overshadow the content. You can actually safely browse through the page and ignore these. There are no other distractions. On any particular topic, one can cover many many more opinions in reddit much faster than in stackoverflow, and in any given browsing session, one can cover many more pages / questions in reddit than one can in stackoverflow.

I suspect there is a good likelihood, my opinions are in the minority but a minority that is not miniscule in size. I am opening a request in stackoverflow uservoice. If you feel supportive of the thoughts expressed above, please support by voting for the request : Provide alternate skin / theme for content focused browsing.

After having worked with Java (and earlier C++) for a number of years, I have been working with Python for the last few months. Since I came to Python from Java, I thought it might be useful to share my experiences, which might be of interest to many programmers. This is not intended to be a language or feature comparison or something that contrasts the various advantages or disadvantages, but is intended to reflect on the softer aspects of how it feels to write Python programs. Hence I have refrained from including code snippets and other programming constructs in this post, but if you believe I am unclear in some of the comments I am making, do let me know and I shall be glad to update the post or write a new one as required.

Programming is Easier and Enjoyable

I think the most dominant impression from the last few months is that python does make programming feel a lot more easier and often more enjoyable. The feeling is not very different between riding a bicycle without gears then riding one with gears. In the latter case one just feels one can cover a lot more distance much more easily though any physicist will tell you the actual effort is not particularly different. It just feels like one has a much bigger toolbox (ie a wider assortment of tools) to work with and therefore the task seems simpler. Why do I think that way ? I believe the following features of python do help (in no particular order) :

• Concise Coding style : The code typically is much more concise, with much lesser verbosity
• Dynamic typing : You really do not need to worry about declaring data types and making sure the inheritance hierarchies especially for all the interfaces and implementations well laid out. The various objects do not even need to be in the same inheritance hierarchy - so long as they can respond to the method, you can call it. This is a double edge sword, but that doesn’t take away the fact that programming under dynamic types environment does seem a lot easier.
• Easier runtime reflection : Java seems to have all the reflection capabilities but I think these are just way too painful to use as compared to python. In python the entire set of constructs (classes, sequences etc.) are available for easy reflection. In case you need to use metaprogramming constructs, python really rocks.
• More built in language capabilities : Items such a list comprehensions, ability to deal with functions as first class objects etc. give you a broader vocabulary to work with.
• Clean indentation requirement : It took me about 2-3 days to get over it but, it seems that python code is much easier to read since if you do not indent it correctly it will be rejected.

Need to spend time on understanding how to write code in Python

One of the statements I have heard in different contexts is that Java programmers when they start using python write it as if they are writing a java program using python syntax (unpythonic in python parlance). This actually took me a fair amount of time to understand. I did look at a lot of other python code to attempt to understand this in greater detail. I realised that python offers many more capabilities than java and that as a person who had written java code earlier, I was able to be immediately productive using the constructs in python which mapped onto the equivalent constructs in java easily. However it did take a long time to be able to start using the other constructs. One of the exercises that did help was to get away from the problem at hand and try to work on something entirely different (especially a program which had a strong algorithms element to it with complex data structures), and slowly review each line with how it might be better done in python.I realised it is easy to start coding in python but it takes some effort and time to start using the entire python toolbox effectively. While I reviewed other programs written in python, I did think there was one area where the prior exposure to java helped. Class design. I am not sure if this was an issue with the programs I read and thus there was an issue with the sample references I used. However while these helped me understand how to write code in python differently, I have a strong feeling that the programs I wrote were much stronger in terms of class design. There are many situations especially given the fact that python supports both function oriented programming and object oriented programming, where one wonders what is a better way to design the logic in a particular context. I thought it generally made sense to use proper class based design and use the functional constructs in situations which started getting a little loopy or algorithmic.

No Compile Cycle

Another thing I really enjoy about python is - no compile cycle (its implicit). So while in the middle of my editor, I could exit to the shell prompt, run the source immediately without trying to deal with an ant script in between which compiles java code and then recycles the web application. This has boosted my net productivity quite a bit.

Productivity

So are the statements that one can be 5 to 10 times more productive in python supported by my experience. While I haven’t gotten through the full life-cycle yet, prima facie I do believe 5 times does seem like a good number. However I would introduce the following caveats. It will not happen for your first project .. more like your second project onwards, primarily since it does require a lot of effort to start understand how to write pythonic code and that does take away a lot of those benefits initially. Secondly it does not factor in refactoring. Point refactoring is much much easier in python, but bulk refactoring is much much more difficult since automatic refactoring is tough. Thus when I am refactoring, I sometimes feel I was able to get it done faster in python, but in many other cases I thought I ended up taking a lot more time.

Switched to python completely ?

Not really. But I feel happy I have more options available to me. One thing that really sucks is the poor refactoring capabilities. This is unlikely to be an issue with python alone and is likely to be an issue with all dynamically typed languages. However if you want to use dynamic languages, be prepared to spend some more effort during refactoring since many of the automatic refactoring capabilities you may have gotten used to, may not be available. The other issue is performance. Java wins hands down on performance by a big margin. If I have to ever get back to another project where performance was particularly important and the primary constraint in performance was not network or disk IO but was likely to be the CPU, and it would be acceptable to substantially reduce development productivity in order to get the performance gains, I shall be found to be writing Java code for certain. The next project I work on, I am likely to “first” evaluate whether python fits the bill and switch to other languages if I do not find it appropriate enough.

An interesting technology related issue has cropped up in India recently especially in the context of what is being suggested as apparent misuse of unsecured WiFi networks by terrorists.

To sum up, there seems to be some evidence linking the usage of unsecured WiFi networks by terrorists. This has led to a situation where Telecom Regulatory Authority of India (TRAI) seems to be requiring that all Wireless Networks be made secure and is directing the Internet Service Providers to ensure the same for all such networks connected to the internet. The ISPs claim (quite reasonably in my opinion) that they are unable to be able to ensure the same since people may connect up PCs from their home via WiFi routers and that they cannot monitor the status of such routers which are under the control of the respective families. More information on this issue here : TRAI plans to prevent WiFi abuse. There is also a thought to declare all unsecured networks are illegal which IMHO places a tremendous burden on families especially those who may not be as computer savvy.

There clearly are competing interests here :

(a) The government needs to ensure that all internet traffic is traceable. There might be a whole number of privacy concerns here but these are perhaps not relevant in this context, since there already seems to be a sufficient infrastructure to trace traffic to the IP addresses, what seems to be missing here is the traceability into the using party given the fact that anyone can apparently easily use a unsecure wireless network connected to the net.
(b) Many families may have installed such WiFi routers for convenient access to the net from their homes. They may not be particularly educated in all cases sufficiently to know and understand the necessity of and how to secure their domestic WiFi networks. Besides declaring unsecure networks as illegal may make it mandatory to require every family sufficient education on how to secure the network. (The current proposal seems to be putting this onus onto the ISPs)

While the following suggestion is unlikely to help the currently installed base of WiFi routers, here’s a thought. Can we not require and mandate that each such WiFi device require some minimum form of reasonably strong password verification before allowing WiFi based routing (ie. one can connect to the device to set the password in the first place), and manufacture this requirement into the firmware of each device (ie. there should be no reasonable way to bypass it). In other words, such devices will implicitly disallow any WiFi based networking except between a PC the router itself when no password has been set (The connection needs to be allowed so that the password could be set to some basic minimum required strength). It could then perhaps makes sense to declare devices which do not conform to these norms as not valid under the law. Moreover It is likely that current WiFi router providers could be required to issue a firmware upgrade for their existing models where such upgrades are feasible.

Its an interesting situation where common good needs to be balanced with that of individuals. This is probably not the only solution. Maybe such Wifi devices already exist and I simply haven’t been following the space adequately enough. I am also certain that strong password authentication is only a start and possibly there are other measures to secure the networks further .. but at least its a start which is unlikely to be controversial either from a user’s or vendor’s perspective. Maybe there are other solutions. While the issue is currently being debated in India, its probably relevant to all the corners of the globe. Any thoughts ?

You can see both the windows task manager and the google chrome task manager. Google chrome seems to be consistently under reporting the memory utilisation. Is there something more than meets the eye (at least mine) or is there some part of the memory which Windows Task Manager reports (eg. that used by the kernel on behalf of the user process) but google chrome doesn’t or is it Windows over reporting memory usage ? Note : I captured the snapshot after pausing for some time to allow the memory reporting to catch up in both the tools.

Any thoughts or conjectures ?

PS: Sorry the rows are not in the same order in the two tables .. so you might have to mentally sort them to figure out the matching ones.

Update :Just realised that since the overall browser memory utilisation is now captured and shown separately from that of a particular page, it just makes measuring memory utilisation of a particular web page so much more accurate and easier. I wonder how long before the following things happen :

• Architects / Leads start laying down “thou’s web page shalt not consume more than ## kBs in Google chrome” to the developers
• My favourite web page is better than yours since it consumes lesser memory (and I have the snapshot to prove it)

This is not a post in defense of outsourcing. However every so often I come across a rant like Why Outsourcing Sucks and much as I emotionally connect to such posts which talk about poor software quality, I find the rant and the underlying thoughts a highly misguided exercise. This post attempts to place outsourcing in an overall economics based perspective and explores some of the future trends in how they are likely to impact software quality.

Before attempting to explore and define problems around outsourcing, lets first understand why it exists.

There is something right about outsourcing.

Clearly there has to be something right about outsourcing for it to have come so far. Stating that outsourcing results in crappy code or shitty quality and therefore is wrong questions a collective wisdom of the market, a collective wisdom that while I have reservations about, I will always place above mine. So how does outsourcing help ?

• Outsourcing has supported the wall street model of seeking fiscal efficiencies : Wall street encourages companies to be fiscally responsible and financially competitive. Quality of products is in relative terms an irrelevant parameter. While toplines and bottomlines are to be maximised, quality is meant to be at a level “the market will bear”. For a variety of reasons, including all listed below, outsourcing has helped support better fiscal efficiencies.
• Outsourcing helps reduce costs (in the short to medium term at least) :This is a no brainer. However it is often assumed to be the primary reason for outsourcing, which I am not so sure of.
• Outsourcing helps reduce risk :It allows companies to have lesser people on their rolls, and allows them an ability to ramp up or ramp down under growing or shrinking market scenarios in a much easier way.
• Outsourcing increases a customers negotiability (initially):Anyone who has attempted to get software delivered from an internal development team and an external vendor will most certainly tell you that the negotiating power is much higher at least in initial days with a external vendor than with internal development teams.
• Outsourcing allows ability to service an exploding demand :Lets face it, for all the jobs that outsourcing might have transferred overseas or made redundant, these are a small fraction of the total number of jobs that outsourcing has created. There is only one reason for that - exploding demand for software development and maintenance. If the outsourcing surge had not happened all the participating economies individually would’ve been worse off than they are today (though the extent of the difference is quite different across various economies).
• Outsourcing has relatively little to do with quality :Outsourcing is a fiscal phenomenon. Management is a function of business imperatives. Quality is a result of engineering processes. Business imperatives impact engineering processes far more than outsourcing does. These imperatives are not driven by outsourcing alone, but by many other yardsticks which drive outsourcing itself. If you worry about ways to improve software quality, focus on these business imperatives and not on outsourcing alone. Some of these imperatives are quarterly / annual targets, get it to the market at all costs, we need to get in there first we will fine tune the offering later etc. etc.

Relationship to Software Development.

I shall now explore some of the factors I touched on above in the context of Software Development.

• Cost Reduction : Cost reduction is achieved in many different ways.
  • Transferring jobs to lower paying markets : This is probably the easiest understood mechanism. There’s enough written on this topic for me to add anything more.
  • Transferring jobs to lesser skilled people : If you want to reduce costs, one way is to hire lesser skilled people thus having to pay them less. One particularly nasty way this manifests itself is that under such scenarios, seasoned and experienced personnel either need to work across a larger number of projects simultaneously (architects, consultants etc.) or need to move out of development and into management (a much much more common occurrence at least in India). There are very real economic underpinnings to why 8 or more out of 10 developers in India are likely to say project manager when asked what would they like to be in the next 5 years. Whichever way you look at it, the average skill level deployed on a project in engineering terms does go down.
  • Increasing % utilisation of associates : Higher focus on costs leads to a higher focus on billable utilisation. This reduces the amount of time available to learn, study, research, prototype etc.

  To summarise, cost reduction leads to a quicker movement of senior engineers into management roles, greater breadth of projects for senior engineers who continue to be in software development, and lesser ability to focus on learning and research.
  

• Risk Reduction : Since outsourcing allows customers to ramp up or down easily, it increases the onus on the vendors leading to a large number of people “on bench”. This allows project head counts to be increased easily when required. It is sufficiently known that adding 10 people to your project is not going to increase your productivity ten fold of that you would get by adding 1 person. However the ability to ramp up given the bench strengths tempts managers into quickly adding as many people to get the next release out faster. Since people get added in a hurry under situations when the next release is needed quickly, net productivity per person suffers, and the increased difficulty in doing the skill ramp up of all the people rapidly results in quality sliding down.




• Increased customer negotiability :While this is definitely helpful in many ways in terms of helping lower costs, there are some serious side effects as well. I would argue that even the best amongst us is unlikely to be unimpacted by a higher negotiating power. The simple law of nature is that more the negotiability, use it. Vendors are of course perfectly aware that vendor negotiability is directly proportional to the time spent on the project and that it will improve over time. When customer negotiability gets used to an extreme, in a competitive scenario, the vendor simply says yes to expectations that may be unlikely to be fulfilled. While questionable, this is a practice which has grown to an extent where quite often customers factor it in and is often defended by if I don’t say yes, my competition will. This sets in motion a negative spiral that only the very brave can even attempt to try to compensate. And as the saying goes fools rush in where angels fear to tread.




• Exploding demand :One of the big reasons why quality has been falling is this. As the demand explodes, it is harder and harder to maintain average skill levels. It is but obvious today that average skill and training levels are far lower today than 5 or 10 or 15 years ago (and they have been continuously falling). The ability to deploy skilled and trained personnel on a consistent basis to the exploding demand is non existent in the current combined planet wide educational systems. Falling quality is as much a function of our educational and training systems to keep pace with demand as with anything else.




• Business Imperatives :In todays economic environment, managers are rewarded much more for meeting the quarterly and annual targets, than 5 year targets. We have created this system and it is likely to be around for quite some time to come. There are good reasons why it exists, it allows for much better fiscal risk management, and results in superior capital investment decisions. For software developers however it creates a conflict, because good software does not get created in a quarter (and if it were to, the target would just get moved by adding much more functionality in the same quarter given the exploding demand). Even if it saddens us deeply, probably the most appropriate way to deal with it is to recognise its merits in fiscal and economic terms and deal with it as a reality that is unchangeable in the short to medium term.

Having said that there is an increasing awareness of the cost of maintaining poor quality software. Customers and Vendors are both finding it increasingly difficult to keep a lid on costs when maintaining and extending such software. For companies where this cost is a high percentage of their total costs, this is threatening their very ability and viability to stay competitive. I think most business managers today understand this and are trying to find a way out of this but only with varying degrees of success.

Why criticism of outsourcing is occasionally misplaced

I have often heard of issues such as differing time zones, cultural mismatches, communication snafus, skill diferences etc. as big issues with outsourcing. These are all indeed solvable problems. It would be a great underestimation of ingenuity of today’s business enterprises if we assume or believe that they are less than capable to do so. I submit that the reason companies haven’t gotten around to working on these issues as much as they could’ve is simply because these problems are not high up on the radar. As I have attempted to explain, managers are concerned with a plethora of issues, software quality being only one of them. If vendors were to find their business drying up, due to poor software quality, some of which is as a result of aspects of outsourcing as listed above, believe me vendors will fix the problem pronto. They have the skills, they have the ingenuity, but the economic drivers are not strong enough today. This is not a problem with outsourcing per se but with relative importance of economic drivers.

Another difficulty is in what is perceived to be quality. There is sometimes a difference in how quality is perceived across the development community and business managers. For a developer poor quality is when the software does not work as requested or as stated or as designed for. For a manager, poor quality is when the customer turns away and declines to offer repeat business due to unhappiness with the software.

Finally this issue about lower skill sets. I completely agree at the risk of making a broad generalisation that the average entry level skills in the software development market are far lower in India than in the United States. But this is a manifestation of the fact that India got into the software act a lot later and when the development demand boomed, it fortunately or unfortunately had the ability to deploy a large number of software developers even though at lesser entry skill levels. (The statement above is likely to hold for other highly developed and developing economies as well). If hypothetically India and other developing countries had not got into this act, I have no hesitation in believing that given the booming demand, average skill levels in the industry would’ve gone down sharply in the developed countries as well. This is again an economic event much more a function of the population levels, wage disparities and output of the educational systems. To blame this on outsourcing is to shoot the messenger.

So whats next

Let me summarise some of the points made so far :

• Businesses will continue to attempt to be more financially prudent :What this means is that while this prudence benefits us in general terms, it will continue to attempt to increase productivity and drive down costs.
• Businesses will continue to be focused on quarterly and annual results :Love it or hate it, this system is here to stay for some time. If we can adjust software development to be in tune with it, probably all of us can be better off. Methodologies such as short iterations and frequent releases, YAGNI, continuous refactoring are likely to be better supportive of this environment.
• Exploding demand for software :This is likely to continue unabated at least for a few more years or a decade. A large part of the world doesn’t yet invest heavily into software development yet but if food and energy demands are any indication, as more countries start consuming more software, this demand growth is likely to accelerate.
• Crappy software is better than no software :This is probably the most difficult and perhaps a most profound thought for many developers. As software demand grows, and as our worldwide ability to keep pace with it shrinks, crappy software is likely to deliver positive economic results compared to lack of software. Combine this with focus on short term results and you will realise that customers are more likely to tolerate buggy software than ever before in times to come, and if customers tolerate it, vendors will find ways to test these tolerance levels.

So will things never improve for software quality ? I think two factors will help, one in the really long term and one in the medium term.

• Software demand will start slowing and productivity improvements and better training will catch up eventually : While it does look very far away, it will happen one day. We are finding ways to create better tools, improved methodologies, offer better training etc. Software demand will peak one day and ability to service it will catch up. Thats when customers will start being intolerant of poor quality software. Thats when vendors will most certainly deliver good quality software.
• SAAS model will help take the load off the demand supply gap :In terms of ability to create an economic impact per developer, the Software as a service - SAAS model works far superior to custom application development. (For the purposes of this discussion a SAAS offering is expected to be simultaneously used by a large number of customers). Development costs are only a small part of the total revenues in case of SAAS. However customer biteback in case of defects is much stronger due to a larger number of customers using the same software instance. Thus SAAS will take some pressure off development cost reduction, focus on a higher quality threshold from day one, and take some pressure off the demand supply gap. This to me is the biggest positive side of things in the days to come from a software quality perspective. However even this will take some time to play itself out. When that does happen, SAAS is likely to be the biggest demand killer in terms of the demand for software developer head counts on a worldwide basis.

If you are a developer or user who is frustrated with the quality of software, there are the following things you can do :

• Recognise that some of the current situations (problems ?) are here to stay and be prepared to deal with them : If you can accept it (and it might be unacceptable for many) it will allow you to deal with the issues in a more rational and non emotional way.
• You can choose to focus on buying / supplying high quality software : There certainly will be some customers and vendors who will go down this path. Try to identify customers / vendors / partners who share this belief system. Even if the entire industry quality levels don’t improve substantially, your individual experiences are likely to be better. However there is an economic impact of this choice and make sure you can absorb that by figuring out a way to offer a compelling value proposition to your customers. Individual developers who are particularly upset about quality levels should be willing to work for companies with a higher commitment levels to quality along with the concomitant adjustments to individual growth path and economic outlook if any.
• Move to SAAS wherever feasible :I am not suggesting this out of any particular fascination for SAAS. There is built in argument in SAAS which is based on sound economics which encourages high quality software development. And while arguments based on emotional or moral merits are tough to win, arguments based on economics win by themselves. Developers wanting to work on high quality software development are better likely to find their needs satisfied with SAAS vendors.

PS : Lest it be misconstrued, let me clarify I am often anguished and occasionally offended by the state of software quality. I do not intend to justify poor quality at all. However I have learnt that that before solving a problem, it is important to frame it correctly. This post is an attempt to frame the current situation appropriately in economic terms and not in either emotional or moral terms since that has to be the first step in dealing with it.

Interesting post in the Agile Journal, “Software Testing in an Agile Environment“. Great article and a reflection on how the software testing function would be influenced in an agile environment. IMHO, the author does a nice job, but perhaps could have gone a little bit further in terms of exploring how the development and testing functions could get merged within the same set of people, and this post attempts to do the same. Some comments I have are as follows.

But, for the QA professional an Agile approach causes discomfort - In the ideal world they would have a ‘finished’ product to verify against a finished specification. To be asked to validate a moving target against a changing backdrop is counterintuitive

This is no different than developers adapting themselves from a situation where they would’ve expected a set of clean requirements and/or design specifications. To be asked to develop a moving target against changing backdrop is counter-intuitive as well.

For some, the role of QA is now questionable citing Test Driven Development (TDD) as the key to testing. But, what is most important is that QA is directly involved in the agile scrums all the way through, to be an integral part of the team designing the tests, at the same time as the requirements and solutions evolve.

Absolutely.

1. “You only need to unit test - TDD testing is sufficient”

For the vast majority of commercial developments this simply isn’t true. Even strong proponents of agile development recognize the need for their armory to include a range of testing techniques.

Here the assumption is that TDD is in some way meant to constrain itself to unit testing. While most examples of TDD do show unit testing, I haven’t actually come across an opinion which says TDD is limited to unit testing. In fact this is one of the most important aspect to be understood if one needs to place QA in the right perspective in TDD environments.

TDD stands for test driven development. These tests could be white box or black box, unit or integration. Once it sinks in that TDD encompasses all forms of testing, it is easier to work out the workflow for the delivery teams. The most obvious implication is that the QA function needs to work very closely with development in deciding the tests of all types up front (prior to writing code). Given the agile proclivity for executable code and executable tests as the specification for code and unit tests, the same would apply to functional and acceptance tests as well. One issue here is usage of Record and Playback tools. I suspect these can no longer be used in a TDD environment simply because these assume the application under test is completely ready when writing the tests. (Record and Playback tools can be used as supporting tools additionally .. but they cannot meet the expectations of a Test Driven Development workflow). Coming back to executable tests whether these be coded in jUnit or HttpUnit or whatever one’s choice of tools, these can still be coded upfront and checked in before the code gets written to ensure that the code that eventually comes out meets the QA expectations.

Therein to me lies the biggest adaptations that testing functions shall need to adapt to. For most small to medium projects, in a TDD environment, the members (who perform both development and testing roles) write the tests and the code thus obviating the need of a separate testing department. Agile Environments will drive development and testing skills to be merged within the same set of people. While this increases the learning curve, it also substantially increases productivity since the entire communication stream between development team and testing team is largely eliminated, the occasional blame game between the two now just needs to get resolved within one persons mind.

However there are situations where I think classification of members into developers and testers might be called for eg. where a particular product needs to be tested on a v. wide range of hardware / software platforms independently, where writing the executable tests calls for a high level of capability and skill that is of a specialised nature etc. For such teams, their rhythm will now need to adapt. Instead of developers writing code and delivering it to testers, testers shall write tests and deliver these to developers who shall then need to write code to pass them. I believe this is the most important leap that the author of the article wasn’t able to make.

3. “Developers write the tests using open-source tools, so we no longer need testers, or automation tools”

Professional testers fulfill a different and equally valid role from their development colleagues.

Heres the tradeoff . Separating developers and testers has benefits of increased specialisation but levies high costs in terms of communication overheads, resolution of mismatched understandings, and crossing departmental boundaries in some cases. Merging the responsibility into one team member (development + testing) helps reduce all this costs even though the same person now needs to be trained in terms of having both the skillsets. Importantly, it clearly identifies where the buck stops (and eliminates the occasional ping pong between development and testing) resulting in developers verifying their own code with a much higher level of vigour resulting in a higher quality. I would submit there would be a large number of projects out there (but certainly not all) where merging these functions and removing the separation between developers and testers makes sense.

Often, TDD projects have at least as much test code as application code and, therefore, are themselves software applications. This test code needs to be maintained for the life of the target application.

All the more reason why development and testing responsibilities could get merged.

7. “Developers have adequate testing skills”

If testing was easy everybody would do it and we’d deliver perfect code every time. Sadly, many organizations that invest in increasing a developer’s coding skills and provide them with the latest integrated development toolsets, fail to see the need to develop the equivalent testing skills for their QA team, or provide them with the tools to do the job either effectively or efficiently.

Even better option - train team members to do both development and testing and equip them with the appropriate tools.

An independent testing team serves as an objective third-party, able to see “the big picture”, to validate the functionality and quality of the deliverable. While developers tend towards proving the system functions as required, a good tester will be detached enough to ask “what happens if…?” When you include business user testing as well, you are more likely to have a system that is fit for purpose.

Why would you want to have developers who cannot see the big picture. This is a training issue not a separation of function issue. Moreover I have always found it a bit strange that one can trust one person to write the code but not believe in him to be able to think through various combinations. I would argue that it is more cost effective to have the members simultaneously trained in “how to write code …” and “what happens if …” thinking.

10. “Developers and Testers - are like oil and water”

Since the dawn of time there has often been a “them and us” tension between developers and testers. This is usually a healthy symbiotic relationship which, when working correctly, provides a mutually beneficial relationship between the two groups resulting in a higher quality deliverable for the customer.

I would submit that this has been a high cost separation. Probably at least half (and perhaps many more) of the projects out there would benefit from these responsibilities getting merged in terms of increased productivity and quality. A small project should only have customers (or their representatives or proxies if not available directly) for specifying what is needed and conducting acceptance testing, and members (who have abilities to develop and test) who service these needs. Larger projects may choose to have a more classified team (domain experts, developers, tool builders, graphic designers, testers etc.), however such classification does and will make them lesser agile (as in the english word, not the methodology) to some extent.

A lot of posts talk about how caching can improve performance and how one can use tools such as memcached for the same. It is a great tool, and I am certain it does wonders, but I do wonder if it and its usage is getting too much press at the cost of other caching options.

This post talks about the various dimensions that should be carefully examined before deciding upon the caching strategy. I would like to argue that the most difficult part of caching is not necessarily learning or selecting the caching tool but the detailed analysis of the application, its architecture and its data flow, that actually precedes the caching tool selection. I also submit that once this analysis is conducted and the various dimensions as described below well understood, the actual cache design then becomes a relatively simpler and lesser risky activity. Before getting into the dimensions of cache design, there is some important groundwork to be done.

Note : I am referring to data caching. I am not referring to optimisation of opcodes / byte codes / instruction sets etc. which is a different class of caching.

Know your application well :

Let me emphasise this. If you want to decide on the most appropriate caching strategy, you need to have a sufficiently good understanding of your application. This is relatively easy if you are enhancing the caching capabilities in an existing application or are rebuilding an existing application. But if you are building a completely new application, some of the data listed below may not be easily available and you may need to project a little bit after understanding the requirements and a high level data / logic flow. One of the things you really need to have either a clear understanding or at least some reasonable projection of, are the various data elements in the application, how frequently will they be accessed and modified and given the processing flow, which elements make sense to be cached. You will also need to have a good understanding of how critical is it to have the most current data (eg will a 1 min. stale data be acceptable) and the transactional semantics around the data processing (are strict ACID semantics a must ?).

Note that for the rest of this post when I mention data, I shall be referring to cachable data.

Know your performance targets :

Unless you are clear about your performance targets, it will be quite difficult to work out the appropriate strategy. I would suggest you should have clear targets defined (for a specified hardware) before proceeding down this path. Some times the target specifies very ambitious hardware which you may not have access to in early stages of the development. Try to convert these targets into appropriate targets for the hardware you shall be working with (with some reasonable margin of safety).

Know your architectural guidelines / preferences :

Will you be using a single server or multiple ? When handling future growth would you prefer to scale up or scale out ? What languages will you be using ? Would you prefer to use a single database instance or would you want to create vertically partitioned shards ? As we shall see, these are extremely important considerations before getting into cache design. Sometimes the decision making is not sequential, so you may end up revisiting these topics once you get into your cache design.

Dimensions

We shall now explore some of the various dimensions that influence cache design.
Continue reading »

Google just launched “Knol” today (announced here on their blog). Back in january, I had written up a blog post - Will the knol be a knowall ?, so I was immediately curious to get a first hand experience. Here’s my fresh off the oven first impressions of Google Knol in no particular order.

• Current topics focused on healthcare : Most existing content seems to be focused on health care topics (at least the featured ones seem to).
• Anyone can author a knol : If you have a google id, you can start writing up a knol. The blurb suggests that knols are authoritative but there doesn’t seem to be any verification required for authoritativeness. If you can write a blog post or a wiki page - you can write your own knol.
• Multiple knols can refer to the same topic : While any person can author a knol, multiple authors can create different knols all referring to the same topic.
• Collaborative editing is supported (not the default): It is possible to create a knol and then invite additional authors / reviewers.
• Name Verification is supported : Google has provided for name verification for the author. However for now it seems to be enabled only for US residents.
• Nice user interface : The UI actually seems much nicer looking than most other google assets. It of course continues to offer nice usability like all google properties do.

I went ahead and wrote up my first knol Comparing and Contrasting Knols, Wikis and Blogs, just to get the feel of writing a knol. Here are my observations on a knol after writing one.

• If you can write a blog or a wiki you can write a knol. : It doesn’t take any more editing skill set to write a knol than it takes to write a blog post or a wiki. (Hardly surprising).
• The focus is similar to writing an article : When you are writing a knol, somehow one feels pushed to write a relatively comprehensive article. Its possible to write a knol of a few lines / one paragraph, but I just suspect we won’t see too many of them.
• Revisioning is supported : Revisioning is automatic and supported. Readers can actually go back and read your earlier revisions or even run a diff across them. (Hmmm .. can’t casually write something stupid and then just go back and update it with something lesser stupid).
• Separate field for adding references : There’s a separate field for adding references / citation. In Wikis / blogs - we just do it inline I guess. Kind of brings a more formal flavor to writing which requires these to be listed at the end.
• Its like a time independent blog, or strongly individualized wiki : If you are writing a blog, but seem to be essentially writing long articles which are of an enduring nature, maybe a knol might be a more useful platform. Similarly if a number of authors are collectively contributing many articles on your wiki but each maintaining a set of articles independently, again a knol might be a useful thing to look at.

Implications on corporate environments
Finally to come to my favourite topic of attempting to visualise how knols could get used in corporate intranets. I really couldn’t think of knols adding much more to corporate intranets than wikis could. The existing format for wikis and blogs is in my opinion quite adequate for internal and possibly collaborative publishing requirements of corporate intranets. However I suspect that knols actually might be a nice way to publish external facing content from subject matter experts. I think corporates could look at implementing knols on their external facing web sites which could contain content authored by their CXOs or domain experts. The focus on author and his capabilities is likely to be better suited to this environment and may allow the content to be projected with a sense of strength that a casual wiki or perhaps even a blog is unlikely to be able to project. But for that someone needs to start writing a knol engine first.