Category: management

Software development is about the middle and not just the end

Posted by Dhananjay Nene – January 14, 2010

Software development often gets similar treatment as other disciplines when it comes to matters of management. Often at a cost. And the cost comes from the fact that, the learnings and best practices gleaned from a number of disciplines are hard to apply in the field of software development, without understanding and dealing with some of the essential differences. And those differences and their implications are what this post explores.

There was an interesting post recently Why program with continuous time?.

Today I read a confusion that I’ve heard many times before about continuous time, which I’d like to bring up, in part because I love continuous time, and in part because there’s a much broader underlying principle of software design.

I don’t see why the lack of continuous streams leaves a “gap”. In the end all streams are discrete

“In the end”, yes. Just as in the end, numbers are displayed as ascii numerals. However, programming is more about the middle than the end, i.e., more about composition than about output. … Similarly, continuity in space and in time is better for composition/modularity, leaving discreteness to the output step.

The post above is completely unrelated to management. Its talking about a programming style called Functional Reactive Programming. But an idea it eloquently brings out is programming is more about the middle than the end, i.e., more about composition than about output. And thats a thought that I would like to expand upon.

While arguably in all disciplines the middle is also important, I would emphasise that it is particularly more so in software development. And that stems from a number of uncertainties that abound in software. The requirements are often not available in adequate detail, and to the extent available, are not stable. Moreover solutions to these problems are not all thought through upfront. Not because people don’t do their homework – its simply because it is actually extremely difficult to visualise all the moving parts, and then anticipate and resolve all the challenges. In fact many of the challenges don’t show up until you start programming. Thus programming often is an exercise in identifying surprises, resolving them, working out algorithms, discovering flaws with them, fixing these flaws only to find more surprises. Rinse and repeat. Basically its a significant exercise in uncertainty management. In such a situation, estimation results in underestimates since these are based on visible not actual challenges and surprises. The solution that does get applied with some success is to use historic estimation errors to compensate current estimates. (Note that thats not always obvious since the estimate could have the compensation already built in).

To add to the problem there is usually no Bridge 2.0, Lathe 2.0 or a Marketing Campaign 2.0 (and to the extent there might be some, these are far less dependent upon the quality of the work that went into the 1.0). Some programmers are continuously focused on 1.0, while many are thinking about the 10.0 even as they work on the 1.0. Optimising for 1.0 alone may not be the right way to approach an initiative. In the good old design methodology one attempted to anticipate and build for a lot of future changes. This requires a lot of extra complexity and development in terms of layering, parameterisation, open extensibility etc. In conventional agile methodologies the focus is much more on the 1.0, but there is a deliberate and definite cost introduced in terms of creation of a large number of automated test cases. These automated test cases is the assurance premium to be paid, in order to provide the comfort to undertake the necessary refactoring challenges when moving from 1.0 to 2.0. Thus whichever way one looks at it, the 1.0 is always suboptimised to allow for a more optimal 2.0, 3.0 etc. (note: I am referring to 1.0 figuratively as the current initiative and higher version numbers to future initiatives building on the current one). But selection of the necessary suboptimisations, with a view to optimise the overall sequence, requires a substantial understanding of the software development processes, the code, the tools etc.

Finally programming has a far far higher human element involved than other disciplines. After all there’s hardly any raw material dependency and the tooling is often a desktop, the cloud and set of open source libraries. Thus it is subject to the vagaries and frailties of human behaviour. Coordination, Communication, Productivity, are all held hostage by this fickle human element. While programming is not unique in this respect, achieving high team productivity is as much a function of morale, motivation, and communication as perhaps any other discipline. Thus progress is often non linearly proportional to such soft factors.

I believe there’s some benefit in evaluating software development management styles with how sports teams are managed. In sports, team performances are rarely consistent. Moreover even though team playing skills are critical, raw talent is also very important. But the analogy is most appropriate when one evaluates the game. The game is often a series of surprises reacted to by immediate on the fly action. Thats where adaptability and ability to counter attack surprises, can be effectively brought to bear. And in sports as in software development, player morale and motivation needs to be enhanced even as the player arrogance or dysfunctional behaviours are contained.

Since different sports are structured differently let me clarify the roles here. In some cases the manager and coach are different. In others they are vested in the same person. In my mind the manager’s role is resource management, broader strategy planning, impediment removal and stakeholder communication. The role of the coach is to enhance team skills, ensure fitness and training, and in case of many sports actively guide the team in refining the on field strategy when the game is in progress. Now lets consider the players. In most cases the players are living the game. They are in the middle. A substantial portion of their interesting professional time is spent in the middle. And when they are doing that, they are continuously dealing with a set of constant surprises, even as they broadly attempt to achieve the deliverables required for the end. A slight difference between sports and software development is that in case of sports the time is fixed and the score is variable. In case of software development at least in the non agile world, the score (feature set) is fixed and the time therefore becomes the variable. But the interesting takeaway from this analogy is that once the game starts, the ends become the background and the game dominates. The players and the developers are all living in the middle.

Why is this particularly relevant ? It is so because it helps us understand the implication of management styles in software development teams performance. If one wants to optimise – one needs to very well understand the middle and not just the ends. And (to come to the point) thats where I have a big issue with software developers, who end up imagining that as they rise up in the hierarchy, they somehow can stop worrying about coding and quality assurance, and instead start focusing only on managing people and deadlines. Thats a mistake. Since the game is played in the middle, you need to be able to help your team play the game. And you need to do it as a coach or a captain. For that, so long as you want to manage an ongoing game, you need to keep on playing and keep on being either sufficiently conversant about whats happening in the middle or be the captain and play the game with the team. Thinking that somehow you’ve grown in stature enough to not worry about the tactics, and focus on things increasingly strategic might actually make you unhelpful. The game requires skill and continuous adaptation – and you will need to be able to play, until you one day decide, you are no longer interested in managing the game but only the periods between.

The non playing managers who attempt to manage the game on the other hand display some interesting smells. They stop understanding the sport. They start believing that since they had played some particular sport many years ago, its easy for them to now manage any other sport. They are unable to appreciate the actual game play that happens. So when required to score a particular number of goals in 90 minutes (say in soccer), they start pushing in more team members (alas, this is a luxury not available to sports teams). Now on field coordination starts getting more difficult. The newer members being relatively junior are the ones who start getting intercepted more often (thus the team keeps on losing the ball more frequently). If the team luck is all run out, the same managers also attempt to run the tactics and gameplay selection. This is based on some fairly old set of experiences. The spectators (customers) start seeing far more confusion on the field. And the objectives change from playing the game well to just scoring the required number of goals. And in the end the spectators may get to see the number of goals promised, but only through a confused dissatisfying game, and that too sometimes only after many bouts of extra time. And soon enough once the non playing managers realise they are unable to effectively control the game play, they try to manage other variables. But really whats starting to happen is that they are hurting the gameplay, and sometimes instead of managing the game, they start managing the spectators with the game on autoplay. Suboptimisation maximised. Alas, sometimes the spectators also get used to this and start forgetting what a good game used to be like.

To summarise, ends are good to have, ends are important, ends provide the background urgency. But software development is not so much about managing the end goals as much as managing the middle. If you want to build good software and help your team build good software, make sure you continue to focus on how software is built not just the quantitative targets. Make sure you live and enjoy the middle. More often than not, the ends you achieve eventually could be amongst the best possible ends that were practical – but more importantly the customers will also go home satisfied. So long as you keep your eye on the ball and don’t lose sight of the middle. And if you want to view it a bit philosophically, in the end we are all dead. Only the middle matters.

Authors Note : This post has been substantially revised from its earlier version from a syntax, grammar and punctuation perspective. And the philosophical note (the last line) was added later.

Service Oriented Architecture is primarily about business and not technology. Bollocks!

Posted by Dhananjay Nene – October 26, 2009

There’s quite a few times I’ve heard / read a gross oversimplification of architecture in reference to business and technology. And while I believe I understand the ‘essential cause’ which drives such a simplification, I’ve often felt quite frustrated at the resultant impression thats provided by such a simplification. In many ways and forms, it boils down to the statement (not exactly the same since I’m not quoting directly), quite similar to the one below :

Service Oriented Architecture is primarily about business and not technology

This also reflected in the recent SOA Manifesto which states as its very first described value :

Business value over technical strategy

Allow me to straight away start picking some holes into this :

  1. Anything that a business does – whether it is soa, software architecture, building architecture or simple plant and machinery design, to the extent (which is exactly 100%), technology serves the business goals, all technology activities (and non-technology as well) are at the end of the day about achieving business objectives and therefore about business. So why single out architecture? And even more so why single out SOA?
  2. Architecture is also about business. But its not the same as saying its primarily about business and not so much about technology. For a moment lets step away from Software/Hardware Architecture and look at Building Construction Architecture. The legendary creation of Ayn Rand – Howard Roark, for all his eccentricities and seemingly portrayed egocentric and egotistic behaviour did meet the test of business objectives to the extent of making the residents of his creations extremely satisfied. And at no point would you gather the impression that he in any manner put construction technology to any secondary position to his business context and objectives. At the end of the day thats what architecture is. It is not about making one of business or technology more important than or subservient to other. Its about effectively mapping the two to provide a strong technology solution appropriate to the business needs.

I suspect one of the important causes here is that people have forgotten that the A in SOA stands for architecture, and therefore shorn of architecture, business and technology can be seen to be competing in a non win-win form.

So if SOA stands for Service Oriented Architecture, then I must submit that architecture is the art of getting the two working together. And I am of the opinion that an exercise suggesting one is more important than other is an exercise in a field unrelated to architecture. I suspect many practicing software architects will agree with this. I suspect Ayn Rand wouldn’t disagree as well.

The best amount of polyglotism is that you can manage successfully

Posted by Dhananjay Nene – September 28, 2009

Polyglotism was one topic I got into some discussions on twitter earlier this week. So this is to just pen down some of my thoughts on the same. Polyglotism is increasingly being talked about, ever more so since Java started moving from the realm of being a language into a runtime environment to host a number of languages. Yet polyglotism is not new. Most programmers have been mixing their favourite languages with SQL for ages. The advent of the web introduced HTML and javascript as additional languages that web programmers needed to know to be able to successfully write web programs. At the same time other tools started appearing which started to reduce your requirement to know these languages. Object Relationship Mappers could help you get out of writing SQLs (at least so they claimed :) ) in many cases. And one of the features of GWT is to ensure that you work primarily on only one language.

But the goal of this post is to neither encourage nor discourage the use of polyglotism. It is to suggest that at the end of the day even if polyglotism really stimulates the technogeek sensors in us, it needs to be evaluated not in technology terms, but in business and economic terms. A point that is also partially supported by The plague of polyglotism. Unless we use tools like GWT and hibernate to keep us monoglots, (which is likely to be a very small proportion), polyglotism is the default state that we shall operate in. So its not worthwhile to fight that trend per se. The important aspect is to ensure that we understand the costs of supporting each new language in the mix. And the reason to do so is to evaulate whether the benefits of adding the language outweigh the costs in the medium to long term. So here are some of the costs (in no specific order)

  • Syntax and libraries : This is probably one of the lowest costs even though it is the most apparent. For every language we add to the mix, generally for a long running project, at least three developers (if you have reasonable risk management objectives) need to learn the new language syntax and the libraries they work with. Along with this is the cost of books, training courses and materials etc. (yes for many enterprises, many developers but those purely self motivated will need to be formally provided all of these).
  • Idioms and Philosophies : Knowing the language is not adequate. You need to understand its philosophy, its design intent and its typical idioms. Being able to use a language to write code is not the same as being able to use it idiomatically. And while that may not constrain you in over small code bases in the short term, over a period of time for reasons such as performance, consistency and ability to understand other libraries that you may reuse, you will be forced to deal with these aspects of the language as well. The book Dreaming in Code talks about how Java programmers who started writing python could not write idiomatic python code, and thus a fair degree of code had to be redesigned or rewritten. And that is by no means the only such instance. I suspect similar trends are likely to be found if java programmers move to writing ruby or scala code. And its a trend you might have likely observed when people comfortable with writing relatively simpler languages such as Visual Basic or PL/SQL or even more complex languages like C++ moved to writing Java. Knowing the language is not the same as knowing it idiomatically – and that takes time. Which means – do not take on high risk activities when you absorb a new language, it will take a few iterations for your team to absorb the new language – far far more time than it will take to read a book or attend a training course.
  • Skills to manage multiple idioms in one head : Unless you are going to have different programmers for different languages, the programmers working on multiple languages at a time will not only have to deal with the multiple idioms, but will need to concurrently apply them. In my experience, this is not a trait we are born with, and it takes some time and experience with it. It will take some time for them to learn that the best practices are not identical across various languages. It will take even more time to figure out why they are different. So there is a good likelihood, you will not be in a situation to have all your developers work on all the languages simultaneously – especially some of the Junior ones.
  • Skill portfolio management will become more difficult : For typical enterprise or ISV teams, the managers need to ensure adequate skill availability across all teams. As the number of languages in use increase, this management process will become harder. Which means there is an automatic economic disincentive which rises is in magnitude, each time you add yet another language to the mix.

So is polyglotism good or bad ? I submit such a value attribute cannot be assigned to polyglotism per se. It needs to be really evaluated in your context, preferably beyond a project window into the medium or a long term. What is indeed obvious is that each new language will bring in some very strong capabilities and aspects where it is superior to the other languages in the mix. Equally obvious is that each new language will bring in associated costs. Just like there is no perfect language, there is no perfect number of languages that make sense in every context. Just like you do your homework in terms of language selection, make sure you do the same for your language portfolio not just in technology but in management and economic terms as well.

Why should I switch to Scala ?

Posted by Dhananjay Nene – August 17, 2009

This post is a role-play and does not reflect my individual opinion about scala accurately. I am convinced about the capabilities and features of Scala along with the fact that it deserves the mantle of a long term replacement for Java. However language adoption goes beyond technical capabilities, and this post is a speculation on what a typical manager might be dealing with when attempting to decide whether to switch to Scala.

So I have been reading a lot about Scala lately and even opinions about how it will be a long term replacement for Java. I’ve also read some interesting writeups about Scala adoption such as On Scala’s Future and A Tipping Point for Scala. While I used to code a lot, my responsibilities today require me to interact with and address a lot of issues including those faced by our customers, our development teams and also engage with my peers and superiors on many other difficulties bedeviling our organisation. This gives me little time to try out Scala. I know I should be doing that, but sincerely I do not have the time. So I rely on the feedback of my team, the trade journals and other influential architects within and outside my organisation.

I have heard about many developers switching from Java to Python / Ruby. However I have heard of relatively only a smaller number of large Java shops which have done the shift – most of the switch stories I’ve heard reflect a smaller sized teams. I can feel the excitement Scala has generated amongst the development teams – the brevity, the functional programming model introduction, the exciting stuff being done concurrently et al. I have no doubt that, given so much excitement it must really be a good language.

To introduce my organisation – it is one of those shops which service many projects concurrently. Given the tremendous business and growth, I must confess we do not always have the luxury of being able to hire the most top notch talent. We do have a lot of projects we use Java for, and thats a language our customers are comfortable with. I’ve had some of the senior people check out Scala to gain some feedback into the language. But at this stage I must say I am inclined to evaluate the shift but not convinced enough to do so. I am sure that I could if convinced drive the change to Scala incrementally. However my fear stems from the fact that if things don’t turn out well, despite all the great advice I’ve received – its going to be my rear end on the line. So here’s some of my concerns regarding evaluating the shift to Scala and there are many of them, so some of you might be able to help me through this thought process.

  • Functional Programming : I’m sure in many ways it rocks. But my guys tell me they are not sure how to use it in the typical bread and butter applications which read from database, do some processing and write back to the database. Does Functional Programming help me in this context ? Will my team scale into being able to write functions with no side effects assuming thats a desirable goal ? What if they tie themselves up in knots and my release to the customer is risked ? I can’t afford that. Is functional programming even desirable in such contexts ? So I am not sure if in these contexts I should just ditch functional programming and work with just normal imperative programming capabilities of Scala. I am so confused, and afraid.
  • Different Syntax : While Scala runs on the JRE, its syntax is very different from Java. From what I could gather, it is much easier for a Java programmer to read (make sense of) simple Python code than to read Scala code. Is it true ? So even if I do get compatibility in terms of the runtime environment, would I be picking up a language that is syntactically so different a language that it would involve a substantial relearning curve ? I remember when we had to learn Java and Javascript. For better or for worse these were indeed relatively minor modifications of the C/C++ syntax, compared to what I sense as the syntactic shift between Java and Scala. Am I wrong ? If so, could you help point me to resources which help me understand that Scala code is not much different than Java ?
  • Sample code : Guys, I need your help. I need to see some good sample code. Some code which reflects how a typical application is architected, designed and programmed in Scala. And I don’t need it for a complex multi threaded actor based processing – I just need to see simple J2EE server based departmental applications maybe a simple recruitment tracking or library maintenance application. If I find a good one, I’ll just take it and give it to my team and say – there, thats how we’re largely going to build it, and even if we make a few changes along the way we at least have a reasonable template that we can build from.
  • Dumbed down environment : I remember my great adventures with C and vi and make. But my team today is very different. They want great IDEs. They must have syntax highlighting, autocompletion and nice refactoring capabilities. If I ask them to move, some of them might be excited about the change and be willing to overcome these short term hurdles. But there are some of them who will not be keen to do so and may be disinclined to support such a shift. And at the end of the day my ability to conduct this shift is a function of my ability to carry a large proportion of them along with me. Even when I considered a shift from svn to git, the IDE support was a big issue even though quite obviously git capabilities were really exciting. I couldn’t push along that change, and in this case we are talking of changing the language.
  • Is this a good time to shift to Scala ? I remember the early adopters of Java from 1996 thru 2001. While they gained a lot of experience, JRE and J2EE really matured only post JRE 1.3. Scala seems to be coming out with so many enhancements so fast, I am not sure if it has stabilised. I am told there is a 2.8 coming out in a few months. So if I train my team and Scala continues to change rapidly will I have to keep on retraining my team regularly ? And what about the customers I take to production. Will the frequent upgrades mean I end up supporting multiple customers on multiple versions of Scala ? Maybe Scala is stable but it would be helpful for someone important enough to make a clear statement that there are no new major shifts anticipated anytime soon and that these version shifts are likely to be no faster than the JRE version upgrades (which were fast enough).
  • Support from peers and superiors : I remember the day I decided to shift to Java. What made the move easy for me was the sheer fact that Java was a big paradigm leap away from the then dominant C++. Not only was it cross platform with binary compatibility thrown in for good measure, Sun ensured that it made all the right noises to appeal to the enterprise architects and all the business managers. I see the senior developers in my team clamouring for the shift to Scala, but my peer managers and my superiors don’t display even the fraction of the enthusiasm they displayed during the Java shift. The implication for me is that the risk cover I get when I order the shift is far lesser than what I had when I made the move to Java. Which means if things don’t quite work out well, I’m really going to be screwed.
  • Business friendliness : I understand all the nice talk about the technical excellence of Scala. But I really need to translate all these great language features into a projected ROI that I can use to convince others about. So I would like to see actual case studies of applications that were moved to Scala and what impact it had on the time and cost so that I can use it to compute my ROI. And what scares me is that learning curve may risk the initial applications long enough to push my breakeven point of shifting to Scala well beyond a 12 month and perhaps even a 24 month period. I fear things might not be as difficult but in absence of known studies, I am likely to lean towards projecting a worst case scenario rather than an optimistic one.

So folks, I am asking for your help. And while a lot of you may think that people like us who balk at the thought of limited IDE support are wimps, please remember that 80% of us don’t fit into the top 20%. And if you would like Scala to be popular, you need us as much as we need you. And if you are not too sure, please remember Lisp and Smalltalk are great languages as well.

Most american graduates are unemployable because …

Posted by Dhananjay Nene – June 23, 2009

Came across this post Top Indian CEO: Most American Grads Are ‘Unemployable’ leading to some substantial discussion (not all of it impassionate). I would first like to clarify that I am assuming the content in the post to be accurate based on the fact that other Indian sites are reporting that “No response or clarification from Nayar has as yet been issued”. Moreover it is not targeted against Mr. Nayar or HCL but to the suggested thought process which attempts to make process orientation so critically important overriding many other qualities.

What perhaps hasn’t been clearly stated but is important is that that the standards for employability here are set by the employing organisation based on its context and preferences which may be different from standards required by many other organisations and I would submit that the statement ought to be looked at in that context as well.

To quote from the article :

The official wanted to know why HCL, a $2.5 billion (revenue) company with more than 3,000 people across 21 offices in 15 states, wasn’t hiring more people in his state. Vineet’s short answer: because most American college grads are “unemployable.” (In fairness to HCL, the company recently announced plans to open a delivery center in another state, North Carolina, and invest $3.2 million and hire more than 500 employees there over the next five years under a Job Development Investment Grant.)

Many American grads looking to enter the tech field are preoccupied with getting rich, Vineet said. They’re far less inclined than students from developing countries like India, China, Brazil, South Africa, and Ireland to spend their time learning the “boring” details of tech process, methodology, and tools–ITIL, Six Sigma, and the like.

As a result, Vineet said, most Americans are just too expensive to train–despite the Indian IT industry’s reputation for having the most exhaustive boot camps in the world. To some extent, he said, students from other highly developed countries fall into the same rut.

So why are some graduates are unemployable ?

Some of these are folks who are working on creating tons of new languages (Java, C#, Scala, Closure, Erlang, Python etc.), Operating Systems (Linux, Mac and Windows), frameworks for web applications, clustering, fault tolerance and scalability, schemaless and distributed databases to ensure availability and fault tolerance at a wide scale, competing messaging architectures that each service a particular problem differently etc. etc. Would you believe it some have set up clusters of  hundreds of thousands of machines and service search requests very rapidly. And some others are working on creating innovative and disruptive models in social networking, application integration, peer to peer networking etc. Many are working out next generation mobile technologies including building (not building on) android, iphone OS and the palm (whatever OS it has). And so many in their spare time are spending a whole bunch of time creating open source software, blogging and micro blogging about all the work they have done and sharing it with the wide world so that they can learn off it. And if these are consumer and technology stories, there are a whole bunch of people building critical business infrastructure architectures and frameworks and solutions as well. Even after you complete reading this post and come back to it a month from now, I shall still be busy trying to figure out what exactly India has been able to deliver that can challenge this. Of course let’s not miss out on the fact that most of these activities are conducted by fairly small sized teams. (I am aware some of the examples I quote could have non-american heritage, but that per se is not likely to detract from the issue)

Let’s call a spade a spade. When you have large or very large software construction and maintenance contracts, there are multiple ways to deal with it. In my experience, many Indian companiess have honed to a fine art the process of recruiting, deploying and juggling large armies of programmers to service such expectations (not all Indian companies are based on the same model). Many companies have indeed managed to acquire, retain and expand customer engagements through a combination of technology and business innovation. And they have done a good job of it in the context they’ve defined for themselves. In my limited understanding and experience it is not technology innovation, creativity or extraordinary technical prowess that is at the top of the list of skills that get deployed (though these are indeed found in sufficient levels across the board and some of the prowess can be impressive) – it’s the clockwork project management and business methodologies, techniques and innovation (and even these do result in projects with delays) with their reliance on “another brick in the wall” that gets the customer serviced.

So, if true, the “employable” was perhaps in that context.  If so based on the earlier two paragraphs, I submit it reflects positively and negatively on the employing organisation. Perhaps even more so than the graduates themselves. I do wonder if “incompatible” would have been a better choice of word. And I wonder what it reflected more poorly upon, as well who should be feeling more sad about it.

Disclaimer : These views are mine, mine alone and should not be puported to be shared by any other people or companies I’ve been associated with in the past, present or the future.

Is a large corporate making money off open source or open standards an oxymoron ? In a Sun / Java Context

Posted by Dhananjay Nene – April 22, 2009

The recent acquisition of Sun by Oracle reignited a thought that had been going through my mind for a long time. It simply boils down to whether a large corporate can make money off open source or open platforms. Now quite obviously it in itself is not a truism. But the point remains that large corporates which have become large in the conventional economy do find the going a little difficult when trying to make money off open software.

The way I perceived it, Sun was going through similar difficulties. The hardware business was delivering dwindling margins post the dot com bust, and the software business was under threat from upstarts such as Linux which offered a substantially similar software stack at near zero licensing costs. One of the crown jewels asset Sun had in its stable was Java. And while it was (and continues to be) a wonderful asset, it just was incredibly difficult to make money off it. Now java hasn’t been open source exactly for most of its life, but it was a sufficiently open stack nevertheless to cause the same difficulties that open source software would in terms of monetisation.

There’s an excellent blog post written a year and half ago by Michel Bauwens, Can the experience economy be capitalist? which does refer to some of the underlying issues. I suggest you do read it, but would like to quote a few points from it below.

First of all, in the field of the immaterial, we are no longer dealing with scarce goods, but with marginal reproduction costs and non-rival goods. With such goods, sharing does not diminish the enjoyment of the good, since all parties retain their ability to use them. The emergence of peer production shows a new form of creating value, that is in fundamental aspects “outside the market”. Typically, in commons-based production we have a common pool, accessible to everyone (Linux, Wikipedia), around which an ecology of business can form to create and sell scarcities (usually services and experiences). In sharing-oriented production (YouTube, Google documents), we have proprietary platforms that enable and empower the sharing, but at the same time, sell the aggregated attention (a scarcity), to the advertising market. Finally, in the third crowdsourcing mode, companies try to integrate participation in their own value chain and framework.

So the good news is that indeed business is possible. But I would like the readers to entertain the following proposition, nl. That:

1) The creation of non-monetary value is exponential

2) The monetization of such value is linear

In other words, we have a growing discrepancy between the direct creation of use value through social relationships and collective intelligence (open platforms create near infinite value through the operations of the laws of Metcalfe and Reed), but only a fraction of that value can actually be captured by business and money. Innovation is becoming social and diffuse, an emergent property of the networks rather than an internal R & D affair within corporations; capital is becoming an a posteriori intervention in the realization of innovation, rather than a condition for its occurrence; more and more positive externalizations are created from the social field.

Quite eloquently argued here that open platforms create near infinite value which is difficult to be captured by business and money. I still remember companies such as Borland, HP, Sun, IBM making a fair amount of money through selling C/C++ compilers and IDEs (I remember it used to be almost $5000 per seat). However this was in the days when the community capability of creating similar competing software stacks was only in its infancy. No longer so today. Now communities, open standards, open processes and open source are fairly well established sources of delivering asymmetrically significant capabilities at a fraction of the cost, a fraction which is made even smaller when the same is incurred by a small motivated group of individuals or small highly agile companies. There in lies the difficulty. Companies are trained to and built to deliver linear and symmetric capabilities in the context of costs they incur. However they have a relatively poor handle on monetising in scenarios where small teams can deliver exponentially asymmetric capabilities.

The blog post I referred to above, identifies the right area to look at in this context – Identify what is scarce and Monetise the scarcity. So when any software has a sufficiently large utility and can be managed through open processes to be able to satisfy a globally substantial demand at a fraction of the cost (eg. apache httpd), that particular capability (eg. static and dynamic file serving over http) is no longer scarce enough for a commercially focused company to make money out off. Thats why the money shifts where the scarcity is – how to leverage such software and put it to good use. This is where the existing commercial successes around open source are based on eg. RedHat, JBoss, IBM, Oracle etc. They monetise the support, training, services and consulting around such software. In many ways Sun could be argued to be a victim of its own success. It not only went much further than any other large corporate in terms of creating open specifications, it also contributed substantially to sufficient knowledge dissemination and tooling to allow many other individual, SME and large corporates to be able to compete with Sun in these very areas that Sun could’ve monetised. Sun thus created its own competition for monetising the scarcity that got created around Java even as many others cried out hoarsely that Sun simply wasn’t open enough. Will there be an incentive for Oracle to reverse that somewhat ? I suspect that could be a logical option if it focuses on ensuring a good ROI for Sun’s Java investments and assets and more so to be able to make itself more capable in the space around Java than the competition (ie. IBM).

So is a large corporate making money off open source or open standards an oxymoron ? Viewed narrowly yes. Because while it is possible for a small team to make some money with adequate ROI off open source or open standards, it is unlikely to be feasible for large corporates. They really need to figure out what is the new scarcity that gets created off such initiatives, come to a judgment that the opportunity arising from such scarcity is large enough and worthy of their time, attention and investment, ensure that there is no current available capability of anyone else disruptively delivering exponentially asymmetric value in that space, and finally occupy that space before other large corporates do.

It does boil down to the fact that while individuals and small companies may find it easier to monetise open source and open standards, the same is going to be generally tough for large corporates. Or as Michel Bauwens said in the blog post I referred to (in my opinion perhaps a little exaggerated but not entirely off the mark)

For all of this, we will need new policies, major reforms and restructurations in our economy and society.

But one thing is sure: we will have markets, but the core logic of the emerging experience economy, operating as it does in the world of non-rival exchange, is unlikely to have capitalism as its core logic.

Talk Slides : Programming Language Selection

Posted by Dhananjay Nene – March 29, 2009

Slides of my talk yesterday on programming language selection (from a technical and a business perspective)

Programming Language Selection

Background article (not exactly same but on a similar topic) : Improve your web based software development and maintenance ROI with dynamic programming languages

Talk Announcement : Seminar: Strengths and weaknesses of various programming languages – 28th March

Agility and/or Process Maturity : How do I perceive these terms and why these should not be confused

Posted by Dhananjay Nene – March 23, 2009

Agility and Process Maturity are two very different terms to me. I have been rather pleased with the results I have found after having started working with agile methodologies more than 4 years ago. That makes me a candidate for being a biased blogger. But it still does allow me the ability to state my views clearly on the topic after that explicit disclaimer.

Process Maturity (especially CMM and ISO oriented maturity) focus on ensuring consistent, measured and self improving quality of output. This is achieved through multiple checks and balances, and detailed documentation of processes, and regular audits to identify points of weakness and their redressal. At least to the extent I have been exposed to them, people are looked at as cogs in the assembly line (replaceable and faceless) and focus is on ensuring that the assembly line works at a consistent pace even as people move around from job to job (or from assembly line to assembly line). Thus the focus is on setting expectations at modest levels and ensuring that these expectations can be met even as various events occur which introduce risk into the process, since the process methodology and documentation is geared to help address many of these risks.

Agility on the other hand probably stemmed from some of the concerns around the non-people-centricity and heavy-documentation-focus of these process maturity models. It instead ended up focusing on Individuals and their Interactions and on accepting the necessity of Change and thus adopting it wholeheartedly as an aspect that is inevitable and thus best accepted as a feature rather than a bug. Thus rather than slowing down the delivery, it attempts to substantially speed it up by removing many process steps which may seem to be impediments to progress. While substantially reducing most control and measurement points such as documentation, reviews, audits, they replaced them with steps which allowed quality to be maintained by introducing higher skills and discipline into the people (test driven development, pair programming, daily scrum).

Thus if a development team tells me they follow a process maturity model, assuming they are being completely honest in that statement, it tells me that as a customer I can expect them to set a pace and generally largely stick to it with incremental improvements while shielding me from having to deal with many of the internal risks they battle with.

If on the other hand a development team tells me they follow the agility model and again assuming they are being completely honest, it tells me they have a lot of respect in their own capabilities and to their ability to adaptively deal with internal risk through better earlier risk detection and redressal (Test Driven Development / Continuous Integration etc.) rather than through process oriented controls such as Documents / Audits / Reviews etc. I would also generally expect a much much higher productivity and quality from the team but with a higher statistical variance.

Ceteris Paribus, as a customer I would feel that the Agile Team is likely to introduce more schedule risks in situations of higher person turnover or lesser developer disciplines, while the team that focuses on Process Maturity is more likely to introduce schedule risks in situations of changes in requirements or in situations where a very quick time to market is called for.

What is obvious to me is that these methodologies have been defined with very different attitudes. If we play a “one word that comes to your mind” game with the terms, its “power” with “agility” and “control” with “process maturity”. Both are important and desirable and play a role, but the slider is positioned differently in each of the cases.

It has been rather obvious that agile processes have been occupying a greater mindshare over the past 5 years (the same thing happened with process maturity methodologies in the decade before that). I had been concerned that, as with many other trends in Software, the word Agile might also start getting abused where people start using it quite differently, to leverage on its momentum and the mindshare. It has been happening for a while, but the article Strategies for Scaling Agile Software Development The Agile Process Maturity Model just blew me off. I have only the greatest respect for Scott Ambler and especially for all his writings on Persistence. But in this article all the agile methodologies have been clubbed together as “Level 1 development” and many of the rather heavy weighted processes some of which probably led to the formation of the agile alliance in the first place are listed in the “Level 2 (disciplined) delivery”. I must clarify that agile software development covers all aspects of lifecycle including testing and measurement and agile software methodologies deliver as well (contrary to what the article “seems” to suggest) and rather than building discipline into processes, reviews and audits it builds discipline in the people that form a part of that process.

Since agility and process maturity are based on different attitudes I can’t see how one says I am going to put both the things together unless one intentionally and quite deliberately repositions the slider at a level with offers lesser power than agility and also dilutes process control compared to process maturity. But thats not the impression I got from the article. It seems to suggest that one can take these “level 1″ agile processes add “level 2″ disciplines to it and then add some uncertain “level 3″ elements to them to make things better than where they stand today, something that I am a little skeptical about.

So when a development team tells me they are following the “agile process maturity model (APMM)” I am likely to believe that they are either (a) too confused, (b) want to appear to leverage the agile momentum while using the strict control regime or (c) successfully figured out a way to move their slider somewhere in between with the risk that they lost some of the best elements of the two methodologies while attempting to lose the worst.

Having said that, it will be interesting to read Scott’s future articles on topic and to follow this trend. I have been proven wrong in the past, and it could happen again for me to have to eat my words. But moving that slider between power and control to attempt to achieve the best of both seems a little risky, and you can be sure I am unlikely to be an early adopter for the fear of getting severely burnt.

Update :I forgot to add that amongst the many items listed in Level 2, those such as (throwaway) sketching, whole team, test-driven development (TDD), continuous integration, daily standup meeting are accepted aspects of typical agile methodologies whereas those in the level 2 methodologies which agilists often avoid as actively managed and maintained charts are not even listed eg. in case of RUP – use case diagrams, class diagrams, sequence diagrams, state diagrams. So what is the contribution of other methodologies such as RUP in this case is largely unclear. I do not know the other Level 2 methodologies so can’t comment on the same.