Why merging development and testing makes sense in a TDD / agile environment Is google chrome under / partially reporting RAM usage ?
Aug 14

Outsourcing does not suck. Our understanding of it does !

software Add comments

This is not a post in defense of outsourcing. However every so often I come across a rant like Why Outsourcing Sucks and much as I emotionally connect to such posts which talk about poor software quality, I find the rant and the underlying thoughts a highly misguided exercise. This post attempts to place outsourcing in an overall economics based perspective and explores some of the future trends in how they are likely to impact software quality.

Before attempting to explore and define problems around outsourcing, lets first understand why it exists.

There is something right about outsourcing.

Clearly there has to be something right about outsourcing for it to have come so far. Stating that outsourcing results in crappy code or shitty quality and therefore is wrong questions a collective wisdom of the market, a collective wisdom that while I have reservations about, I will always place above mine. So how does outsourcing help ?

  • Outsourcing has supported the wall street model of seeking fiscal efficiencies : Wall street encourages companies to be fiscally responsible and financially competitive. Quality of products is in relative terms an irrelevant parameter. While toplines and bottomlines are to be maximised, quality is meant to be at a level “the market will bear”. For a variety of reasons, including all listed below, outsourcing has helped support better fiscal efficiencies.
  • Outsourcing helps reduce costs (in the short to medium term at least) :This is a no brainer. However it is often assumed to be the primary reason for outsourcing, which I am not so sure of.
  • Outsourcing helps reduce risk :It allows companies to have lesser people on their rolls, and allows them an ability to ramp up or ramp down under growing or shrinking market scenarios in a much easier way.
  • Outsourcing increases a customers negotiability (initially):Anyone who has attempted to get software delivered from an internal development team and an external vendor will most certainly tell you that the negotiating power is much higher at least in initial days with a external vendor than with internal development teams.
  • Outsourcing allows ability to service an exploding demand :Lets face it, for all the jobs that outsourcing might have transferred overseas or made redundant, these are a small fraction of the total number of jobs that outsourcing has created. There is only one reason for that - exploding demand for software development and maintenance. If the outsourcing surge had not happened all the participating economies individually would’ve been worse off than they are today (though the extent of the difference is quite different across various economies).
  • Outsourcing has relatively little to do with quality :Outsourcing is a fiscal phenomenon. Management is a function of business imperatives. Quality is a result of engineering processes. Business imperatives impact engineering processes far more than outsourcing does. These imperatives are not driven by outsourcing alone, but by many other yardsticks which drive outsourcing itself. If you worry about ways to improve software quality, focus on these business imperatives and not on outsourcing alone. Some of these imperatives are quarterly / annual targets, get it to the market at all costs, we need to get in there first we will fine tune the offering later etc. etc.

Relationship to Software Development.

I shall now explore some of the factors I touched on above in the context of Software Development.

  • Cost Reduction : Cost reduction is achieved in many different ways.
    • Transferring jobs to lower paying markets : This is probably the easiest understood mechanism. There’s enough written on this topic for me to add anything more.
    • Transferring jobs to lesser skilled people : If you want to reduce costs, one way is to hire lesser skilled people thus having to pay them less. One particularly nasty way this manifests itself is that under such scenarios, seasoned and experienced personnel either need to work across a larger number of projects simultaneously (architects, consultants etc.) or need to move out of development and into management (a much much more common occurrence at least in India). There are very real economic underpinnings to why 8 or more out of 10 developers in India are likely to say project manager when asked what would they like to be in the next 5 years. Whichever way you look at it, the average skill level deployed on a project in engineering terms does go down.
    • Increasing % utilisation of associates : Higher focus on costs leads to a higher focus on billable utilisation. This reduces the amount of time available to learn, study, research, prototype etc.
    To summarise, cost reduction leads to a quicker movement of senior engineers into management roles, greater breadth of projects for senior engineers who continue to be in software development, and lesser ability to focus on learning and research.

  • Risk Reduction : Since outsourcing allows customers to ramp up or down easily, it increases the onus on the vendors leading to a large number of people “on bench”. This allows project head counts to be increased easily when required. It is sufficiently known that adding 10 people to your project is not going to increase your productivity ten fold of that you would get by adding 1 person. However the ability to ramp up given the bench strengths tempts managers into quickly adding as many people to get the next release out faster. Since people get added in a hurry under situations when the next release is needed quickly, net productivity per person suffers, and the increased difficulty in doing the skill ramp up of all the people rapidly results in quality sliding down.

  • Increased customer negotiability :While this is definitely helpful in many ways in terms of helping lower costs, there are some serious side effects as well. I would argue that even the best amongst us is unlikely to be unimpacted by a higher negotiating power. The simple law of nature is that more the negotiability, use it. Vendors are of course perfectly aware that vendor negotiability is directly proportional to the time spent on the project and that it will improve over time. When customer negotiability gets used to an extreme, in a competitive scenario, the vendor simply says yes to expectations that may be unlikely to be fulfilled. While questionable, this is a practice which has grown to an extent where quite often customers factor it in and is often defended by if I don’t say yes, my competition will. This sets in motion a negative spiral that only the very brave can even attempt to try to compensate. And as the saying goes fools rush in where angels fear to tread.

  • Exploding demand :One of the big reasons why quality has been falling is this. As the demand explodes, it is harder and harder to maintain average skill levels. It is but obvious today that average skill and training levels are far lower today than 5 or 10 or 15 years ago (and they have been continuously falling). The ability to deploy skilled and trained personnel on a consistent basis to the exploding demand is non existent in the current combined planet wide educational systems. Falling quality is as much a function of our educational and training systems to keep pace with demand as with anything else.

  • Business Imperatives :In todays economic environment, managers are rewarded much more for meeting the quarterly and annual targets, than 5 year targets. We have created this system and it is likely to be around for quite some time to come. There are good reasons why it exists, it allows for much better fiscal risk management, and results in superior capital investment decisions. For software developers however it creates a conflict, because good software does not get created in a quarter (and if it were to, the target would just get moved by adding much more functionality in the same quarter given the exploding demand). Even if it saddens us deeply, probably the most appropriate way to deal with it is to recognise its merits in fiscal and economic terms and deal with it as a reality that is unchangeable in the short to medium term.

Having said that there is an increasing awareness of the cost of maintaining poor quality software. Customers and Vendors are both finding it increasingly difficult to keep a lid on costs when maintaining and extending such software. For companies where this cost is a high percentage of their total costs, this is threatening their very ability and viability to stay competitive. I think most business managers today understand this and are trying to find a way out of this but only with varying degrees of success.

Why criticism of outsourcing is occasionally misplaced

I have often heard of issues such as differing time zones, cultural mismatches, communication snafus, skill diferences etc. as big issues with outsourcing. These are all indeed solvable problems. It would be a great underestimation of ingenuity of today’s business enterprises if we assume or believe that they are less than capable to do so. I submit that the reason companies haven’t gotten around to working on these issues as much as they could’ve is simply because these problems are not high up on the radar. As I have attempted to explain, managers are concerned with a plethora of issues, software quality being only one of them. If vendors were to find their business drying up, due to poor software quality, some of which is as a result of aspects of outsourcing as listed above, believe me vendors will fix the problem pronto. They have the skills, they have the ingenuity, but the economic drivers are not strong enough today. This is not a problem with outsourcing per se but with relative importance of economic drivers.

Another difficulty is in what is perceived to be quality. There is sometimes a difference in how quality is perceived across the development community and business managers. For a developer poor quality is when the software does not work as requested or as stated or as designed for. For a manager, poor quality is when the customer turns away and declines to offer repeat business due to unhappiness with the software.

Finally this issue about lower skill sets. I completely agree at the risk of making a broad generalisation that the average entry level skills in the software development market are far lower in India than in the United States. But this is a manifestation of the fact that India got into the software act a lot later and when the development demand boomed, it fortunately or unfortunately had the ability to deploy a large number of software developers even though at lesser entry skill levels. (The statement above is likely to hold for other highly developed and developing economies as well). If hypothetically India and other developing countries had not got into this act, I have no hesitation in believing that given the booming demand, average skill levels in the industry would’ve gone down sharply in the developed countries as well. This is again an economic event much more a function of the population levels, wage disparities and output of the educational systems. To blame this on outsourcing is to shoot the messenger.

So whats next

Let me summarise some of the points made so far :
  • Businesses will continue to attempt to be more financially prudent :What this means is that while this prudence benefits us in general terms, it will continue to attempt to increase productivity and drive down costs.
  • Businesses will continue to be focused on quarterly and annual results :Love it or hate it, this system is here to stay for some time. If we can adjust software development to be in tune with it, probably all of us can be better off. Methodologies such as short iterations and frequent releases, YAGNI, continuous refactoring are likely to be better supportive of this environment.
  • Exploding demand for software :This is likely to continue unabated at least for a few more years or a decade. A large part of the world doesn’t yet invest heavily into software development yet but if food and energy demands are any indication, as more countries start consuming more software, this demand growth is likely to accelerate.
  • Crappy software is better than no software :This is probably the most difficult and perhaps a most profound thought for many developers. As software demand grows, and as our worldwide ability to keep pace with it shrinks, crappy software is likely to deliver positive economic results compared to lack of software. Combine this with focus on short term results and you will realise that customers are more likely to tolerate buggy software than ever before in times to come, and if customers tolerate it, vendors will find ways to test these tolerance levels.

So will things never improve for software quality ? I think two factors will help, one in the really long term and one in the medium term.

  • Software demand will start slowing and productivity improvements and better training will catch up eventually : While it does look very far away, it will happen one day. We are finding ways to create better tools, improved methodologies, offer better training etc. Software demand will peak one day and ability to service it will catch up. Thats when customers will start being intolerant of poor quality software. Thats when vendors will most certainly deliver good quality software.
  • SAAS model will help take the load off the demand supply gap :In terms of ability to create an economic impact per developer, the Software as a service - SAAS model works far superior to custom application development. (For the purposes of this discussion a SAAS offering is expected to be simultaneously used by a large number of customers). Development costs are only a small part of the total revenues in case of SAAS. However customer biteback in case of defects is much stronger due to a larger number of customers using the same software instance. Thus SAAS will take some pressure off development cost reduction, focus on a higher quality threshold from day one, and take some pressure off the demand supply gap. This to me is the biggest positive side of things in the days to come from a software quality perspective. However even this will take some time to play itself out. When that does happen, SAAS is likely to be the biggest demand killer in terms of the demand for software developer head counts on a worldwide basis.

If you are a developer or user who is frustrated with the quality of software, there are the following things you can do :

  • Recognise that some of the current situations (problems ?) are here to stay and be prepared to deal with them : If you can accept it (and it might be unacceptable for many) it will allow you to deal with the issues in a more rational and non emotional way.
  • You can choose to focus on buying / supplying high quality software : There certainly will be some customers and vendors who will go down this path. Try to identify customers / vendors / partners who share this belief system. Even if the entire industry quality levels don’t improve substantially, your individual experiences are likely to be better. However there is an economic impact of this choice and make sure you can absorb that by figuring out a way to offer a compelling value proposition to your customers. Individual developers who are particularly upset about quality levels should be willing to work for companies with a higher commitment levels to quality along with the concomitant adjustments to individual growth path and economic outlook if any.
  • Move to SAAS wherever feasible :I am not suggesting this out of any particular fascination for SAAS. There is built in argument in SAAS which is based on sound economics which encourages high quality software development. And while arguments based on emotional or moral merits are tough to win, arguments based on economics win by themselves. Developers wanting to work on high quality software development are better likely to find their needs satisfied with SAAS vendors.

PS : Lest it be misconstrued, let me clarify I am often anguished and occasionally offended by the state of software quality. I do not intend to justify poor quality at all. However I have learnt that that before solving a problem, it is important to frame it correctly. This post is an attempt to frame the current situation appropriately in economic terms and not in either emotional or moral terms since that has to be the first step in dealing with it.

Viewing 13 Comments

    • ^
    • v
    Thanks for giving the valuable information.
    Will continue reading this blog.
    • ^
    • v
    Thanks Niyaz.

    Hope this might have projected outsourcing from a different angle.

    Dhananjay
    • ^
    • v
    "Outsourcing" is not really single phenomenon that can be given a blanket "sucks" or "rules" rating. Rather it is an umbrella term that actually encompasses a whole bunch of different ways in which a business in a "rich" location can leverage the cost differential between them and a "poor" location. There are many different models of engagement, and each will give you a different result. See for example the experience of New York based social|median and Pune-based True Sparrow Systems - they are extremely happy with their situation.

    Saying "outsourcing" sucks is akin to saying, "software companies suck". The statement makes no sense - are you talking about Microsoft, or Apple, or Infosys, or HSBC software, or Xerox PARC, or AirTight networks, or the small budding startups in Pune?

    Some models of outsourcing suck.
    Some models are great.
    And there is no reason to simplify it by attaching a single label.
    • ^
    • v
    @punetech

    Not sure if your comment is directed at this post or the one it was referring to. I have actually not attached a label to outsourcing .. instead have explored the economic dynamics around it and the fact that it is perhaps not such a well understood process and it is our understanding of it that sometimes sucks (ie. it is often so casually maligned).
    • ^
    • v
    You are right. My comment was directed at the post that you referred to.

    I did want to point out the fact that there are different models of engagement. This issue is a little orthogonal to your analysis. I meant it as something else to think about, in addition to what you said.

    How the outsourcer and outsourcee interact makes a huge difference to the final outcome.

    Do you send over detailed specs and just want them implemented, or do you send over a high level concept and outsource the architectural thinking too? Or do you also involve the outsourcee in roadmap evolution details?

    What is the incentive structure in place? Bonus for delivery on deadline? By number of bugs fixed? At the other end of the spectrum is that the outsourcee has equity in the outsourcer!

    Which side of Venkat's Impossibility Triangle are you playing in?

    What are you really looking for? Junior programmers who can learn on the job? Or experienced and/or the best of the best? Like.com was looking for the latter and decided that in this case the cost advantage that India had was not enough to overcome the overheads.

    If you were setting up a local (non-outsourced) team, you wouldn't evaluate it purely on economic terms. You would be worrying about all the above factors, and more. And the success of your team would depend upon which point you chose on each of those spectrums. Same is true of outsourcing too.

    The different ways of outsourcing are different points (or technically, clusters of points) in a multi-dimensional space. Treating them all alike is over-simplification - good for headlines ("Outsourcing sucks"), but bad for understanding them and making best use of them. I guess, I am just elaborating another way in which "Outsourcing does not suck. Our understanding of it does."
    • ^
    • v
    Wow, great and very in depth post.

    There have been a deluge of posts and rants about outsourcing recently. While I agree that there are some genuine reasons for it having a bad reputation, there are some great freelancers and outsourcing companies out there who do a fantastic job.

    The arguments for different time zones, cultural differnences etc. are moot in my opinion - those are all known facts which can be easily dealt with. In my experience, problems with outsourced projects tend to be caused largely by the buyer - wanting to save money, and not taking the time to research and communicate with their freelancers properly.

    I've outsourced a number of projects over the past year and been very happy with most of the results. When there has been a problem I'm realised that it was mainly a breakdown of communication, and largely my fault, and I've learned from it.

    Ranting and complaining about outsourcing, as so many people seem to at the moment, won't help - it's here to stay! So all parties should endevour to learn, improve and turn outsourcing into the valid and respected business it should be.
    • ^
    • v
    Quality is a result of engineering processes.

    I think this is unequivocally false. It might hold true for making physical widgets, but in my experience software quality (and productivity) is a direct consequence of the talent and knowledge level of the programmer.

    The most serious problem I have found in working with an outsourced programming team is that the contracting firm has little visibility into the makeup and selection of the team. Usually, if you contract a ten person team from the average Mumbai outsourcing shop, their team lead will be a pretty decent programmer and there will be two or three guys who are satisfactory, but the rest will be a net drag on your productivity.

    A bad programmer produces less work, and that work is inferior: more bugs, poor structure, slow algorithms... Code review (which is absolutely essential when working with outsourced teams) takes a lot longer, and you need more rounds of it. Past a certain point the drag on the technical leadership will become prohibitive.

    If your process discipline is not 100% (and for most small software companies, it isn't), those guys on the left-hand side of the bell curve are going to slip some grotesque, monstrous logic errors into your programs.

    I think there's definitely a market in Canada and in the U.S. for "creme de la creme" outsourcing teams: people two to three sigma right of centre on the talent curve, highly productive programmers with MSc.-level computer science backgrounds, who are at least somewhat familiar with Unix (I've found that most Indian programmers aren't).

    The traditional draw of outsourcing to India has been that you can get a programmer for 30%-50% of what you'd have to pay one here. If we had it to do again, I think we'd gladly pay double or triple that salary to get a smaller team of really exceptional people.
    • ^
    • v
    Greg,

    Without attempting to characterize your experiences (and I am certain yours is a very valid experience as are likely to be many other different ones), one of the issues I have indeed pointed out is a reduction in average skill levels and the economic drivers which are leading to that.

    I think a customer who is not satisfied with the service, should change providers. But to imagine that you cannot find people two to three sigma right of the centre on talent curve in India is unlikely to be accurate. You'll certainly find them if you look hard enough (I would like to believe I have worked with scores of such people myself both in US and in India).

    With regards to engineering processes, there are firms which will have rigorous training and mentoring capabilities to ramp up the skill levels, and there are many who dont. You will find all kinds of firms in all corners of the world. Caveat Emptor would be my operating advice.

    Dhananjay
    • ^
    • v
    I will correct myself slightly here. In my limited and empirical experience, a large number of the highly talented programmers in India have since moved to United States, and a large number of those who didn't have moved into Management roles (both transitions I guess are highly driven by economic considerations). So it is a relatively harder to find right of 2/3 sigma programmers who are also highly experienced, in India. YMMV
    • ^
    • v
    In case to avoid the problems of outsourcing, the vetting process must be really thorough. Visibility is sure a problem, hence constant status checks are necessary IMHO. This would help in case of exiting a bad provider as early as possible. Still, productivity is lost if all we do is test providers. We should try the best in the vetting process to eliminate the bad apples.

    From my experience, there are sure good people in the programming team, but most of them are scattered throughout different providers and different teams. Most of the setup is just as what Greg mention, a capable team lead, followed by few other capable team members and the rest are just net drag. I am not saying that every team is like this, but even with big providers, they tend not to put all their eggs in one basket, and they will handle the risk by distributing their top talents rather than keeping them as one team.

    And to find top talents are sure is a problem in the market. If all you depends are top talents, then its going to be very hard to bill the client and meet your sales target. This is what I believe IMHO to be the reason for such exercise (of distributing talents and just keep adding people in the team even if it can be a drag).

    I would just say, that the system work for the providers that way. Anybody considering outsourcing, just need to shop a bit harder I suppose.
    • ^
    • v
    An interesting and detailed article. I particularly like your points about short term economic advantage, at least from Wall Streets point of view, and something being better than nothing. Both of these are very strong forces that can't be ignored, even though most people wish they could be!
    • ^
    • v
    • ^
    • v
    Thats a lot of information and analysis. But I always thought the quality of software was bad because somehow the client was never focussed on it. If I were a client paying for some software, then it better met my standards. If we think it is crappy, but the client is happy : is the software really crappy? Maybe not. It is perhaps just what is enough. That situation needn't change at all, right?
 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus