Posts Tagged: cloud


2
Nov 09

Five important trends on the enterprise architect’s radar

It is no secret that the internet architectures are influencing enterprise architectures. This post attempts to summarise some of the recent trends in the internet space, which seem to be carrying some momentum sufficient enough to influence the enterprise. So without further ado, these trends are :

  1. REST : The Representational State Transfer architecture style builds on the essential elements of those constructs which made the internet so globally scalable. A detailed explanation of the rationale and strengths of REST are completely beyond the scope of this article. If your job requires you to be continuously aware of emergent trends and whether they fit your enterprise architecture needs – this is the one must explore trend.

    Impact : Web based architectures, Service Oriented Architectures, wide availability and immediate usability of data and processing requests (resources) through simple HTTP URIs and minimal integration effort

  2. Interoperable Cloud : The interoperable cloud is the ability to create a private cloud and also leverage a public cloud. This has been made possible by offerings such as the Ubuntu Enterprise Cloud which allows you to build a private cloud or use a public cloud such as Amazon EC2 while being able to access them using the same set of APIs thanks to open source efforts such as Eucalyptus. This allows you the flexibility of initially using either a private or public cloud and then subsequently shifting to the other, or being able to use both simultaneously.

    Impact : Large servers vs cluster of commodity servers, virtualisation, elastic deployments, flexible hardware procurement / provisioning, infrastructure management in organisational hierarchy.

  3. NoSQL : While I am unhappy with the name, it has stuck. This refers to a set of options now available to store your data unconstrained by many RDBMS requirements (eg. flexible schema, key value pairs etc.). Some of the databases also allow you to store data in a distributed manner over a number of servers with an intent to support high availability in write intense scenarios even as they may require you to move towards eventual consistency. These options increase your manouverability / flexibility as an architect even as they require you to meet a different set of challenges.

    Impact : Relational databases, data storage strategies, data distribution strategies, vertical vs. horizontal scalability, transactionalisation, consistency and availability

  4. Polyglotism : Developer costs now occupy an increasing percentage of total costs, development time is being an increasingly dominant factor for time to market, and ability of software to change and adapt quickly to newer demands is now a critical success metric. One of the solutions is to write different parts of the software in a different languages most appropriately suited for concise and rapid coding as well as supporting quick reaction changes to each part appropriately. Thus it is conceivable to have some of the business rules written in a dsl written using jruby and some of the algorithms written in clojure in a software built on the JEE platform.

    Impact :Development culture and processes, minimum developer skill and scalability, risk management for managing required vs. available skills.

  5. Decentralised processing : Thanks to many developments which are leading to increasingly distributed processing including REST and NoSQL, applications will need to be a set of collaborating network based components (we’ve heard this before with distributed objects as well). However especially given some of the lesser guarantees that such architectures can provide around immediate guaranteed processing, latency issues, distributed control and asynchronous processing, a particular piece of business logic may get satisfied in a staggered fashion across a number of collaborating components. This may increase challenges in terms of currency of available data even as it helps actually deliver on the vision of distributed objects and simplifies individual component development. While asynchronous capabilities such as those supported by MQ series and the like have been used in the enterprise for ages, I do anticipate increasing use of lighter messaging constructs such as PubSubHubbub within the enterprise.

    Impact : Application partitioning, network based components, difficulty in supporting fully synchronous workflows.


27
Feb 09

Data Sans Frontiers : Data on the cloud : Facebook and its new statements

A little over a week ago after the Consumerist published its commentary on Facebook’s modified terms of service, I had deleted my data from my Facebook account and had written at length about what I perceived were the issues with the modified TOS : Why I deleted my Facebook data. Commentary on Internet data privacy rules. A couple of days after that Facebook under pressure from all the adverse feedback, reverted to its old TOS and assured to work with the community to work out a new TOS over the coming few weeks. Earlier today, it released a proposed Facebook Principles and a proposed Statement of Rights and Responsibilities for community review as a part of that process. This was covered by TechCrunch, Silicon Alley Insider, Mashable and Inside Facebook.

Unlike my earlier post, this does not critique Facebook’s actions but instead attempts to analyse the statements in the context of a newer dimension of data thats mobile on the cloud. This brings in both a set of challenges and opportunites and I shall try carefully to not fall into the FUD zone as I discuss these challenges. Finally it also discusses these statments and its implications on user’s and their awareness education challenges. In this exercise I treat Facebook as one of the early providers who is facing and attempting to address these challenges of “mobile data on the cloud” in terms of its privacy and security implications. But in the mashup world, it is likely to be a challenge that both the users and a number of other service providers have to eventually start dealing with.

Allow me to add my annotations on these statements.

Proposed Facebook Principles

A very nice and clear enunciation.

  • Point 1 : Note the specific mention of “service”. Thats what in my opinion is leading to many of these challenges.
  • Point 2 : Perfect in principle. However one technical issue needs to be resolved. How does one define “received information“. If my status update is seen by another user and it has appeared on his wall, has he received information ? Probably not since it is quickly clarified that this is particularly outside the Facebook Service. So we are probably talking about an external application to which facebook feeds the data, another user who might have downloaded some data on his desktop, potential third party services such as search engines (which usually aren’t allowed to scour this data) or other analytics services. I think its a fair statement. The only suggestion I would have is that there probably needs to be some amount of user (re)education to understand its implications.
  • Point 3 : Clearly refers to the fact that information will be shared with other users and services including through tools (desktop apps / mobile apps ?). Sets up the ground for API usage.
  • Point 4 : A great point. Establishes a level of symmetry amongst the parties which was one of my sore points in the earlier modified TOS
  • Point 6 : Clearly documents API usage to access data
  • Point 10 : Notes that the data is not just mobile across organisations (ie Facebook and its applications) but also across geographic boundaries.

OK, lets quickly review the next statement.

Proposed Statement of Rights and Responsibilities :

Point 2 is an important one to note, especially 2.2 and 2.3. Point 2.2 is wonderful to the extent that it clearly defines that facebook eventually will delete data that a user deletes or in case the user deletes the account. It also specifies that content shared with others may remain until they delete it. Now this content is in all likelihood going to be third party applications and services. Here’s an interesting tripartite arrangement – theres now going to be (for practical purposes) a mini-TOS between the user and Facebook which this statement covers, between Facebook and the third party provider, which I think is likely to be covered within this document as well (well, at least parts of it) and probably one between the user and the third party service. Its the last one which could be worrisome, since the user may now need to worry about each TOS with each third party provider. Most users are unlikely to review each of those terms with the level of detail, yet find their data moving to the third party provider. However, how and when that could get deleted is also addressed later in this statement. Point 2.3 also clearly documents the fact that subject to the user privacy and application settings Facebook assumes a whole bunch of capabilities around such intellectual property. Unlike in the modified TOS, this gives the control to the user to decide how the intellectual property will be treated. Since the control is with the user, I think it is a fair clause to ensure that Facebook itself is not at the receiving end of any intellectual property rights dispute.

In point 3 Facebook attempts to secure the fact that its users do not harvest information or otherwise maliciously gather access to store information that they shouldn’t get access to store. Why the focus on store ? The clauses around using the API are likely to have strict conditions around what data can be stored by the third party application and for what duration (which is true today as well). So its the social or screen scraping or other non-API ways of collecting data that Facebook is trying to secure against.

Point 5 sets up the environment for users to respect property rights of each other. Point 5.7 especially ensures that services clearly document their privacy policies when collecting data and inform the user that the data is being shared with the service and not with Facebook. Point 5.8 ensures that users don’t post other user’s sensitive information and then let it stream over the cloud potentially into other applications and services. Quite reasonable in my opinion.

Point 9 is probably one of the most interesting sections. The important point here is that Facebook has clearly communicated with the user, the undertakings of the applications, websites and services. Coupled with 5.7 and 5.8 above, it essentially creates an environment where the user is clearly aware of the boundaries between Facebook and other services, and that once the data crosses that boundary, that data for all practical purposes is governed by the agreement between the user and the other service. 9.16 was one which did arouse my curiosity a bit. We now start seeing the outlines of why the TOS needed to be modified in the first place.

Finally point 12 also goes to a great distance in assuaging the concerns raised about unilateral and quiet change of TOS earlier. It allows for upfront communication with the users, a reasonable notice period and curiously a vote based user veto (not sure why this really was required, but then I would be surprised if 30% of the user base actually voted).

So what about data sans frontiers ?

In the earlier modified TOS, Facebook had attempted to secure a very large number of rights in order that it would stand protected (if and) when large parts of user’s data start getting shared with other applications, websites and services. Thanks to the user reactions, it has now crafted a very different statement. In this statement while Facebook now assumes far fewer rights, and those it assumes to me seem like part of a fair and symmetric relationship. The way I read and interpret this is that under this statement, Facebook clearly defines a set of boundaries or frontiers (ie. the Facebook Service), and undertakes and provides a mechanism to properly share and control the sharing of such data within such frontiers. However it also makes it abundantly clear, that the user is now adequately aware that it has far lesser control the moment the data crosses such frontiers, and at such a stage this data will now get controlled by the third party services, that the undertakings by the third party service are clearly specified and the user is fully in the loop on this issue. Again seems fair from a corporate perspective and in terms of securing itself from misbehaviour or other inappropriate activities or accidents on the part of the third party services.

The catch is now that the onus is on the user. He has to clearly understand the implications of (perhaps even more detailed) privacy controls, and agree to the fact that once the data crosses the facebook frontiers, facebook has taken adequate precautions in terms of ensuring that the appropriate undertakings having been provided by the users and the services, and that any issues that crop up further could perhaps be bilateral issues between the users and the third party service.

I don’t know that all users are actually in a shape to understand all these implications, and suggest that some mechanisms be put in place to provide users some tutorials or presentations which explain these in as easy and lucid manner as feasible. Users will need to exercise the Caveat Emptor clause before subscribing to / using other services which integrate with Facebook. Users will need to be aware that based on their privacy settings and application requirements, their data is now moving from host to host, company to company, and country to country. On the whole while this does place a far bigger burden of exercising caution on the user, it also puts the user in control and is a far more symmetric statement than the earlier modified TOS.

Finally in an increasingly mashup and service integration driven world, I fully anticipate many more companies and users having to deal with these very issues in other non Facebook contexts as well. This statement is likely to also play a role in the ongoing discussion about the capabilities and the risks of having data in the cloud. If these statements can be agreed upon in a format that is acceptable to all, there’s a good likelihood it will provide the precedent for many more service arrangements being drafted in the future in a relatively similar and consistent fashion.

A programmer is unlikely to be a good lawyer : I am a programmer and have very lay understanding of legal matters. This opinion is based on my understanding but such an understanding could be wrong or inaccurate when looked at by a more skilled person. It is most certainly not a legal opinion and could contain discrepancies, inaccuracies or improper assessments on my part. If you note or find any, please add your comments so that others can benefit from your views as well.


27
Jan 09

Software / IT Terms in early stages of abuse or ripe for Abuse

Abuse : improper or excessive use or treatment (Merriam Websters)

Ahh! That is a term we haven’t found being used in IT context too often. Have we ?

Abuse in the past

Well let us start with some terms that have been used, and in my opinion abused in the past :

Web 2.0 : If there was indeed a term that actually set itself up for abuse this was it. There was nothing in it for people to refer to and hold on to as a basic and essential premise around which it gets applied. Eventually Tim O’Reilly the author of the term had to come out with a clarification What is Web 2.0, in which he said it refers to the Design Patterns and Business Models for the Next Generation of Software. Amongst the many trends he pointed out were

  • The Web as a Platform
  • Harnessing Collective Intelligence
  • Data is the next Intel Inside
  • End of the software release cycle
  • Lightweight programming models
  • Rich User Experience

He also defined the core competencies of the Web 2.0 companies

  • Services, not packaged software, with cost-effective scalability
  • Control over unique, hard-to-recreate data sources that get richer as more people use them
  • Trusting users as co-developers
  • Harnessing collective intelligence
  • Leveraging the long tail through customer self-service
  • Software above the level of a single device
  • Lightweight user interfaces, development models, AND business models

Did Web 2.0 get abused. You bet. In many ways it was too clearly defined. But inherent in the number of attributes that were associated with Web 2.0 was the risk that anything sharing but a fraction of the attributes would want to call itself Web 2.0 (given the popularity of the term). Moreover it was actually hard not to accidentally abuse it for the same reason. However people have got around that eventually and now treat Web 2.0 as simply a moniker enhancer, and then evaluate the application or company to figure out whether it indeed is Web 2.0. The overuse of the term is leading to it becoming less relevant today

Service Oriented Architecture (SOA) : While I’m pretty sure this term has been around since early 90′s and especially used in the CORBA and Tuxedo worlds, the earliest online web page I could find which described it is “Service Oriented” Architectures, Part 1” which describes it as A service-oriented architecture is a style of multitier computing that helps organizations share logic and data among multiple applications and usage modes.. A very ironic article was “BEA Fits Tuxedo With SOA Makeover” especially considering that CORBA and Tuxedo users were probably amongst the earliest proponents of the term. Instead of seeing SOAP-WS* as one more element in a continuum of possibilities, it became the “in thing” which got weighted down by so many promises, expectations, goals and so much scope creep to its perceived capabilities. Herein lay the seeds of the disappointment one sees today especially in situations when it is doing rather well.

Lifecycle of abuse :
The following are the stages through which a term that is subject to abuse goes through :

  • Initial formulation of seed thought and early implementations or practices.
  • Further refinement of thought into a notion or hypothesis that has been tested and refined through a few experiences.
  • Success visibility by neighbours leading to greater chatter along with active promotion of the term and its benefits.
  • A positive spiral of both field successes and good PR leading to a strong mindshare
  • High mindshare attracts a large and diverse community of “second stage thought movers”
  • Desire to brand or market oneself in terms of using that term to sound “in” or “leading edge”
  • Attempts to reevaluate the terms meanings in one’s own context to achieve the best fitment leading to a term meaning modification (either newer attributes being introduced or existing attributes associated with the term getting diluted)

A term’s essential value proposition is that it manages to convey one sometimes composite and sometimes complex notion succinctly. Yet successful people, successful companies, and successful terms act as great magnets. Once a term is successful, everyone attempts to use it in their own context. This is when the term starts becoming different things to different people. Thats when it goes down the Blind men and an elephant path. Thats when the term loses its core value proposition, and people using it get steadily more confused over a period of time. One way to avoid this disintegration is to be aware of the process and attempt to slow it down by drawing attention to the core notion and value proposition of the term.

Similar to the terms above, I believe other terms are now getting ripe for being abused. However sometimes abuse leads to improper expectations and sometimes even to “<Term> is Dead” which is actually an indication of the terms abuse dying rather than its basic set of offerings which continue to do well. A couple of times I have blogged about things *not* dying are SOA ain’t dead but it certainly is transforming and Java : the perpetually undead language.

I do see many terms in Software development and IT that are in the stage where they are being abused or are starting to be abused. So before we have to bury them eventually for not meeting the expectations such abuse creates, can we rein in the abuse. I doubt it, but this post is an effort in that direction.

REST : This one theoretically should be the least likely to be abused. Simply because it is so well defined in Roy Fielding’s Dissertation. But even he had to eventually write in his post REST APIs must be hypertext-driven.

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

Strong words indeed. And if you think if these were against some small application, the social site REST API is the OpenSocial REST API which is a rather prominent standard (and its kind of hard to imagine google and other participants wouldn’t realise the distinction). Yet the underlying shift is that for internet based applications people are increasingly relying on simple HTTP based API and since perhaps HTTP is too bland a name to term an API with, they go with the sexier sounding REST. Not really necessary. If I have an HTTP API, its all right to call it an HTTP API. Another term that did start getting used alternatively was POX/HTTP (Plain Old XML over HTTP) but the slight issue with this term is that JSON is starting to be another data payload vehicle so POX doesn’t necessarily cut it universally.

There is another issue as well with using REST as the moniker to describe HTTP based APIs. Roy Fielding eloquently points out that REST is so different from RPC. Yet in a logical sense, many requirements for HTTP APIs are very RPC like. It is possible to represent such an API using REST semantics (instead of user/delete/123 call it userdeletion/create/123 .. a name mangling trick). Moreover REST requires statelessness, a requirement that not all applications necessarily comply with. I am not suggesting they should, just that should they choose to not be completely stateless it would be difficult to term their APIs REST. These were both issues I had referred to in my earlier post Fomenting unREST : Is RESTfulness a semantics game ? Why does REST require statelessness ?.

Clearly REST is not going to cut it as a universal mechanism of describing simple HTTP based APIs. I would suggest call them HTTP APIs, or perhaps think of a better term which is not REST since it seems rather likely that many of the HTTP APIs we shall develop may simply not meet the requirements or expectations of REST. The more we use REST the way it should be, the less we are likely to abuse it. So may I request that we use REST only when the API adheres to all the expectations as specified by Roy Fielding ?

Cloud :

I am certain the term cloud began thanks to the abstraction of internet that got used in tons of network topology diagrams. So it perhaps referred to the segments of the network that formed the entire plumbing of the internet and all the external network segments which were not relevant to the topology under discussion, and to the applications and services that resided on these “external” network segments (primarily the internet). However further along the way additional attributes started getting implied. An example is the rapidly scalable infrastructure (the scale on demand attribute). This blog is hosted on a pre specified hardware by a web hosting company. Is it on the cloud ? Its a little unclear (for certainly it does not have the scale on demand attribute). Occasionally implied attributes are multi-tenancy. IMO thats stretching it too far, multi tenancy is an optional attribute of applications and services that can be hosted on the cloud, it isn’t the attribute of the cloud itself. It would be better to keep the scale on demand attribute away from the cloud term and use it as an additional qualifier (eg. elastic cloud). However one of the effects of that is that it is leading to a proliferation of cloud related terms which is if you pardon my pun, making the situation a little too cloudy. Some of the terms are for lack of a more civilised response quite amusing eg. “Internal Cloud”. I mean if cloud represents the external segments of the network and the application and services that are hosted on them, why would one call the internal network the internal cloud as if the external network would be the external cloud. But then whats the difference between the network and the cloud. Or does the network becomes the cloud or should it be that the cloud becomes the network. This highly avoidable tendency of clouding everything also has a cloudy term for itself cloudwashing – slapping the word “cloud” on products and services you already have. See the point ? Having said that, I still think it is preferred to have a proliferation of qualified terms rather than a proliferation of contextually implied attributes into a single term itself.

So can we keep cloud restricted to the “external network segments and the applications and services hosted on them” ? Thank You. And lets keep out an eye on all the additional terms that further qualify cloud. Many of them might not make sense, so just decloud them.

*****-as-a-service :

It all probably began with Software as a Service (Saas). Saas rocks and is a term which is catching on. But it is also now getting a ton of other “as a service” followers. Just google for this and the following words crop up as the first part of -as-a-service : platform, desktop, hardware, desktop, sql, data, ui, infrastructure, PC, research, os, malware, identity management, analytics, middleware … get the picture ? Now in this current global economy where more than a third of the economy is based on services, just imagine the sheer proliferation of terms that are likely to happen simply because someone doesn’t say I offer hosted middleware, no it is middleware as a service. This blog will recast itself as Gratis chronological opinion as a service. People are going nuts, and there’s no one to stop them. In the meanwhile whats happening to the one which started it all – SAAS ? Theres something called Level 1 SAAS where each customer has its own customized version of the hosted application and runs its own instance of the application on the host’s servers. I have read at least one blog post which refers to ISVs needing to customise Saas offerings for individual enterprise customers. So rather than newer implied attributes being added to the term Saas, existing ones are being removed in a manner where any outsourced software hosting will be Saas. Pretty soon, a business unit which outsources software development, maintenance and hosting to another intra corporate business unit will start calling it a Saas offering to keep up with the pressure to seem current. Pretty soon all software will tend to Saas.

Can we please keep Saas restricted to uncustomized offerings made by other service providers which essentially rent out the use of the software they’ve built or own on an as-is basis (with version upgrades from time to time of course) ?

Mashups :

Lets just for a moment look at how wikipedia describes Mashups.

a mashup is a web application that combines data from more than one source into a single integrated tool.

I see a number of enterprise vendors starting to offer Mashup products. That in itself is welcome. I however also anticipate a full wave of rebranding of Integration Services, SOA, Portal Servers into Mashups in addition to the few new mashup features being introduced. Also expect such buzzwords to drive newer budgets and customer excitement even though relatively little has changed under the covers. Eventually the term could become all encompassing where it covers all forms of system integration and service aggregation which will eventually make it useless as a term. In the meanwhile authors of existing Web 2.0 Mashups built on Google Maps, Flickr and Twitter will be left wondering, “Hmm.. was this was mashups was all about ?”


Agile : I don’t know that its terribly abused, but when I see presentations that combine elements of Agile with layered sequences (eg documented design followed by review followed by development followed by testing) and heavy management control procedures and call it the right way of doing agile – it smacks of abuse to me. Thats like taking a cheetah, cladding him with medieval knight armour, and saying ain’t it a very nice and strong fastest animal on earth ? I don’t think it is feasible to adapt a heavy process to agile or vice-versa to somehow take on the attribute of being agile. Agile requires a migration from processes which encourages repeatability through documentation, division of work and control into one which achieves high productivity and quality through capability enhancement. This requires a reduction in the control processes and an increase in the capabilities training in a manner where the quality and control is not maintained through processes built around a replaceable workforce, but instead through a set of (relatively) informal and frequent communication channels built around a increasingly capable team.

A lighter parting note.

Finally on a lighter note a friend suggested another term that is being abused. “Software engineering – a Term that should never have been applied to software and insults engineering.” I agree even though I know the comment probably stemmed from the usage of this word in this blog’s tagline. While Software Craftsmanship has been discussed for long, I find many software building efforts not worthy of being called craftsmanship. Therefore I contribute a new term to the lexicon – Software Construction And Maintenance. This is one term (or its acronym) I am not concerned of getting abused. In fact it is unlikely to get used.