There have been many high profile outages lately which have caught peoples attention.  These failures are being used as an argument for why critical systems should remain internal and not be deployed as SaaS or in the Cloud.  Some of these outages included Google App Engine’s performance issues in early July , Rackspace’s loss of their Dallas data center due to power failure and the fire in Seattle that took Authorize.Net offline for 12 hours to name but a few.

What amazes me is how so many people point to this and argue that this is proof for why Cloud and/or SaaS is bad and that everything should be in house.  It’s preposterous.  The fact that these systems went down with a data center failure (or otherwise) is nothing more than an argument for inadequate system design, where High Availability (HA) is concerned.  The bottom line is it takes planning, forethought and good design to make a system highly available, and most systems simply are not designed with that in mind.

The reasons for not making a system highly available are many and include the following:

1. Naivete: People don’t believe it could happen to their system and thus choose not to put in the time, effort and cost of making a system highly available
2. Cost: Bottom line is it costs a lot of money to make a system HA and for a lot of firms, particularly when starting out or for smaller businesses, it just not a viable option
3. Difficulty: Its bloody hard to make a system HA.  Its one thing to ensure no data loss,  its quite another to ensure little to no down time.

For most of my career I have built systems for the World’s largest financial companies including the World’s leading Investment Banks and Stock Exchanges.  These firms take high availability very seriously as a rule, but even with their resources and decades of experience systems still go down.

Consider the London Stock Exchange (whose system I did not design), who last year had a very public outage when they were down for most of a trading day.  This was not a SaaS system or one deployed in a Cloud.  It was an internal system run by a highly reputable company whose business is based on being reliable and never losing a trade.  These exchanges, for the most part, have highly redundant systems, multiple backup data centers, design for High Availability and run fail over tests regularly, yet they still experience downtime from time to time.

The point is, failures happen, whether the system is run internally, or in the cloud.  Whether its a SaaS system or one of home grown legacy design.  The objective is to minimize those failures and the downtime associated with them.

That said,  with today’s technologies, some careful planning and good design, it is possible to build systems that should almost never go down, even in the face of a 9/11 type event, but thats a topic for another day.

So I was reading an article this evening over at CloudAve about the latest Google Reader and how it still can’t be used offline with full features. In particular the article focuses on its inability to allow you to read articles offline  and then flag those articles as already read, such that when you get back online Google Reader doesn’t present them to you again, which is a waste of time.

Well this got me thinking about the general challenges of making SaaS based applications usable in an offline mode.

Consider the following application scenario:

1. You have designed a simple SaaS application to do basic contact management
2. Users need to be able to use it offline as well as online
3. Users may log into the application from many computers
4. The systems allows more than one person to edit any one particular contact record

Now, I specifically structured this scenario to allow each of the points above to allow each one to layer on additional complexity which needs to be considered when designing the system.  We will tackle in each turn.

1. Basic SaaS Contact Management

Well this isn’t too hard if you want a basic system (and we don’t plan for any of items 2-4, or other complexities such as data design for multi-tenancy, fine grained permissioning, etc ).  All you need is a database which contains contact records, which can be read or edited either through a public interface (which doesn’t necessarily need to be Web Service based) or by using a GUI of some kind (through the API or tightly bound to the database) which you provide.

2. Using the Contact Management System Offline

Now its gets more difficult.  You face the following issues:

1. Method two of using a GUI tightly bound to the database is no longer an option, unless you want to be maintaining that data access method as well a separate API for the offline users to use, with all the inherent potential for inconsistent behavior two data access methods would imply
2. The offline user needs to have full access to a copy of his/her relevant data on the machine that they are using offline
3. How do you accomplish this if the potential data in question is large (say many Gigabytes just to make it fun)
4. Said offline data needs to be stored in a secure manner, which is searchable, editable, and more importantly synchronizable
5. Metadata capturing the status of each record also needs to be stored along with those records
6. The metadata needs to be compared against the database when the user gets back online in order to determine how to update or merge the data

3. Allowing for Multiple (Potentially Offline) Computers To Be Used By Each User

Well this is not too terribly difficult once you can solve the issues in #2 above.  As long as the multiple computers are all online tis is infact really trivial with a SaaS application and is infact where SaaS shines. How often have you cursed not being able to easily synch your iPhone, MS Outlook on your Home machine and MS Outlook at Work in real time.  Well with a SaaS based app, this is no problem because they all read and write the same master copy of the data which resides on the central server.  That is, as long as they are online.  So what if they aren’t all online?  Well the following could arise and needs to be considered when designing your SaaS system.

1. Assume the user has 2 computers, one at home and one at work (we will ignore the iPhone for now)
2. Assume he edits some data at home (in offline mode) and forgets to sync it online before heading out to work
3. Assume that when at work he edits more records (lets assume in offline mode again just for fun), and that some of the records are the same as what he edited at home.
4. At the end of the day he sync’s to the central server from the office machine
5. He gets back home and wants to sync the home machine and carry on from where he left off

The problem is thatthe records at home have been edited so they need to be sync’ed, but some of those may be stale and older than what was done at work. So, how do we do it?  Well we ideally need to do the following:

1. We need to check all records on the home machine against the central SaaS system
2. For any record which was edited on the home machine but not of the server copy, just upload it
3. Any records which were edited at home and at work need to be merged (which is always a pain and error prone so you need some good rules and just stick to them)
4. Any record edited on the server version and not at home should be overwritten with the central copy.

After all of that,  the system will be in as clean a state as it can be and the person can now continue to edit the records.  Its important to note that if the records had been sync’ed in the morning that this procedure would not be needed, which makes the case for auto saving/syncing the data often, to minimize the potential for data collissions in cases like this.

4. What If Multiple People Can Access and Edit The Same Data

Well the issue in this case is one of concurrency, and it needs to be allowed for whether the users are online or offline.  Consider Person A and Person B are both editing the same contact records while offline.  Both then go to sync their data (and they have both edited some of the same records maybe not with the same changes to those records).  So what does your system need to do?

1. This gets a lot worse if they submit the data simultaneously, but we will not worry about that at this time
2. Each record for each person needs to be compared against the central data copy
3. If the server copy has not been modified, then just upload as normal
4. If the server is modified, perform a merge in accordance with the systems rules
5. Sync the local copy of the data to the new refreshed server version if needed

That’s really about it.  it turns out that multiple users is really no different (from each user perspective), than if the had edited on two machines as a single user.

So whats the point of all this.  Well the bottom line is that if you are designing a SaaS based application you better think about all of this.  It not sufficient to assume that

• Everything is stateless
• No data concurrency issues will arise
• There is only one user on an account or that can edit any particular data

As software designers we need to make sure that what we design works when these use cases are considered, which is very difficult, especially if you are a startup and trying to get product out the door in a hurry.  While I am a fan of Agile programming and have used variations of it building application for Wall Street Investment Banks, even before the term Agile existed, the idea of minimal viable product would get you into trouble here.  These types of features are hard to layer on later and you will be forced to go back and redo what you already completed resulting in increased development cost and time to market, to say nothing of dissatisfied customers if you already shipped a version that needed to be drastically changed.

Thus my personal design mantra:

Implement only what needs implementing, but don’t design yourself into a corner.

In my experience it pays in the long run to make sure of your design upfront, even if it costs a little bit of time.  But don’t get crazy and produce a 6000 page design doc because that will be wrong too.

Well this is my new blog.  I hope you’ll find it useful and informative.

As you can see from my bio page, I have been designing and building enterprise scale, high performance systems for the worlds largest financial institutions for about 25 years now, and I thought I could use this blog to share my experience and enter into discussion with a broader technology audience.

The blog will feature a wide array of content including a lot of “How To” type articles, opinions, discussion on technologies and whats in the press.   In addition, I will also touch on topics relevant to technology start ups, including Venture Capital (VC), marketing, setting up, and anything else that comes to mind.  While I have spent most of my career working for the World’s largest financial firms,  I do not intend to write articles about Investment Banking or Finance (unless something really interesting comes up and I need to rant).

Mostly, this blog will be to share my experiences and pass along lessons learned over 25+ years of designing large scale mission critical systems.  Topics which I expect to touch on include, but are not limited to:

• Service Oriented Architecture (SOA)
• Software as a Service (SaaS)
• Enterprise Architecture
• Designing and coding for Scalability and Performance
• Distributed, Cluster and Cloud Computing
• Designing for High Availability

In addition to the “How To” material, I expect the bulk of the rest of the blog will focus on things which I see happening in the technology space which I think are either interesting and/or make me shake my head in wonder.

I hope people will find this useful and will check back often.  I also want to encourage active debate and interaction and would gladly entertain suggestions for topics people would like to read about.  If it’s a topic I feel I have experience with and can contribute something useful, I will do my best to write an article about it.

Thanks for taking the time to read my blog.

Paul K Michaud