One Monday morning about a month ago, I was browsing through open issues in our bug tracker looking for something to work on. It was my week on “technical support”. At Signal, the engineer on technical support spends a week working on non customer facing issues with our infrastructure, process, etc, that are too large to simply knock out as you encounter them.

An issue entitled “Add monitoring to detect jobs that do not start” caught my attention. We already had monitoring in place to alert us when a job fails. However, we had nothing to alert us if that job never starts in the first place. We have many scheduled jobs that run in the background, some pretty important, and have been bitten on more than one occasion by reoccurring jobs that had not run for quite some time.

I bounced some ideas off the team, and we all agreed that that it made sense to create a new application to monitor our scheduled tasks. The app would know what is supposed to run, and when. We would change all of our jobs to “ping” the app when they start and finish. If the app detects that something did not run when it was supposed to, or did not finish when it was supposed to, it could alert the team.

And just like that, Proby was born.

“What’s with the name?”, you ask? I likened the application to a probation officer, keeping an eye on wayward tasks.

Since going live, Proby has caught several issues with our scheduled jobs that would have gone unnoticed…many more than I would have anticipated. It has also evolved into a fairly complete application. It is easy to setup new tasks. It holds onto the execution history of each task, allowing you to see a trend of the run time for each task, and which runs resulted in an alarm. It supports alarms via email or SMS, using Signal’s SMS messaging API. It has several settings that allow you to tweak exactly when an alarm is sent, cutting down on the number of false alarms. And, thanks to designer extraordinaire Drew Myler, it looks amazing!

Shortly after creating Proby, we realized that there is no way we are the only ones who have had scheduled tasks go days without running, unnoticed. So today, we are making Proby available to the general public. It is currently in closed beta, and we are letting people in slowly to flush out any bugs or other issues. Use of Proby is also FREE while it is in beta.

Interested? Head on over to http://probyapp.com/signup and signup to participate in the closed beta! And, don’t forget to follow @probyapp on Twitter for updates!

What the hell is a code retreat?

For those who have never attended a code retreat, the format is simple. The group is given a problem. In our case the problem was Conway’s Game of Life. The code retreat is then broken up into several 45 minute sessions. Each session you pair with somebody new, and you take a stab at solving some portion of the problem. All code is written using TDD. At the end of each session you delete all of the code you created. The problem, and the time limit on the sessions, are specifically chosen so that there is little to no chance that you will come up with a complete solution in the allowed time. Solving the problem is not the point. The point is to practice.

We as software developers don’t give ourselves enough time to practice. At work, or even with open source or side projects, we’re always trying to get something done. Rarely, if at all, do we practice solely for the sake of practice. This is what we focused on at the code retreat. At the end of each 45 minute session, we deleted our code, and discussed what we learned. We were encouraged to try new ideas, to really push ourselves outside of our comfort zone. Knowing that the code would be deleted in a matter of minutes, there was no need to worry about screwing up the design, or creating something that would ultimately become unmaintainable. We had a safety net, and were encouraged to use it.

Corey said that we never get the chance to write “perfect code”. Perfect code doesn’t exist in the real world. In the real world products need to ship. It is simply not practical to spend the amount of time on a piece of code that is required to make it “perfect”. There are always trade offs of one form or another. However, code retreats give you the opportunity to practice writing perfect code. There is nothing to ship, nothing to “get done”. The code lives for 45 minutes, and then it disappears, giving you a unique opportunity to practice writing tiny pieces of perfect code.

Structured chaos

Sounds pretty hectic, right? But, the sessions were far from unstructured. Throughout the retreat, we were encouraged to keep in mind the four rules of simple design:

• Runs all the tests
• No duplication
• Proper naming (expresses ideas and reveals intent)
• Small (small methods, small classes)

The morning sessions were primarily focused on getting familiar with the problem. As the day progressed, we were encouraged by Tyler and Corey to try out certain ideas, or to think about tackling old problems in new ways. Can you solve the issue using polymorphism instead of if statements? Can you do it by capping all of your methods to 3 lines? What happens when you focus on proper naming of your tests and your methods? These “challenges” were designed to push us out of our comfort zone. It worked.

Some thoughts

I quickly realized that 45 minutes was not enough time to solve the problem. However, it took me almost the entire day to finally get comfortable with the fact that even solving a significant portion of the problem was out of reach. About half way through the day, I caught myself trying to solve a chunk of the problem that was way too large for the amount of time we were given. This resulted in me rushing a solution in an attempt to get my way too large of scope test to pass. At the end of the session, I was left confused with a pile of crappy code (unnecessary abstractions, no encapsulation, etc), at least half of which I wasn’t sure why we wrote.

I love that feeling you get when you create something, or solve a problem. This is why I am a programmer. Not being able to finish the problem, or even a significant portion of it, was incredibly frustrating. It took me the entire day to finally let go of that.

It was also great to see the different ways people were approaching the problem. At the start of the code retreat most pairs were pursuing a similar approach. But by the end of the day, the variety of approaches was simply amazing. With each session, everybody learned something new. Since we were required to find a new partner for each session, the combined experience of each new team seemed to produce a new approach, slightly different from the approaches each of the partners had tried in previous sessions. It was almost like looking at an old tree. Each session resulted in the forking of each branch on the tree. By then end of the day, there were a whole lot of branches.

Practice, practice, practice…

At the end of the code retreat, we all briefly shared with the group what we learned that day. Everybody had something new to say. The code retreat had a way of showing all of us what we needed to work on and what we needed to explore.

When writing this blog post, it suddenly hit me.

This is exactly what practice is designed to do!

When I practice something else (karate basics or kata for example), it has the same effect. Serious practice makes it very clear what needs to be developed. It makes no difference what it is that you’re practicing. The key to getting better at something is to practice, and to continue developing the “problem areas” identified by your practice. But it can’t be just any practice. As Corey stated during the code retreat:

Practice doesn’t make perfect. Perfect practice makes perfect.

Perhaps the single biggest thing I took away from the code retreat is that it showed me how to practice writing code.

What I learned

First off, I learned that I need to slow down…way down. All too often I think I know the solution to a problem before my fingers even touch the keyboard. And, perhaps worse, is that I am so focused on implementing that solution, that I will often ignore warning signs (code smells) that should be alerting me that my solution may perhaps not be the best solution. This code retreat made me realize that I need to do a much better job of listening to my tests, and listening to my code. I need to do a better job of keeping my nose open at all times for code smells, and to stop when I smell something. Something that doesn’t feel right in the tests is a sure sign that something is wrong with the design. Stop, take a step back, think for a minute about what is causing the smell, and fix it.

There is nothing wrong with having a solution in mind before you start coding. In fact, I think that most times you want to have a solution in mind before your fingers touch the keyboard. However, you should keep an open mind as you start to build your solution. Don’t assume your initial solution is the best solution. In fact, it’s probably safer to assume that it is not, and allow your tests to guide you to a better one.

Go to a code retreat!

If you’re serious about getting better as a programmer, and have never been to a code retreat, your missing out on an incredible learning experience. Make it a point to go to one. Since code retreats are essentially group coding practice, you can attend them over and over and be sure to walk away with something new every time. I’m already looking forward to the next one.

I have so much to learn.

To address this issue, we decided to create daily summaries of the subscription list data, and report off of the summary data instead of the raw data. The vast majority of our list reporting is already broken out by day, so this seemed logical. And, we decided to use MongoDB to store the summary data.

Why MongoDB?

We are already running MySQL and CouchDB in production. Why not use one of those? Why introduce a 3rd database product to the architecture?

The summary data is live data, not archived data (like the data we are storing in CouchDB). As subscriptions are created/destroyed for the current day, we need to increment/decrement the appropriate metrics. There are also some use cases where we need to alter summary documents for days in the past. This means we need support for atomic operations in the underlying database, to prevent race conditions from skewing the stats. CouchDB does not have the ability to atomically update a value in a document with a single call, so that took it out of the running. Some of our more active lists see several new subscriptions a second. Dealing with document update conflicts would have been a nightmare.

We decided to go with MongoDB over MySQL because of the structure of the summary data. We had several metrics that we wanted to keep tabs on (opt ins per day, opt outs per day, etc). Some of these metrics had a nested nature to them. For example, in addition to keeping track of the number of opt-outs per day, we also want to keep track of the reasons those subscriptions were canceled. Also, there are several ways that a user can opt-in to a subscription list. For example, each list can have several different keywords for entry via SMS. We needed a way of breaking all of these metrics out by method of entry.

Also, the main data structure containing the stats is repeated twice within the document. Once for the current day’s stats, and once for the totals for that subscription list up until the date in the document. This “to_date” data structure keeps us from having to evaluate ALL of a list’s documents in order to determine how many subscribers there are on a given date. With the “to_date” data structure, the specific day’s document is all we need.

We decided that this data structure was better represented as a single JSON document instead of a series of tables in MySQL. It seemed much cleaner to have a single document per list, per day that contained all of the information for that list’s daily activity than to have it scattered about in a series of relational tables. Choosing MongoDB meant giving up on SQL’s aggregate functions (AVG(), MAX(), MIN(), SUM(), etc), but the benefits provided by the simplicity of the data structure seemed to make up for the loss of these functions.

An example summary document

{
  "campaign_id": 1,
  "subscription_campaign_id": 2,
  "account_id": 1,
  "date": ISODate("2010-10-13T00:00:00Z"),
  "stats": {
    "sms_101": {
      "confirmed_opt_ins": 100,
      "unconfirmed_opt_ins": 15,
      "unconfirmed_opt_outs": 10,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 5 },
      "confirmed_opt_outs": 20,
      "confirmed_opt_out_reasons": { "UI": 10, "CNF": 10}
    }
    "sms_102": {
      "confirmed_opt_ins": 200,
      "unconfirmed_opt_ins": 30,
      "unconfirmed_opt_outs": 20,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 15 },
      "confirmed_opt_outs": 35,
      "confirmed_opt_out_reasons": { "UI": 15, "CNF": 20 },
    }
    "email": {
      "confirmed_opt_ins": 300,
      "unconfirmed_opt_ins": 35,
      "unconfirmed_opt_outs": 25,
      "unconfirmed_opt_out_reasons": { "UI": 20, "BULK": 5 },
      "confirmed_opt_outs": 30,
      "confirmed_opt_out_reasons": { "UI": 20, "BULK": 10 }
    }
  }
  "to_date": {
    "sms_101": {
      "confirmed_opt_ins": 1100,
      "unconfirmed_opt_ins": 1000,
      "unconfirmed_opt_outs": 200,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 100 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 300, "CNF": 100 }
    }
    "sms_102": {
      "confirmed_opt_ins": 2200,
      "unconfirmed_opt_ins": 200,
      "unconfirmed_opt_outs": 150,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 50 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 250, "CNF": 150 }
    }
    "email": {
      "confirmed_opt_ins": 2050,
      "unconfirmed_opt_ins": 125,
      "unconfirmed_opt_outs": 75,
      "unconfirmed_opt_out_reasons": { "UI": 75 },
      "confirmed_opt_outs": 750,
      "confirmed_opt_out_reasons": { "UI": 600, "BULK": 150 }
    }
  }
}

A possible relational database schema

MongoDB’s Atomic Operations

As I mentioned above, MongoDB’s atomic operations were key to us choosing MongoDB for this task.

MongoDB does not support locking or transactions like you would find in a traditional relational database. However, it does support a number of operations that are guaranteed to be atomic on a single document. If your data is designed so that all related data (or at least data that would need to be updated at the same time) is contained within a single document, then the atomic operations supported by MongoDB should be more than enough to handle the majority of use cases.

For this project, we made heavy use of the $inc operation. $inc will atomically increment or decrement a value anywhere in the document by the specified value. If no property exists in the document with that name, $inc will create it, and set its initial value to the value you wanted to increment it by. For this project, we simply initialized the data structure holding the metrics to an empty JSON hash when the summary document is created. The first time $inc is used to increment or decrement some metric, it will insert the metric into the hash, along with the proper initial value. Subsequent calls to update the document using the $inc operator will then update that value accordingly.

Using the $inc operation also meant we didn’t have to read the document to get the current value of the field in order to increment/decrement its value. We were simply able to increment/decrement the value by making one call to the database, keeping things nice and simple.

Atomically incrementing a document’s confirmed subscriptions count

collection = db.collection('subscription_statistics')
today = Date.today.to_time
collection.update({'campaign_id' => 1, 'date' => today},
  {"$inc" => {"stats.sms_101.confirmed_opt_ins" => 1}},
  {:safe => true})

MapReduce

MongoDB also supports MapReduce, providing you the ability to evaluate your data in ways that simply can’t be done using their standard queries. For this project, we needed to support summing the values of specific keys across several documents (to calculate the total opt-ins over a date range, for example). Initially, this sounded like a good fit for MapReduce. However, MapReduce will run the specified map function against each document in the database. The more documents your database has, the longer it will take MapReduce to run. I hoped that since we had built an index for the fields that MapReduce was using to determine if a document should be selected, that MongoDB would utilize that index to help find the eligible documents. Unfortunately that was not the case.

In our case, since we are only dealing with 365 documents for a year’s worth of statistics, it was considerably faster for us to find the documents using MongoDB’s standard queries and sum the data in ruby code, than to use MapReduce to do the same. If we were evaluating ALL of the documents in the database, then MapReduce would have been a much better option. I understand that 10gen is hard at work on making MapReduce faster for MongoDB 2.0. Having a strong MapReduce framework can be a powerful tool for a statistics database. Hopefully we’ll be able to utilize it in the future.

The Results

The results were staggering. On our largest list, the overview chart we display showing the current number of subscriptions per day over the last 30 days went from taking 37726ms to load to just 502ms. And the summary report for the list, which contains a wealth of statistics for the list including opt-ins / opt-outs per day, subscriber count by opt in keyword, and a series of summary statistics, went from taking 64836ms to load to just 322ms.

Dramatically reducing the size of the data being evaluated had an equally dramatic effect on the amount of time it took to evaluate that data. And, MongoDBs atomic operations and dynamic queries made this project a blast to work on.

def some_method(required_1, required_2, options={})
  # Do something awesome!
end

some_method("foo", "bar")
some_method("foo", "bar", :option_1 => false, :option_2 => true)
some_method("foo",
            "bar",
            :option_1 => false,
            :option_2 => true,
            :option_3 => "something",
            :option_4 => "something else")

This may not look like much. However, this feature alone is capable of producing some very readable code, and is used extensively in APIs throughout the Ruby ecosystem. Consider for a moment what these APIs would look like if Ruby did not have this capability, which isn’t hard to imagine for those of us with a background in a language like Java. You would either be forced to require that each parameter be specified:

# What is this code doing?  What do the nil values,
# or even the true and false values map to?
some_method("foo", "bar", false, nil, true, nil)

or accept a hash or a request object that contains all of the necessary parameters:

# This is much more readable, but requires that the
# options hash be created on its own line.
options = {:option_1 => true, :option_2 => false}
some_method("foo", "bar", options)

Providing optional parameters via hash key/value paris at the end of a method call produces code that is incredibly readable. You have the names of the attributes right next to their corresponding values! There is no ambiguity whatsoever as to which values match up with which parameters.

It is also very flexible. The order of the attributes in the hash does not matter, like it does for required attributes. And, it is very easy to add new options, or delete old ones.

This approach also makes it easy to specify default values for options that were not specified when calling the method:

def some_method(required_1, required_2, options={})
  defaults = {
    :option_1 => "option 1 default",
    :option_2 => "option 2 default",
    :option_3 => "option 3 default",
    :option_4 => "option 4 default"
  }
  options = defaults.merge(options)

  # Do something awesome!
end

There are however a few minor drawbacks to this approach. The first is documentation. Methods that take a hash of options as a parameter do not convey any information about the valid options in the method definition alone. And, it is possible that the method in question simply forwards the options to another method, sending you on a wild goose chase to determine the set of valid options the code supports.

# Looking for a list of valid option keys...no help here.
def some_method(required_1, required_2, options={})
  do_something_awesome_with_the_options(options)
end

This is why it is so important do document your public API if you are using this approach. Take a look at the ActiveRecord::Associations::ClassMethods documentation. This page documents, in a very clear and easy to read mannor, all of the supported options for each method.

It is also worth pointing out that while this approach is great for optional parameters, it is ill suited for required parameters. Required parameters should be specified outside of the options hash, making it clear that values for the required parameters must be provided. While it’s true that stuffing all of your parameters inside a hash means you’ll never have to look at another wrong number of arguments error again, it will make your code difficult to understand, and easy to misuse.

• Mean reads / second
• Mean writes / second
• Mean requests / second for DELETE, GET, HEAD, POST, and PUT requests
• Mean view requests / second
• Mean bulk HTTP requests / second
• Counts for various HTTP response codes

In addition, there is a plugin for individual CouchDB databases and individual couchdb-lucene indexes. The database plugin will report:

• Database size
• Number of documents
• Number of deleted documents
• Number of update operations

The couchdb-lucene plugin will report:

• Size of the index
• Number of documents indexed
• Number of deleted documents

The kind folks over at Scout have just released two new, official plugins based on the ones I created. The CouchDB Overall plugin combines some of the more important CouchDB metrics into a single plugin, and the CouchDB Database plugin reports the same set of the stats as the database plugin listed above.

The original plugins can be found at https://github.com/signal/scout-plugins/tree/master/couchdb. More information can be found here. I hope you find them useful.

Thanks to Doug Barth for some help on the plugins, and Derek over at Scout for putting together the official plugins.

However, polyglot persistence comes with its own set of drawbacks. One of those drawbacks is the loss of foreign keys, which are very important in maintaining data integrity. Another drawback is that Object/Relational Mapping (ORM) libraries typically focus on a specific database, or type of database. So, writing code that manages relationships between objects backed by different databases hasn’t been nearly as easy as writing code to manage relationships between objects in the same database.

Tenacity’s goal is to address some of these issues. Tenacity is a ruby gem that provides an ORM independent way of managing relationships between models backed by different databases.

Tenacity works by extending popular Ruby ORM libraries to respond to a set of methods that the tenacity core uses to build and manage relationships between objects. By extending the ORM libraries to implement this interface, tenacity is able work with the objects in a generic way, without having to know what database is backing the given objects. This approach also allows you to continue using your favorite ORM libraries. To use tenacity, you simply need to include Tenacity inside your model.

Tenacity is heavily based on ActiveRecord’s associations, and aims to behave in much the same way, supporting many of the same options.

This initial release of tenacity supports belongs_to, has_one, and has_many associations, and the ActiveRecord, CouchRest, and MongoMapper ORMs. However, there is still plenty of work to be done. Feedback, bug reports, and code contributions are always welcome.

Tenacity is free and open source, and can be found on GitHub at https://github.com/jwood/tenacity.

Example

Download

gem install tenacity

The Rise of the App

However, the web does have limitations. One such limitation has long been the ability to interact with the hardware you are using to browse the web. Over the years technologies have been developed to help with this limitation (Java, Flash, etc), but I see those technologies as extensions the web, and not part of the web itself. Those technologies have carried their own set of drawbacks as well, such as requiring the installation of client software, performance issues, and security vulnerabilities.

Nowhere is this limitation more obvious then in the mobile arena. Today’s smartphones are amazing. You have a camera, a GPS, a MP3 player, a web browser, oh…and a phone too, all wrapped up in one device. It’s like walking around with a tiny computer in your pocket. But currently, the web is only able to utilize a tiny fraction of this functionality. Its document centric design, which has been stretched, twisted, and pulled in directions that its creators never could have imagined, appears to have finally reached its limit.

In addition to hardware access limitations, may web applications were not designed to be viewed on a device with such a small screen and no mouse. It can be very difficult to view a 900 pixel wide web page on a screen that is only a couple of inches wide. And while you can zoom in over certain areas of the page, not being able to view and interact with the page in its entirety introduces some big usability issues. Also, many web applications rely on mouse over events to interact with the user. Drop down menus are a perfect example of this. There is no mouse on today’s touch screen smartphones, making these sites very difficult to navigate when viewed on a mobile device.

These limitations have given rise to “the app”. The app, a native application that runs on the smartphone itself, it not a new idea. Smartphones have been around for quite some time, and these phones have always had native applications. But only recently did Apple change the game by providing independent application developers with a way to easily sell and distribute their applications. As a result, there are now hundreds of thousands of applications out there for various platforms, many of which do incredible things with the hardware on the mobile device. Things that simply could not be done via a web application.

The rise of the app came on so quick and so strong that many began to ask if the app killed the web. This is certainly a reasonable question to ask. A native application has access to all that the device has to offer; the camera, the GPS, and yes, even the web. Apps are also designed to run on a mobile device. So, the text is easy to read, the controls are easy to use, and the user experience is generally much better than a web application.

Problems with Native Applications

While native applications have full access the the hardware and address many usability issues that web applications have on mobile devices, they introduce a new set of problems.

Some platforms charge program fees that must be paid to develop applications for the particular platform, even if you plan on creating a free app, or an app you never plan on releasing. While generally inexpensive, it is a new cost that web application developers have never had to deal with.

Deployment is more complicated for native applications than it is for web applications. Even though Apple’s App Store, Google’s Android Market, and other application marketplaces have made the distribution and installation of native applications much easier, it still does not compare to the ease of deployment of a web application. On the web, you have complete control over what you deploy, when you deploy it, and to whom you deploy it to. This is not the case with native applications, which can be subjected to drawn out, illogical review processes before being made available to the public. If you have a critical bug that needs fixing, or you’re trying to beat your closest competitor to market with some killer new feature, this process can be agonizing. And, even after publishing the latest version of your application to the marketplace, there is no way to force your users to upgrade. This could leave you supporting older, buggy versions of your application, or maintaining old services still in use by those old versions of the application.

If you charge for your application, distribution of that application is not free, or even cheap. Apple’s App Store, Google’s Android Market, and Palm’s App Store all take a 30% cut of the sale price of your application. Some may argue that this is a small price to pay for distribution to such a large audience. However, the audience is larger on the web, and distribution is piratically free.

The development of native applications is largely platform dependent. Programming languages and APIs vary significantly from platform to platform, making it practically impossible to develop a single application capable of running on multiple platforms. Cross platform development tools, such as PhoneGap and Rhomobile, address this issue by allowing you to develop native applications using HTML, Javascript, and CSS. However, these tools don’t address the other issues mentioned in this section. In addition, you also run the risk of some App stores outright banning applications not built using their tools (looking at you Apple). Though Apple has recently softened their stance on this issue, I would not rule out future restrictions.

A Mobile Web App in Native App’s Clothing

The sad part is, the majority of today’s apps don’t need to be native applications. They do nothing to utilize the features exclusive to native applications. Most of these apps simply fetch data from a web service and display it nicely to the user. Some take advantage of the GPS, or store data locally on the device. However, these apps could just as easily be mobile optimized web applications.

“Look and feel” is often cited as a reason to develop a native application instead of a mobile optimized web application. Though this argument certainly has its merits, there are plenty of options out there to help make a web application look and feel great on a mobile device. A bit of mobile optimized CSS (fluid layouts, properly sizing images, etc) can go a long way in terms of making a web application look awesome on a mobile device. And, there are projects out there, such as jQTouch, jQuery Mobile, and Sencha Touch that can help your web application look and feel more like a native application if that’s what you are looking for.

Web applications also have access to a few features that, just a short time ago, were only available to native applications. The device’s location is accessible via Javascript’s Geolocation API, eliminating the need to build a native application solely for access to the device’s GPS. Web applications also have access to persistent local storage on the mobile device via the HTML5 Web Storage API. While the Web Storage API falls short of a being a true database (it’s more like a hash, allowing you to store key/value pairs), it can go a long way in addressing the storage needs of most applications.

A good comparison between mobile apps and native apps can be found here.

Making the Web a Friendlier Place for Mobile Web Apps

The mobile web is growing by leaps and bounds, but much work is needed in order to make the web a friendlier place for mobile web applications. The W3C realizes this, and have started a Device APIs and Policy Working Group to create a specification that will allow web applications to access device hardware and services (contacts, calendar, camera, microphone, SMS, MMS, email, etc) in a standardized way. The specification is far from complete, and it will likely be quite some time before these APIs have been implemented by the various web browsers. But, the mere fact this is being worked on is a great sign for the future of mobile web applications.

There are non-technical hurdles preventing the web from being a more mobile friendly place as well.

First, browsing the web on a mobile device isn’t a very pleasant experience right now. Having “real” web browsers on mobile devices, such as WebKit, have gone a long way in fixing this. However, many sites are not optimized for mobile devices, making them difficult to use. I think this prevents the majority of people from browsing the web on their mobile device like they would on their desktop or laptop computer.

Second, we’ve been told for years that “There’s an app for that”. Personally, when I’m looking for a mobile application take care of some task, the first place I hit is the Android Market. Having one place, where apps are broken up into categories, and searchable, makes finding the right app for my needs incredibly easy. There is no such place for web applications. This used to be the case for desktop applications as well, but not anymore. The Ubuntu App Center, and now the Apple Mac Store, aim to serve the same purpose as the App Store or the Android Market, but for desktop computers instead of mobile devices. Perhaps it won’t be long before we see a categorized, searchable index for mobile web applications as well.

Conclusion

Opting to build a mobile optimized web application instead of a native application is not a new idea. However, the idea has been slow to catch on, due to some of the reasons I’ve outlined above.

Obviously, even after the Device APIs being worked on by the W3C have been implemented, there will always be a need for native applications. Games that need hardware accelerated graphics, or apps with complex user interaction will most likely stay native applications for the foreseeable future. But I hope that the other applications, especially the applications that are simply a front to a web service, are re-born as web applications at some point in the near future. For these apps, it’s time to come home.

Kudos to Heroku for continuing to find ways to make their platform easier to use.

The script is below. Hope you find it useful.

See everybody next month!

Information about the meeting can be found here. I hope to see you there!

The case for Polyglot Persistence was made throughout the day by several of the speakers. Most people seemed to acknowledge the fact that it is very likely one database may not have all of the tools you need to get your job done. Instead of coercing a single database to try and do things it was never designed to do, it is becoming more common for applications to use multiple databases, utilizing each for their respective strengths. But, of course, using multiple databases in a single application comes with its own set of issues, and you should make sure there is a real need for Polyglot Persistence before making that leap.

Thank you very much to the WindyCityDB organizers for putting on such a great event. I had a great time, learned a ton, met some interesting people, and participated in some great conversations. What more can you ask for in a tech conference?

So I started taking notes, documenting everything I could regarding what we learned about CouchDB, the design decisions we made for our application (and their respective trade-offs), and the migration of our application code to use CouchDB. I organized those notes, and posted them on this blog as a case study, hoping it would help others looking into CouchDB.

I received a lot of positive feedback from the posts, making me feel like I had in fact filled that need, at least to some degree. Even better, earlier this year I was contacted by Couchio about combining the series of posts into a white paper that would be posted on their site. This would put the case study in front of a larger audience, potentially helping even more people. I was thrilled.

Today, that white paper was released as the “Epic Interactive Mediums Whitepaper” (love the Epic :)). You can get it here. Many thanks to the kind folks over at Couchio for putting this together. I hope people will find it helpful.

I saw the re-write as a good opportunity to get more familiar with Rails 3 and MongoDB. After getting Rails 3 setup to use MongoDB via MongoMapper (Ben Scofield’s setup script gets you most of the way there), I quickly realized that the tests would not run.

jwood-mbp:addressbook jwood$ rake test
(in /Users/jwood/dev/personal/addressbook)
Errors running test:units, test:functionals, test:integration!

jwood-mbp:addressbook jwood$ rake test:units
(in /Users/jwood/dev/personal/addressbook)
rake aborted!
Don't know how to build task 'db:test:prepare'

Some Googling revealed this post by Yehuda Katz, in which he says:

Additionally, users can completely remove the ActiveRecord gem, and be rid of the generators, rake tasks and other peripheral elements. An example: if DataMapper implements a db:test:prepare rake task, a Rails developer can replace ActiveRecord with DataMapper, and the test:units rake task will use DataMapper’s new tasks.

We’re not using DataMapper here, but the same thing applies to MongoMapper. So, all that is needed is to implement a db:test:prepare task, which I did in lib/tasks/mongo.rake

namespace :db do
  namespace :test do
    task :prepare do
      # Stub out for MongoDB
    end
  end
end

As of right now, the task does nothing. I may end up adding some functionality to it as I work on the project. We’ll see.

Running the tests again, I can see that was all that was needed to get going.

jwood-mbp:addressbook jwood$ rake test:units
(in /Users/jwood/dev/personal/addressbook)
Loaded suite /Users/jwood/.rvm/gems/ruby-1.9.1-p378@rails3/gems/
rake-0.8.7/lib/rake/rake_test_loader
Started
...
Finished in 0.062854 seconds.

3 tests, 3 assertions, 0 failures, 0 errors, 0 skips

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

First, Standup Timer was my first mobile app. Aside from a very small amount of time I’ve spent playing around with the Blackberry development kit (really too small to even mention), I have had no prior experience developing software for a mobile device. My background is largely web development and distributed applications development. Second, I have been doing Java development for quite some time, so I am very comfortable working in Java. Third, if you’re not familiar with Standup Timer, it is a very simple application. It only interacts with a few components of the Android platform, and only consists of a few screens. No GPS, no web, no multi-touch, etc. So, my tour of the Android SDK was far from complete.

Dealing with multiple devices

Supporting multiple screen sizes

The topic of supporting multiple screen sizes has been a popular one, especially when comparing Android development to iPhone development. Supporting multiple screen sizes in your application does add a little complexity to the development process, but for my app (and I’d imagine most apps), it is very manageable.

This issue did not take Google by surprise. They knew that if Android was to be successful, it would need support devices with a wide range of physical characteristics, including screen size. So, from the beginning, they made it possible for applications to support multiple screen sizes with little effort. The Android platform contains many features to help with this, including the ability to pre-scale images for phones with different resolutions, allowing you to specify your screen component sizes in density independent pixels (dips), and allowing you to easily center and stretch your components to fill the screen. These features, along with a set of best practices for supporting multiple screen sizes, are documented very well at http://developer.android.com/guide/practices/screens_support.html.

For applications that do not use the standard Android view components, or do custom graphics, it may be a different story. I have read blog posts from some Android game developers saying it is a larger issue, and some who say it is not. So, I’m not sure. However, for the vast majority of applications, the standard view components work just fine, making supporting multiple screen sizes a very manageable issue.

One thing that did bite me regarding multiple screen sizes is the fact that views will not scroll by default if they happen to flow off the screen. You need to anticipate this, and wrap any views that may flow off the screen with a ScrollView to enable scrolling behavior.

Supporting multiple devices

To me, this issue is much larger than the multiple screen size issue. Android is an open source project, which gives anybody the ability to modify the code however they wish. Google prevents carriers and cell phone manufacturers from abusing this by holding back a few key applications from the open source release, including the Android Market application. Without the Android Market application, phones would not have access to the 20,000 apps currently available for the Android platform.

However, phone manufacturers do tweak the platform to work for their specific hardware. Android 2.1 on one device is not necessarily the same as Android 2.1 on another device. For an example of this, compare 2.1 on the Motorola DROID with 2.1 on the Nexus One. These seemingly minor modifications can be very troublesome for application developers. The web has been flooded with reports of developers growing frustrated with the complexity matrix of Android versions and the exploding number of phones running them, all potentially containing their own tweaks to the platform. Browsing through the change logs of applications I have installed on my Motorola DROID, I can see that several apps have made changes to fix issues on specific devices.

This issue is forcing developers to not only test on every device they plan on supporting, but also to write device specific code to work around any known issues for a device. As Android continues to grow, this level of support will be unsustainable. Developing for a common platform should mean that your application will run fine on any device running that platform. But, because of these device specific tweaks, this is quickly turning out not to be the case for Android. I worry that if Google doesn’t find some way to control this, developers will continue to abandon the platform.

I have even run into what appear to be device related issues with Standup Timer, which is miniscule compared to the size and complexity some of the other apps available in the Android Market. Standup Timer uses an Android API that prevents the user’s screen from blacking out while a timer is in progress, letting you always see how much time is remaining for a meeting. Just the other day somebody left a comment in the Market indicating that this was not working on their device. So far, all of the other reviews have been positive, leading me to believe that this could be an issue with that user’s particular device. The commentator didn’t provide any information about the device they were using, or how they could be contacted. So, I’m not sure if I’ll ever be able to track down this issue. Even if I knew the device experiencing the issue, I’m not sure I would be able to help. Unless you have access the device in question, reproducing these issues is practically impossible. The Android simulator tool is great, but you cannot create a simulator for a specific Android device, running that device’s flavor of Android. You can only create simulators running the “generic” version of an Android release.

Development

Application life cycle

The Android platform primarily communicates with an application through a series of life cycle events. The platform will let the application know when an activity (a screen) has been created, paused (lost focus), resumed (regained focus), etc. The life cycle events are easy to understand, and are specific enough so that I never need to examine the state of my application or the platform within one of these callbacks. The fact that a specific life cycle method was called tells you what state your application is in.

My largest complaint about how the platform manages an application is how it destroys and re-creates the activity (the object that backs the screen) when the phone rotates from portrait mode to landscape mode, or visa versa. It seems to me that this would best be handled by making the appropriate life cycle callbacks to the same activity instance, giving it a chance to redraw the screen. Instead, you are forced to save all of your activity’s state, and then reload it when the new activity instance takes over. Forgetting to save the state of a particular variable means that value will be lost when the phone is rotated. This was the source of several bugs when writing Standup Timer.

Android APIs

I have no major issues with the Android APIs. They seem complete (for what I needed), and most importantly, behaved as expected. The database API seems a bit archaic, especially in today’s age of sophisticated O/R mapping tools. But, at least I didn’t have to catch SQLException after each operation, like the JDBC API.

However, the lack of decent documentation for some of the APIs is a problem. There were a few instances where I was forced into using trial and error to determine the purpose of a method parameter. I feel this level of unclarity is unacceptable for a public API.

UI Creation

The Android SDK offers developers two methods for creating UI screens: programatically using Java code or declaratively using XML. I did not attempt the programatic approach, as it brought back too many nightmares of building UIs for Java applets. The XML approach to creating screen layouts, which is the recommended approach, is very straight forward. Java developers, especially those working in the “enterprise”, have grown to hate XML over the past few years, due to its verbosity and proliferation. But, I think Android’s use of it is very tasteful. At no point did I consider myself to be in XML-hell. Looking back on the final XML that declares my user interface, it doesn’t seem overly verbose or complicated.

I did however struggle at times to figure out how to get the UI to look exactly the way I wanted it. Again, this seems to be the result of poor documentation of the XML elements and their corresponding attributes. Some more complex examples in the sample code included in the SDK would have also helped.

If you keep your screens simple, and don’t display too much information on any given screen, then you shouldn’t have much trouble dealing with how that screen looks in portrait or landscape mode. Screens with a list of items, for example, will usually look just fine when viewed in either portrait or landscape mode. When in landscape mode, the user will just see a little more blank space in the list. For more complicated screens, the Android platform allows you to specify an alternate UI layout for landscape mode. This allows you to completely restructure the elements on a screen to deal with the the wider, shorter screen size without affecting how the screen looks in portrait mode. Given how I struggled to get the views to look exactly the way I wanted, I tried my best to avoid the need to specify an alternate layout file for landscape views. Only two of the screens in Standup Timer actually needed a landscape specific layout file.

adb

The Android Debug Bridge (adb) is a nifty little tool that allows you to interact with an Android device or simulator from the command line. You can install/uninstall applications, interact with files on the device, or run a shell on the device. It also provides a series of commands that allow you to more easily create scripts to run on the device. Not only is this tool useful by itself, but it also enables the creation of sophisticated tools for Android development. Learning how to use adb is a wise investment of your time.

The Eclipse plugin

The Eclipse plugin for Android development is great. It provides tight integration between Eclipse and the Android SDK. Code completion is available for the Android APIs as well as the different types of XML files used by the Android platform. It has great support for Android resource files. You can easily start your application on a simulator with the click of a button, and running your tests is a breeze. I would not attempt to write an Android application without this plugin, or a similar plugin for a different IDE.

However, I did run into a few issues regarding how Eclipse interacts with the Android simulators. On more than a few occasions, when starting the application or running the tests, the Eclipse plugin started a simulator running a version of the Android platform that did not match my project settings (although, I have not completely ruled out user error here :) ). Also, the Eclipse plugin seems to have a few bugs around interacting with multiple simulators. adb allows you to specify the target device or simulator when issuing a command, which is useful when you have more than one simulator running at a time (which is almost always the case given the current state of the Android platform). However, the Eclipse plugin doesn’t always utilize this capability. Sometimes it will detect multiple simulators running, and ask you which one you’d like to target. Other times it will not. In addition to the simulator interaction issues, there are a few areas that could benefit from a bit of polish. The UI builders, for example, are difficult to work with. It became obvious very quickly that I would be better hand coding the XML for the UI than attempting to use the builders.

Testing

Simulators are awesome

The Android simulators are awesome. Once you have the proper versions of the SDK installed, it is trivial to create simulators with different screen sizes, different amounts of available storage, running different versions of the Android platform. And, as previously mentioned, adb is great for interacting with the simulators. While not perfect (you should always test your app on a real device before publishing it), they are pretty darn close.

The only thing missing, and I’m not sure how feasible this is, is the ability to create a simulator for a specific device. For example, if I know that my app is crashing on the Nexus One for some reason, I’d love to be able to create a Nexus One simulator, so I can find and fix the issue. Currently, debugging and fixing device specific issues is not possible without access to the device in question. I know some large mobile shops purchase the devices they plan on supporting, for testing purposes. And, services like Device Anywhere allow you virtual access to a physical device, which would satisfy this need. But these options, especially the first one, are expensive. A developer trying to fix a device specific issue in their free, non ad subsidized application will not have the resources for either of these options.

Automated (unit and functional) Testing

Automated testing on the Android platform is PAINFUL. It’s hard to decide where to even begin. First off, the tests provided with the sample applications in the SDK are very bare bones. Most of them simply assert that an activity has been created, and that’s it. If there was sufficient documentation describing how to write tests for you application, this wouldn’t be that big of a deal. But, there isn’t. Luckily, guys like Diego Torres Milano are writing blog posts and giving presentations in an attempt to fill this gaping hole in documentation. The slides from Diego’s presentation at Droidcon 2009 are a great place to start. Standup Timer, which is open source, also has a complete set of unit and functional tests for you to check out if you so desire.

Another pain point regarding automated tests for Android applications is that the tests cannot be run inside a standard Java Virtual Machine. The implementation of the APIs in the android.jar file provided with the SDK simply throw exceptions. That jar file is only meant for building your application, not running it. So, all tests must be run on the device (or in a simulator). This dramatically slows down the execution of the tests. One alternative proposed by a non-Android group inside of Google is to mock out all of the Android components using a mocking library, enabling the execution of the tests in a standard virtual machine. Although this would speed up the test execution, it comes with its own drawbacks. Mainly, you end up having to mock out a TON of stuff. I think that mocking out too many components significantly reduces the value of your tests. If you mock out everything, what are you really testing?

The SDK contains a few base classes that you can extend for certain types of tests. In particular, the SDK provides support for functional tests (the testing of a particular activity, or screen) and unit tests (the testing of some underlying support code). Writing unit tests is pretty straight forward. With a little bit of investigation and some trial and error, I was able to create a set of tests for my database layer, which will execute against a test version of the application’s database.

Functional tests were much more problematic. In addition to executing much slower than unit tests, they are also more difficult to write. For functional tests, you write code to perform the operations a user would be performing on their phone (entering text, pressing buttons, etc). The problem is that it is not very straight forward to perform these operations via code. It took me quite some time to figure out how to simply enter some text in a text area and press a button via code. Even when I thought I had functional testing figured out, there was still a use case I was unable to write a test for. Most of the issues revolve around gaining access to the components you wish to interact with. For a simple text box or button, it is very easy. You can simply fetch the view by its id, the same way you do in your application. However, once you start nesting components, it becomes much more complicated. I tried several different ways to invoke the context menu (the long press menu) of a list item, inside of a list, which is inside of a tab. None of them worked.

Instead of writing functional tests for activities, I found it much easier to test activities using the ActivityUnitTestCase class with a mocked out instance of the activity. Test classes that extend ActivityUnitTestCase don’t have as much as access to the platform as functional tests. Several API methods will throw exceptions if called from within a ActivityUnitTestCase, and other API methods simply do nothing. However, this can be addressed with some simple tools that should be in every developers toolbox: mocking and stubbing. It is trivial to contain a method call that would throw an exception during testing within a method of its own. This method can then be overridden by a mock implementation of the activity. The mock can then optionally set a flag when the method is called, so the test can verify that some action was performed. One place I do this is where the activity under test tries to start another activity. Calling startActivity from within a ActivityUnitTestCase does nothing. So, I placed the call to startActivity by itself in a protected method, overrode that method in the mock activity class, set a flag in the mock method implementation, and checked to see if that flag was set in the test case to verify that the activity was started.

Deployment

Publishing to the Android Market

Publishing an application to the Android market could not be any easier. The process is very well documented on the Android developers website, and the Eclipse plugin makes creating and signing an installation package a point and click operation.

A fee of $25 is required to create a Android Developers account. This account gives you the ability to upload your application to the Android Market using the Android Developers website. Once published, the application can immediately be found in the Android Market. There is no approval process. Updating your application is just as easy. Simply upload the new version of your application, and it is available in the Android Market immediately.

It should be noted that there is no fee to install the application on your phone.

Summary

Android is a great platform. It is feature rich, and developer friendly. The devices that run it are powerful and easy to use. I also love the fact that it is built on open source technology, and that Google, for the most part, has continued this tradition of openness.

But, the platform is not without problems. Most of my complaints here are minor, and are very addressable. Android is still a young platform, so there is plenty of room to improve. What worries me most about the future of the Android platform is the complexity matrix I talk about in the Supporting Multiple Devices section.

Android was created to be a common platform for mobile application development. A platform where application developers could write a single application capable of running on numerous devices. However, this dream is at risk of becoming just that, a dream. Unless Google can find a way to minimize the impact of device specific issues, it risks losing more and more Android developers. Last year it was already known that 2010 was going to be a big year for Android. With several new devices slated to hit the market, industry analysts are expecting its market share to jump considerably. However, if each new device that hits the market brings with it its own set of device specific quirks, this could also cause the Android development community to quickly erode. Applications are a very large part of the mobile experience today, and they will continue to be in the future. But, without a development community, there will be no applications.

I really hope Google finds a way to address this very serious issue.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

[Show as slideshow]

[View with PicLens]

We’ve all been there. Unfamiliar with the product, we are often unable to explain in detail what is going on, especially over the phone. And usually on the other end of the phone is an under paid customer support representative who simply reads from a script given to him by his manager. Often these representatives know very little about the product they are supporting. And, good luck if you happen to have a question that doesn’t appear on the script.

What happened next I will never forget. On the other end of the line appeared this friendly, knowledgeable technician. I described the problem I was having to him. He listened patiently, and asked a couple of targeted questions to help narrow down the issue. In no time at all, he knew exactly what was going on, and exactly how to fix it. Step by step, he clearly instructed me on what needed to be done to get my printer back online. Before you knew it, I was back in business. That technician, that single person, turned my experience around 180 degrees. I was once again a satisfied customer. Hell, I was more than satisfied.

You cannot underestimate the value of awesome customer support. I will gladly pay more for a product or service if I know that it will come with great support, as I know it will save me from headaches and grief down the road.

Typically, developers don’t serve on the front lines of customer support. However, not directly interacting with and supporting the users of your product means you are missing out on a great opportunity. An opportunity to build customer loyalty, an opportunity to understand how a customer is using your product or service, or simply an opportunity to help somebody.

I think developers should spend at least a few weeks a year directly supporting the users of their product. There are several reasons why.

You will learn how people use your product

You think you know how people are using your product? Think again. You think that user interface you designed makes perfect sense? Maybe it does to you and your co-workers, but odds are it is not as clear to your customers. When on customer support, you’ll see first hand how customers are using your product. If you find that your customers are fumbling around your product, thinking they are doing one thing but actually doing another, it’s a clear indication that your product isn’t as intuitive as it should be. Don’t write off these issues as “user errors”. Instead, get some feedback from your customer, and figure out how to make your product easier to use.

It will become obvious what features are missing, what needs to be improved, or what needs to be taken out

Your sales team, marketing team, and product team may have some great ideas regarding what direction to take your product. But, there is nothing quite like hearing it straight from the customer. This doesn’t mean that you should run out and implement every feature, and make every change requested by a customer. But, if you start to hear similar feedback while on support, it may be an indication of a legitimate need. Take note of it! Or, better yet, just do it!

Developers can fix problems…fast

Developers are in a great position to fix problems, fast. They usually know the product more intimately than anybody else on the team, have easy access to the code, and sometimes have the ability to release a patch. This gives developers the unique ability to provide amazing customer support. There have been times where I’ve fixed an issue while still on the phone with a customer, or shortly thereafter. Think about how you would feel if all of your problems were solved this quickly.

Summary

Happy customers become loyal customers. And, providing awesome customer support is one sure way to keep your customers happy. Providing customer support can be very time consuming, sometimes consuming the majority of a developer’s time (which is why I think a few weeks a year is enough). However, this should be thought of not as an expense, but an investment. An investment in the customer and the developer.

Standup Timer is a very simple application that helps keep your stand-up meetings focused, and on track. I’ve been participating in stand-up meetings of one form or another for a few years now. One thing that is constantly an issue with stand-up meetings is that they often run longer than they are supposed to. Instead of simply stating their status and moving on, participants will often start to ramble, or engage in conversations with other members of the team. Those who practice any form of the Agile software development methodology know that the value of stand-up meetings drops considerably if the meeting is constantly allowed to overflow its time constraints.

So, I created an app for that :) Standup Timer helps to keep your stand-up meetings on track by allotting each participant an equal share of time, and letting participants know when they are about to exceed, or have exceeded their time. The notification comes in the form of a bell ring for the warning, and an air horn when the time has run out. If any time is left after all participants have presented their status, then Standup Timer will keep the clock ticking, so the remaining meeting time can be used for an open discussion.

Standup Timer is easy to use. Simply provide the number of participants in the meeting, the length of the meeting, and press Start to start the timer. On the timer screen, press Next when a participant is finished presenting their status to reset the timer for the next participant. When the meeting is over, press Finished. Standup Timer will remember the number of participants and meeting length from the last time you used it. It also allows you to enable/disable sounds, and to specify when (how many seconds left in their share of time) to warn participants that they are about to exceed their share of time.

Standup Timer is free and open source. You can find the application in the Android Market, and the source code on GitHub at http://github.com/jwood/standup-timer.

[Show as slideshow]

[View with PicLens]

What’s wrong with the relational database?

For 99% of the applications out there, absolutely nothing. The relational database has been the industry standard for data storage over the past 30+ years for good reason. It is an incredibly capable piece of software. Although it may not be the best tool for everything it is used for, it certainly satisfies the needs of the vast majority of applications just fine.

However, while not new, the class of applications mentioned above are becoming more common. These applications either handle enormous amounts of traffic, or deal with tremendous amounts of data. The relational database falls short in a few areas when trying to meet the demands of an application like this.

photo credit: bioxid

A single database server is usually not enough to support these requirements. Applications like this need a true database cluster, capable of adding storage space and processing power on the fly without the application even noticing. However, relational databases weren’t designed to operate in a cluster where all machines are capable of reading and writing data. This is largely due to the promises they make regarding data integrity. In order to fulfill these promises, the database needs easy, quick access to all of the data at all times to verify that duplicates aren’t being inserted, constraints aren’t being violated, etc. This quickly becomes a bottleneck when dealing with very large amounts of data.

There are techniques for scaling out relational databases, but they don’t address every concern. One popular technique is to use one or more slave databases for read requests, while continuing to funnel all write requests through the master database. The master database constantly synchronizes with the read only databases, so the data remains consistent between databases. This technique works great for read heavy applications, but does not help applications that perform just as many creates, updates, and deletes. Data sharding is another popular technique, which involves splitting the data up onto several different databases based on some criteria. But this pushes an extraordinary amount of complexity onto the application, as it is now responsible for determining which database to use for specific data sets. Master-master replication can be used to keep multiple master databases in sync, so any database server can perform read or write operations. However, for some applications there comes a point where the replication can’t keep up with the traffic.

Relational databases are also (intentionally) very strict when it comes to the structure of the data being stored. Data must be broken up into a series of rows and columns. Good object/relation mapping tools hide much of this awkwardness from us, but some applications deal with data that doesn’t map well into rows and columns. A simple key/value store is usually a better fit for applications like this.

How does the new breed address these problems?

The new breed of data stores, called NoSQL databases, make very few promises regarding data integrity. In this new model, data integrity becomes the application’s concern. By not having to enforce any complex data integrity rules, NoSQL databases can scale to levels way beyond that of a relational database. Adding more processing power or storage capacity can be as simple as adding a new machine to the cluster. The database can then store and process the data using any machine in the cluster.

In this model, the data being stored is self contained, and does not rely on any other data in the database. Therefore there is no need for one machine to know anything about the other machines in the cluster. This approach is quite different from the relational model, where data is broken up into multiple tables to eliminate duplicate data, and joined back together when being accessed.

Most of these databases subscribe to a theory called eventual consistency. In situations where duplicate information is scattered across different servers in the cluster, it is not feasible for the database to find all instances of that data and update it as a part of the original operation. Instead, the data will be replicated to the other database servers at a later time. Until that replication takes place, the application will be in an inconsistent state, where simultaneous queries fetching the same data could return different results. Although this sounds terrible, it turns out that in practice it is really not too big of a deal for most applications. Do all customers of an online retailer need to see the exact same set of product reviews 100% of the time? Probably not.

Also, because there are few promises regarding data integrity, NoSQL databases can offer data storage that is much more flexible. The database no longer has to enforce the uniqueness of a column, or ensure that the id of some referenced piece of data actually exists in the database. Some of these databases are true key/value data stores, where you can store just about anything. Others require a certain document format to be used (such as JSON or XML), but still allow you to freely change the contents of that document as you wish.

Still no one-stop-shop for persistence

Although NoSQL databases address some issues that can’t be addressed by relational databases, the opposite is true as well. The relational database offers an unparalleled feature set. While some of these features prevent it from serving the needs of the class of applications described above, they are absolutely required by other classes of applications. In some domains, data integrity is the number one concern. You need to look no further than the classic “try to withdraw money from the same account at the same time” example to justify the need for locks and transactions.

For the vast majority of applications out there, relational databases work great. There are a boat load of tools and libraries that support them, and software developers are very familiar with how to use them. It is safe to say that the relational database has secured its spot in IT departments and data centers around the world, and it isn’t going anywhere. It is far from dead.

Polyglot persistence

An increasing amount of case studies are appearing that describe how real world applications are needing the data integrity offered by the relational database in addition to the benefits offered by NoSQL databases. I believe this trend will continue, as companies are storing more data than ever, and processing that data in different ways than previously imagined.

To address these needs, some companies are beginning to run their relational database side-by-side with one or more of the NoSQL alternatives. Extremely large data sets that require scalable storage space and processing power are moved to a NoSQL database, while everything else, especially data that needs its integrity kept in-check, remains in the relational database. The term Polyglot Persistence has been used to describe the use of multiple databases within the same project.

The benefits of polyglot persistence

The benefits are somewhat obvious. By running a relational database side-by-side with a NoSQL database, you get the best of both worlds. Strict enforcement of data integrity from the relational database, and the scalability and flexibility provided by the NoSQL database. This allows you to use the best tool for the job, depending on your use case.

photo credit: jypsygen

There are a few scenarios where I’ve seen systems take advantage of polyglot persistence. The first scenario involves the need to perform some set of complex calculations on an extremely large data set. The data is either copied/moved from the relational database to the NoSQL database, or inserted directly into the NoSQL database by the application. The application can then use a cluster of NoSQL database servers can then divide the work, process the data, and aggregate the results. The more machines you have in your cluster, the less time the processing will take. The resulting data can either remain in the NoSQL database or be inserted into the relational database, depending on what needs to be done with the results.

The other scenario takes advantage of the schema-less nature of some NoSQL databases. While it is certainly possible to store a serialized data structure in a single column of a relational database, interacting with that data can be a bit more challenging than if that data were in a schema-less, document oriented database. This use case, after all, is what the documented oriented databases were designed for. These types of databases simply treat the data as a collection of key/value pairs, identified by a unique ID. The NoSQL databases provide ways in which you can add structure back into the document so the data inside the document can be queried. These databases are great for storing data that can be radically different from document to document, or data whose structure changes constantly.

The challenges of polyglot persistence

Polyglot persistence comes with its own set of challenges. While potentially getting the best of both worlds as far as features go, you get the complexity and hassle of dealing with not only multiple databases, but multiple databases models.

Determining which database to use to store certain data

With more than one database, you now have to decide where to store the data. It’s no longer a given. If you make the wrong decision, you could be looking at a painful migration from one database model to another as a result. To make this decision, you need to carefully examine how the data will be used.

Increased application complexity

Applications also face increased complexity as they now have to interface with two different (potentially very different) data stores. If done correctly, you should be able to isolate this complexity to the persistence layer of your application, freeing the rest of the application from having to know what database specific data is coming from. But, interfacing to multiple data stores could greatly increase the complexity of that data persistence layer. Your application will now need to know:

• How to connect to each of the databases
• What database to use for specific sets of data
• How to handle the different types of errors from each database
• How to map results from each database back to your application’s object model
• How to handle queries for information across databases
• How to mock out the different databases for testing
• Potentially, how to move data from one database to another

Addressing these concerns could result in a bunch of new application code, and with added code usually comes added complexity, and more bugs.

Increased deployment complexity

In addition to the increased application complexity, you will also face increased deployment complexity.

• Will you need to provision new hardware to host the new database?
• How will you backup the data in the new database?
• How will you manage and control changes to the configuration of the new database?

Training for developers and operational staff

Given that this database will likely be radically different from the relational database that your developers and operational staff are comfortable with, how will you bring them up to speed on how to use and manage this new database? And, given that the majority of the NoSQL databases are still very young, how will you keep your developers and operational staff up to speed with the latest developments on the project?

This is a big issue, especially in companies with large development and operations teams, and needs to be thought through carefully.

• Is there an expert you can hire to help you get up and running, and mentor your staff?
• Is there any training available that you can give to your staff?
• Who can you turn to for support when something goes wrong in production?

Summary

I’ve always been a big advocate of using the right tool for the right job. For the past 30 years, the relational database has been the de-facto standard for persistence. Creative people have managed to utilize and manipulate it to serve all sorts of different use cases, quite successfully. But just because you can fit a square peg through a round hole if you hit it with a big enough hammer doesn’t necessarily mean that you should.

NoSQL databases can be great tools for addressing data persistence cases that the relational database struggles with. In addition, each NoSQL database brings its own set of strengths and weaknesses to the table. They are becoming very important tools to have around, and I believe that our industry will see a steady increase in the adoption of these tools going forward.

Jumping into the Rails source, I found the offending line.

if configuration.frameworks.include?(:action_controller) &&
  ActionController::Base.session_store.name == 'ActiveRecord::SessionStore'

This code assumes that a session store is configured in your Rails app. However, this particular application has no need for a session, so we were disabling it in the configuration by specifying the following:

config.action_controller.session_store = nil

I poked around on the web for a while, trying to find another way to disable the session. No luck. It appeared that the only other way to disable the session was to properly configure a session store in your environment.rb file, and then disable it in your ApplicationController. That seemed lame. Why should I have to configure something that I want to disable?

So, I coded up a simple class to act as the session store for the application that simply raises an error if anybody tries to access the session.

class NilSessionStore < ActionController::Session::AbstractStore
  def get_session(env, sid)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end

  def set_session(env, sid, session_data)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end
end

I then configured the application to use this class as the session store.

config.action_controller.session_store = :nil_session_store

Nice and simple, and it keeps me from having to configure something I never plan to use.