Standup Timer version 1.2 has just been released. Per popular demand (2 requests :)), I have added support for meetings of any length, with any number of participants. Prior to this release, Standup Timer restricted you to meetings 5, 10, 15 or 20 minutes long, and a maximum of 20 participants. These restrictions are still enabled by default, but can be disabled via the application’s settings.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

Standup Timer

Now that Standup Timer has a few releases under its belt, I thought it would be a good time to reflect on my experience with the Android SDK, and Android development in general. But before I begin, I should mention a few points about my background, to help you understand my perspective.

First, Standup Timer was my first mobile app. Aside from a very small amount of time I’ve spent playing around with the Blackberry development kit (really too small to even mention), I have had no prior experience developing software for a mobile device. My background is largely web development and distributed applications development. Second, I have been doing Java development for quite some time, so I am very comfortable working in Java. Third, if you’re not familiar with Standup Timer, it is a very simple application. It only interacts with a few components of the Android platform, and only consists of a few screens. No GPS, no web, no multi-touch, etc. So, my tour of the Android SDK was far from complete.

Dealing with multiple devices

Supporting multiple screen sizes

The topic of supporting multiple screen sizes has been a popular one, especially when comparing Android development to iPhone development. Supporting multiple screen sizes in your application does add a little complexity to the development process, but for my app (and I’d imagine most apps), it is very manageable.

This issue did not take Google by surprise. They knew that if Android was to be successful, it would need support devices with a wide range of physical characteristics, including screen size. So, from the beginning, they made it possible for applications to support multiple screen sizes with little effort. The Android platform contains many features to help with this, including the ability to pre-scale images for phones with different resolutions, allowing you to specify your screen component sizes in density independent pixels (dips), and allowing you to easily center and stretch your components to fill the screen. These features, along with a set of best practices for supporting multiple screen sizes, are documented very well at http://developer.android.com/guide/practices/screens_support.html.

For applications that do not use the standard Android view components, or do custom graphics, it may be a different story. I have read blog posts from some Android game developers saying it is a larger issue, and some who say it is not. So, I’m not sure. However, for the vast majority of applications, the standard view components work just fine, making supporting multiple screen sizes a very manageable issue.

One thing that did bite me regarding multiple screen sizes is the fact that views will not scroll by default if they happen to flow off the screen. You need to anticipate this, and wrap any views that may flow off the screen with a ScrollView to enable scrolling behavior.

Supporting multiple devices

To me, this issue is much larger than the multiple screen size issue. Android is an open source project, which gives anybody the ability to modify the code however they wish. Google prevents carriers and cell phone manufacturers from abusing this by holding back a few key applications from the open source release, including the Android Market application. Without the Android Market application, phones would not have access to the 20,000 apps currently available for the Android platform.

However, phone manufacturers do tweak the platform to work for their specific hardware. Android 2.1 on one device is not necessarily the same as Android 2.1 on another device. For an example of this, compare 2.1 on the Motorola DROID with 2.1 on the Nexus One. These seemingly minor modifications can be very troublesome for application developers. The web has been flooded with reports of developers growing frustrated with the complexity matrix of Android versions and the exploding number of phones running them, all potentially containing their own tweaks to the platform. Browsing through the change logs of applications I have installed on my Motorola DROID, I can see that several apps have made changes to fix issues on specific devices.

This issue is forcing developers to not only test on every device they plan on supporting, but also to write device specific code to work around any known issues for a device. As Android continues to grow, this level of support will be unsustainable. Developing for a common platform should mean that your application will run fine on any device running that platform. But, because of these device specific tweaks, this is quickly turning out not to be the case for Android. I worry that if Google doesn’t find some way to control this, developers will continue to abandon the platform.

I have even run into what appear to be device related issues with Standup Timer, which is miniscule compared to the size and complexity some of the other apps available in the Android Market. Standup Timer uses an Android API that prevents the user’s screen from blacking out while a timer is in progress, letting you always see how much time is remaining for a meeting. Just the other day somebody left a comment in the Market indicating that this was not working on their device. So far, all of the other reviews have been positive, leading me to believe that this could be an issue with that user’s particular device. The commentator didn’t provide any information about the device they were using, or how they could be contacted. So, I’m not sure if I’ll ever be able to track down this issue. Even if I knew the device experiencing the issue, I’m not sure I would be able to help. Unless you have access the device in question, reproducing these issues is practically impossible. The Android simulator tool is great, but you cannot create a simulator for a specific Android device, running that device’s flavor of Android. You can only create simulators running the “generic” version of an Android release.

Development

Application life cycle

The Android platform primarily communicates with an application through a series of life cycle events. The platform will let the application know when an activity (a screen) has been created, paused (lost focus), resumed (regained focus), etc. The life cycle events are easy to understand, and are specific enough so that I never need to examine the state of my application or the platform within one of these callbacks. The fact that a specific life cycle method was called tells you what state your application is in.

My largest complaint about how the platform manages an application is how it destroys and re-creates the activity (the object that backs the screen) when the phone rotates from portrait mode to landscape mode, or visa versa. It seems to me that this would best be handled by making the appropriate life cycle callbacks to the same activity instance, giving it a chance to redraw the screen. Instead, you are forced to save all of your activity’s state, and then reload it when the new activity instance takes over. Forgetting to save the state of a particular variable means that value will be lost when the phone is rotated. This was the source of several bugs when writing Standup Timer.

Android APIs

I have no major issues with the Android APIs. They seem complete (for what I needed), and most importantly, behaved as expected. The database API seems a bit archaic, especially in today’s age of sophisticated O/R mapping tools. But, at least I didn’t have to catch SQLException after each operation, like the JDBC API.

However, the lack of decent documentation for some of the APIs is a problem. There were a few instances where I was forced into using trial and error to determine the purpose of a method parameter. I feel this level of unclarity is unacceptable for a public API.

UI Creation

The Android SDK offers developers two methods for creating UI screens: programatically using Java code or declaratively using XML. I did not attempt the programatic approach, as it brought back too many nightmares of building UIs for Java applets. The XML approach to creating screen layouts, which is the recommended approach, is very straight forward. Java developers, especially those working in the “enterprise”, have grown to hate XML over the past few years, due to its verbosity and proliferation. But, I think Android’s use of it is very tasteful. At no point did I consider myself to be in XML-hell. Looking back on the final XML that declares my user interface, it doesn’t seem overly verbose or complicated.

I did however struggle at times to figure out how to get the UI to look exactly the way I wanted it. Again, this seems to be the result of poor documentation of the XML elements and their corresponding attributes. Some more complex examples in the sample code included in the SDK would have also helped.

If you keep your screens simple, and don’t display too much information on any given screen, then you shouldn’t have much trouble dealing with how that screen looks in portrait or landscape mode. Screens with a list of items, for example, will usually look just fine when viewed in either portrait or landscape mode. When in landscape mode, the user will just see a little more blank space in the list. For more complicated screens, the Android platform allows you to specify an alternate UI layout for landscape mode. This allows you to completely restructure the elements on a screen to deal with the the wider, shorter screen size without affecting how the screen looks in portrait mode. Given how I struggled to get the views to look exactly the way I wanted, I tried my best to avoid the need to specify an alternate layout file for landscape views. Only two of the screens in Standup Timer actually needed a landscape specific layout file.

adb

The Android Debug Bridge (adb) is a nifty little tool that allows you to interact with an Android device or simulator from the command line. You can install/uninstall applications, interact with files on the device, or run a shell on the device. It also provides a series of commands that allow you to more easily create scripts to run on the device. Not only is this tool useful by itself, but it also enables the creation of sophisticated tools for Android development. Learning how to use adb is a wise investment of your time.

The Eclipse plugin

The Eclipse plugin for Android development is great. It provides tight integration between Eclipse and the Android SDK. Code completion is available for the Android APIs as well as the different types of XML files used by the Android platform. It has great support for Android resource files. You can easily start your application on a simulator with the click of a button, and running your tests is a breeze. I would not attempt to write an Android application without this plugin, or a similar plugin for a different IDE.

However, I did run into a few issues regarding how Eclipse interacts with the Android simulators. On more than a few occasions, when starting the application or running the tests, the Eclipse plugin started a simulator running a version of the Android platform that did not match my project settings (although, I have not completely ruled out user error here :) ). Also, the Eclipse plugin seems to have a few bugs around interacting with multiple simulators. adb allows you to specify the target device or simulator when issuing a command, which is useful when you have more than one simulator running at a time (which is almost always the case given the current state of the Android platform). However, the Eclipse plugin doesn’t always utilize this capability. Sometimes it will detect multiple simulators running, and ask you which one you’d like to target. Other times it will not. In addition to the simulator interaction issues, there are a few areas that could benefit from a bit of polish. The UI builders, for example, are difficult to work with. It became obvious very quickly that I would be better hand coding the XML for the UI than attempting to use the builders.

Testing

Simulators are awesome

The Android simulators are awesome. Once you have the proper versions of the SDK installed, it is trivial to create simulators with different screen sizes, different amounts of available storage, running different versions of the Android platform. And, as previously mentioned, adb is great for interacting with the simulators. While not perfect (you should always test your app on a real device before publishing it), they are pretty darn close.

The only thing missing, and I’m not sure how feasible this is, is the ability to create a simulator for a specific device. For example, if I know that my app is crashing on the Nexus One for some reason, I’d love to be able to create a Nexus One simulator, so I can find and fix the issue. Currently, debugging and fixing device specific issues is not possible without access to the device in question. I know some large mobile shops purchase the devices they plan on supporting, for testing purposes. And, services like Device Anywhere allow you virtual access to a physical device, which would satisfy this need. But these options, especially the first one, are expensive. A developer trying to fix a device specific issue in their free, non ad subsidized application will not have the resources for either of these options.

Automated (unit and functional) Testing

Automated testing on the Android platform is PAINFUL. It’s hard to decide where to even begin. First off, the tests provided with the sample applications in the SDK are very bare bones. Most of them simply assert that an activity has been created, and that’s it. If there was sufficient documentation describing how to write tests for you application, this wouldn’t be that big of a deal. But, there isn’t. Luckily, guys like Diego Torres Milano are writing blog posts and giving presentations in an attempt to fill this gaping hole in documentation. The slides from Diego’s presentation at Droidcon 2009 are a great place to start. Standup Timer, which is open source, also has a complete set of unit and functional tests for you to check out if you so desire.

Another pain point regarding automated tests for Android applications is that the tests cannot be run inside a standard Java Virtual Machine. The implementation of the APIs in the android.jar file provided with the SDK simply throw exceptions. That jar file is only meant for building your application, not running it. So, all tests must be run on the device (or in a simulator). This dramatically slows down the execution of the tests. One alternative proposed by a non-Android group inside of Google is to mock out all of the Android components using a mocking library, enabling the execution of the tests in a standard virtual machine. Although this would speed up the test execution, it comes with its own drawbacks. Mainly, you end up having to mock out a TON of stuff. I think that mocking out too many components significantly reduces the value of your tests. If you mock out everything, what are you really testing?

The SDK contains a few base classes that you can extend for certain types of tests. In particular, the SDK provides support for functional tests (the testing of a particular activity, or screen) and unit tests (the testing of some underlying support code). Writing unit tests is pretty straight forward. With a little bit of investigation and some trial and error, I was able to create a set of tests for my database layer, which will execute against a test version of the application’s database.

Functional tests were much more problematic. In addition to executing much slower than unit tests, they are also more difficult to write. For functional tests, you write code to perform the operations a user would be performing on their phone (entering text, pressing buttons, etc). The problem is that it is not very straight forward to perform these operations via code. It took me quite some time to figure out how to simply enter some text in a text area and press a button via code. Even when I thought I had functional testing figured out, there was still a use case I was unable to write a test for. Most of the issues revolve around gaining access to the components you wish to interact with. For a simple text box or button, it is very easy. You can simply fetch the view by its id, the same way you do in your application. However, once you start nesting components, it becomes much more complicated. I tried several different ways to invoke the context menu (the long press menu) of a list item, inside of a list, which is inside of a tab. None of them worked.

Instead of writing functional tests for activities, I found it much easier to test activities using the ActivityUnitTestCase class with a mocked out instance of the activity. Test classes that extend ActivityUnitTestCase don’t have as much as access to the platform as functional tests. Several API methods will throw exceptions if called from within a ActivityUnitTestCase, and other API methods simply do nothing. However, this can be addressed with some simple tools that should be in every developers toolbox: mocking and stubbing. It is trivial to contain a method call that would throw an exception during testing within a method of its own. This method can then be overridden by a mock implementation of the activity. The mock can then optionally set a flag when the method is called, so the test can verify that some action was performed. One place I do this is where the activity under test tries to start another activity. Calling startActivity from within a ActivityUnitTestCase does nothing. So, I placed the call to startActivity by itself in a protected method, overrode that method in the mock activity class, set a flag in the mock method implementation, and checked to see if that flag was set in the test case to verify that the activity was started.

Deployment

Publishing to the Android Market

Publishing an application to the Android market could not be any easier. The process is very well documented on the Android developers website, and the Eclipse plugin makes creating and signing an installation package a point and click operation.

A fee of $25 is required to create a Android Developers account. This account gives you the ability to upload your application to the Android Market using the Android Developers website. Once published, the application can immediately be found in the Android Market. There is no approval process. Updating your application is just as easy. Simply upload the new version of your application, and it is available in the Android Market immediately.

It should be noted that there is no fee to install the application on your phone.

Summary

Android is a great platform. It is feature rich, and developer friendly. The devices that run it are powerful and easy to use. I also love the fact that it is built on open source technology, and that Google, for the most part, has continued this tradition of openness.

But, the platform is not without problems. Most of my complaints here are minor, and are very addressable. Android is still a young platform, so there is plenty of room to improve. What worries me most about the future of the Android platform is the complexity matrix I talk about in the Supporting Multiple Devices section.

Android was created to be a common platform for mobile application development. A platform where application developers could write a single application capable of running on numerous devices. However, this dream is at risk of becoming just that, a dream. Unless Google can find a way to minimize the impact of device specific issues, it risks losing more and more Android developers. Last year it was alraedy known that 2010 was going to be a big year for Android. With several new devices slated to hit the market, industry analysts are expecting its market share to jump considerably. However, if each new device that hits the market brings with it its own set of device specific quirks, this could also cause the Android development community to quickly erode. Applications are a very large part of the mobile experience today, and they will continue to be in the future. But, without a development community, there will be no applications.

I really hope Google finds a way to address this very serious issue.

android, Standup Timer

Version 1.1 of Standup Timer has just been released. This new version stores statistics for your stand-up meetings on a per-team basis. Simply create one or more teams using the new Teams menu option, then specify the team holding the meeting when starting the timer. To view statistics for a given team, use the Teams menu option to pull up the list of teams, and select the team you would like to see statistics for. The Stats tab will display the average statistics for that team across all of its meetings. For a list of the meetings that team has held, click on the Meetings tab. Finally, to see statistics for a specific meeting, simply select the meeting from the list of meetings.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

[View with PicLens]

android, Standup Timer

A couple of years ago I bought a new printer for the house; a nice 3-in-one printer/fax/copier. I always research the electronics I purchase, and this particular model had received many very positive reviews. So, you can bet I was surprised, and a little pissed off, when the printer started acting up just a few short months after I purchased it. I tried like hell to figure out the issue, hoping my efforts would save me from a dreaded call to the manufacturer’s customer support hot line. No such luck. After a few hours of troubleshooting, I looked up the support number in the back of the users guide, dialed it, and prepared for the worst.

We’ve all been there. Unfamiliar with the product, we are often unable to explain in detail what is going on, especially over the phone. And usually on the other end of the phone is an under paid customer support representative who simply reads from a script given to him by his manager. Often these representatives know very little about the product they are supporting. And, good luck if you happen to have a question that doesn’t appear on the script.

What happened next I will never forget. On the other end of the line appeared this friendly, knowledgeable technician. I described the problem I was having to him. He listened patiently, and asked a couple of targeted questions to help narrow down the issue. In no time at all, he knew exactly what was going on, and exactly how to fix it. Step by step, he clearly instructed me on what needed to be done to get my printer back online. Before you knew it, I was back in business. That technician, that single person, turned my experience around 180 degrees. I was once again a satisfied customer. Hell, I was more than satisfied.

You cannot underestimate the value of awesome customer support. I will gladly pay more for a product or service if I know that it will come with great support, as I know it will save me from headaches and grief down the road.

Typically, developers don’t serve on the front lines of customer support. However, not directly interacting with and supporting the users of your product means you are missing out on a great opportunity. An opportunity to build customer loyalty, an opportunity to understand how a customer is using your product or service, or simply an opportunity to help somebody.

I think developers should spend at least a few weeks a year directly supporting the users of their product. There are several reasons why.

You will learn how people use your product

You think you know how people are using your product? Think again. You think that user interface you designed makes perfect sense? Maybe it does to you and your co-workers, but odds are it is not as clear to your customers. When on customer support, you’ll see first hand how customers are using your product. If you find that your customers are fumbling around your product, thinking they are doing one thing but actually doing another, it’s a clear indication that your product isn’t as intuitive as it should be. Don’t write off these issues as “user errors”. Instead, get some feedback from your customer, and figure out how to make your product easier to use.

It will become obvious what features are missing, what needs to be improved, or what needs to be taken out

Your sales team, marketing team, and product team may have some great ideas regarding what direction to take your product. But, there is nothing quite like hearing it straight from the customer. This doesn’t mean that you should run out and implement every feature, and make every change requested by a customer. But, if you start to hear similar feedback while on support, it may be an indication of a legitimate need. Take note of it! Or, better yet, just do it!

Developers can fix problems…fast

Developers are in a great position to fix problems, fast. They usually know the product more intimately than anybody else on the team, have easy access to the code, and sometimes have the ability to release a patch. This gives developers the unique ability to provide amazing customer support. There have been times where I’ve fixed an issue while still on the phone with a customer, or shortly thereafter. Think about how you would feel if all of your problems were solved this quickly.

Summary

Happy customers become loyal customers. And, providing awesome customer support is one sure way to keep your customers happy. Providing customer support can be very time consuming, sometimes consuming the majority of a developer’s time (which is why I think a few weeks a year is enough). However, this should be thought of not as an expense, but an investment. An investment in the customer and the developer.

support

Putting my brand new Motorola DROID to work, I just completed my first Android application, which I’m calling Standup Timer.

Standup Timer is a very simple application that helps keep your stand-up meetings focused, and on track. I’ve been participating in stand-up meetings of one form or another for a few years now. One thing that is constantly an issue with stand-up meetings is that they often run longer than they are supposed to. Instead of simply stating their status and moving on, participants will often start to ramble, or engage in conversations with other members of the team. Those who practice any form of the Agile software development methodology know that the value of stand-up meetings drops considerably if the meeting is constantly allowed to overflow its time constraints.

So, I created an app for that :) Standup Timer helps to keep your stand-up meetings on track by allotting each participant an equal share of time, and letting participants know when they are about to exceed, or have exceeded their time. The notification comes in the form of a bell ring for the warning, and an air horn when the time has run out. If any time is left after all participants have presented their status, then Standup Timer will keep the clock ticking, so the remaining meeting time can be used for an open discussion.

Standup Timer is easy to use. Simply provide the number of participants in the meeting, the length of the meeting, and press Start to start the timer. On the timer screen, press Next when a participant is finished presenting their status to reset the timer for the next participant. When the meeting is over, press Finished. Standup Timer will remember the number of participants and meeting length from the last time you used it. It also allows you to enable/disable sounds, and to specify when (how many seconds left in their share of time) to warn participants that they are about to exceed their share of time.

Standup Timer is free and open source. You can find the application in the Android Market, and the source code on GitHub at http://github.com/jwood/standup-timer.

[View with PicLens]

android, Standup Timer

The mobile industry is exploding. In just a few short years, everybody will have a smart phone. A tiny, internet connected, mini computer right in their pocket. As each month passes, we learn more and more about what the future of this industry holds, and what the mobile handset landscape will look like. If one thing is for certain, it’s that nothing is for certain.

As it stands right now, Apple is king with consumers. Though not the first ones to market with an internet connected mobile device (the Blackberry has been around for a long time, and still holds the majority of the smartphone market share), Apple appears to be the first to really understand what the average, non-corporate consumer wants. A true mini computer. An open device that can play music, run applications, take photos, and provide a pleasant web browsing experience. And they did it in true Apple fashion, making the device extremely easy to use. As a result, the iPhone has become extremely popular with consumers, and is widely considered ”the device to have”.

With the majority of people still without a smartphone, much of the market remains up for grabs. Apple’s competitors are scrambling to catch up, trying to ensure that they they get a piece of the pie. But, one very important question lingers. What can Apple’s competitors offer that would give the average consumer a reason to buy their device instead of buying an iPhone? To me, the reasons are few, and becoming fewer.

(I’d love to hear your reasons in the comments. So please, chime in.)

A comparable feature set

This pretty much goes without saying. Any challenger to the iPhone crown must offer similar features to that of the iPhone. It is very unlikely that a competing device will lure anybody away from the iPhone if it is missing a feature that is now expected to be there. The device must be capable of running apps, taking photos, playing music, etc, for it even to be considered.

A better network

AT&T’s network leaves much to be desired. Having never been an AT&T customer, I can only relay the opinions of my friends and family who are AT&T customers. However, their opinions are one in the same. I’ve not heard a single word of praise when it comes to AT&T’s network. All of my friends and family with iPhones have expressed frustration that the device they love is frequently crippled by a network that is spotty and congested.

It’s no secret that Apple has an exclusive agreement with AT&T, and that agreement has an expiration date. Rumors have been circulating about a jump to another carrier, possibly Verizon, sometime next year. The more wireless carriers offering the iPhone, the less valid of a reason this will become for not purchasing one.

A comparable application ecosystem

Competing devices will need to have an application ecosystem that is at least comparable to the iPhone’s. This is no small task. There are over 100,000 applications in the App Store. Sure, several offer the same functionality, and many are of very poor quality. However, nobody can argue with Apple’s tag line of “There’s a app for that”. There really is an application, in most cases many, for everything you could possibly want to do with your iPhone.

Given their head start, beating Apple at this game will not be easy. Google’s Android OS currently stands the best chance of challenging Apple on this front, with over 10,000 applications already available. The Android OS is open, and capable of running on hardware from any manufacturer. In addition, applications written for Android are capable of running on any device that runs the OS (for the most part). Next year is going to be a big one for Android, with several new devices coming to market from many different manufacturers. Some analysts are even predicting that the number of Android devices in the hands of consumers will surpass the number of iPhones by 2012. This will no doubt attract more application developers to the platform.

However, Android has its own set of challenges awaiting. The fact that manufacturers are free to run Android on devices with very different hardware specifications (screen size, input controls, etc) poses a major challenge for application developers. Perhaps the risk of rendering thousands of existing Android applications useless by releasing a device with dramatically different hardware specs will be enough to convince manufacturers not to do it. Perhaps Google will provide a set of Android APIs that can help application developers deal with this issue. Perhaps a set of best practices will emerge as a guide for developers looking to tackle this issue. Perhaps we’ll see something similar to the PC application market in the mid-late 90’s (and the Blackberry application market today), where only certain devices will be capable of running certain applications. Only time will tell if these issues will prevent the development of the Android application ecosystem.

A killer feature

One wild card that is always in play is the killer feature. Apple’s competitors are only one, innovative, killer feature away from stealing the spotlight for themselves. By “killer feature”, I mean a feature so awesome that when you see it in action, you say to yourself, “I need one of those!”.

Version 2.0 of the Android OS took a stab at this with the introduction of Google Maps Navigation. A fantastic feature, Google Maps Navigation morphs your mobile device into a fully functional GPS unit, complete with a synthesized voice telling you where to go, real time traffic information, and several map overlays showing you the location of everything from ATMs to gas stations. But, is this a killer feature? Frankly, I’m not sure. But, its announcement was enough to cause a significant drop in the stock price of traditional GPS manufacturers, and it certainly has potential.

An incredibly easy to use device

Making devices that are intuitive and easy to use has always been one of Apple’s strengths. Look no further than the iPod for an example of this. Competing devices will need to be as easy to use as the iPhone is to appeal to the average consumer.

How do I get my music onto the device? How to I get the photos I take off? These operations should be simple and intuitive. Motorola’s new Android 2.0 device, the DROID, is seriously lacking in this area. Several steps are required to store data on or pull data off of the device:

• Attach the device to your computer
• Use the device’s menu system to instruct it to mount itself as an external drive
• Locate the files on your hard drive that you would like to store on the device
• Copy and paste the files from your hard drive onto the device
• Unmount the device

For the iPhone, the list of steps is much smaller.

• Attach the device to your computer, and let iTunes do the rest

Are the steps required to store data on the DROID too much to handle for an experienced computer user. No, of course not. But, there is still a large percentage of people out there who would struggle with completing those tasks. Believe me, I know. Many are family and friends of mine who I help complete “simple” tasks on their computers all of the time. These people make up a significant portion of the market. If you want them to buy your device, then you have to make it stupid simple to use.

Summary

Apple has set the bar high with the iPhone, very high. While I can think of several reasons why developers and techies would prefer a different device, I can’t think of many reasons why the average consumer would. And, there are a lot more average consumers than there are geeks.

But make no mistake, Apple’s competitors have the iPhone in their sights. The tide can shift very quickly in this market, especially since most people get a new phone every couple of years. Will the iPhone challengers be able make a dent in the iPhone’s market share? Or, will the iPhone be the de-facto standard for smart phones? Only time will tell.

android, iPhone, mobile development

The days of the relational database being a one-stop-shop for all of your persistence needs are over. A new class of application is beginning to emerge with requirements that exceed the capabilities of the relational database. Some of these applications need unlimited scalability or bullet proof fault tolerance, while others may require blazing fast access or flexible data storage. The relational database was simply not designed to meet the needs of this small but growing class. Instead, a new breed of data stores are gaining momentum. These data stores are looking at data persistence with a fresh set of eyes, diverging from the relational model considerably in order to meet these challenges.

What’s wrong with the relational database?

For 99% of the applications out there, absolutely nothing. The relational database has been the industry standard for data storage over the past 30+ years for good reason. It is an incredibly capable piece of software. Although it may not be the best tool for everything it is used for, it certainly satisfies the needs of the vast majority of applications just fine.

However, while not new, the class of applications mentioned above are becoming more common. These applications either handle enormous amounts of traffic, or deal with tremendous amounts of data. The relational database falls short in a few areas when trying to meet the demands of an application like this.

photo credit: bioxid

A single database server is usually not enough to support these requirements. Applications like this need a true database cluster, capable of adding storage space and processing power on the fly without the application even noticing. However, relational databases weren’t designed to operate in a cluster where all machines are capable of reading and writing data. This is largely due to the promises they make regarding data integrity. In order to fulfill these promises, the database needs easy, quick access to all of the data at all times to verify that duplicates aren’t being inserted, constraints aren’t being violated, etc. This quickly becomes a bottleneck when dealing with very large amounts of data.

There are techniques for scaling out relational databases, but they don’t address every concern. One popular technique is to use one or more slave databases for read requests, while continuing to funnel all write requests through the master database. The master database constantly synchronizes with the read only databases, so the data remains consistent between databases. This technique works great for read heavy applications, but does not help applications that perform just as many creates, updates, and deletes. Data sharding is another popular technique, which involves splitting the data up onto several different databases based on some criteria. But this pushes an extraordinary amount of complexity onto the application, as it is now responsible for determining which database to use for specific data sets. Master-master replication can be used to keep multiple master databases in sync, so any database server can perform read or write operations. However, for some applications there comes a point where the replication can’t keep up with the traffic.

Relational databases are also (intentionally) very strict when it comes to the structure of the data being stored. Data must be broken up into a series of rows and columns. Good object/relation mapping tools hide much of this awkwardness from us, but some applications deal with data that doesn’t map well into rows and columns. A simple key/value store is usually a better fit for applications like this.

How does the new breed address these problems?

The new breed of data stores, called NoSQL databases, make very few promises regarding data integrity. In this new model, data integrity becomes the application’s concern. By not having to enforce any complex data integrity rules, NoSQL databases can scale to levels way beyond that of a relational database. Adding more processing power or storage capacity can be as simple as adding a new machine to the cluster. The database can then store and process the data using any machine in the cluster.

In this model, the data being stored is self contained, and does not rely on any other data in the database. Therefore there is no need for one machine to know anything about the other machines in the cluster. This approach is quite different from the relational model, where data is broken up into multiple tables to eliminate duplicate data, and joined back together when being accessed.

Most of these databases subscribe to a theory called eventual consistency. In situations where duplicate information is scattered across different servers in the cluster, it is not feasible for the database to find all instances of that data and update it as a part of the original operation. Instead, the data will be replicated to the other database servers at a later time. Until that replication takes place, the application will be in an inconsistent state, where simultaneous queries fetching the same data could return different results. Although this sounds terrible, it turns out that in practice it is really not too big of a deal for most applications. Do all customers of an online retailer need to see the exact same set of product reviews 100% of the time? Probably not.

Also, because there are few promises regarding data integrity, NoSQL databases can offer data storage that is much more flexible. The database no longer has to enforce the uniqueness of a column, or ensure that the id of some referenced piece of data actually exists in the database. Some of these databases are true key/value data stores, where you can store just about anything. Others require a certain document format to be used (such as JSON or XML), but still allow you to freely change the contents of that document as you wish.

Still no one-stop-shop for persistence

Although NoSQL databases address some issues that can’t be addressed by relational databases, the opposite is true as well. The relational database offers an unparalleled feature set. While some of these features prevent it from serving the needs of the class of applications described above, they are absolutely required by other classes of applications. In some domains, data integrity is the number one concern. You need to look no further than the classic “try to withdraw money from the same account at the same time” example to justify the need for locks and transactions.

For the vast majority of applications out there, relational databases work great. There are a boat load of tools and libraries that support them, and software developers are very familiar with how to use them. It is safe to say that the relational database has secured its spot in IT departments and data centers around the world, and it isn’t going anywhere. It is far from dead.

Polyglot persistence

An increasing amount of case studies are appearing that describe how real world applications are needing the data integrity offered by the relational database in addition to the benefits offered by NoSQL databases. I believe this trend will continue, as companies are storing more data than ever, and processing that data in different ways than previously imagined.

To address these needs, some companies are beginning to run their relational database side-by-side with one or more of the NoSQL alternatives. Extremely large data sets that require scalable storage space and processing power are moved to a NoSQL database, while everything else, especially data that needs its integrity kept in-check, remains in the relational database. The term Polyglot Persistence has been used to describe the use of multiple databases within the same project.

The benefits of polyglot persistence

The benefits are somewhat obvious. By running a relational database side-by-side with a NoSQL database, you get the best of both worlds. Strict enforcement of data integrity from the relational database, and the scalability and flexibility provided by the NoSQL database. This allows you to use the best tool for the job, depending on your use case.

photo credit: jypsygen

There are a few scenarios where I’ve seen systems take advantage of polyglot persistence. The first scenario involves the need to perform some set of complex calculations on an extremely large data set. The data is either copied/moved from the relational database to the NoSQL database, or inserted directly into the NoSQL database by the application. The application can then use a cluster of NoSQL database servers can then divide the work, process the data, and aggregate the results. The more machines you have in your cluster, the less time the processing will take. The resulting data can either remain in the NoSQL database or be inserted into the relational database, depending on what needs to be done with the results.

The other scenario takes advantage of the schema-less nature of some NoSQL databases. While it is certainly possible to store a serialized data structure in a single column of a relational database, interacting with that data can be a bit more challenging than if that data were in a schema-less, document oriented database. This use case, after all, is what the documented oriented databases were designed for. These types of databases simply treat the data as a collection of key/value pairs, identified by a unique ID. The NoSQL databases provide ways in which you can add structure back into the document so the data inside the document can be queried. These databases are great for storing data that can be radically different from document to document, or data whose structure changes constantly.

The challenges of polyglot persistence

Polyglot persistence comes with its own set of challenges. While potentially getting the best of both worlds as far as features go, you get the complexity and hassle of dealing with not only multiple databases, but multiple databases models.

Determining which database to use to store certain data

With more than one database, you now have to decide where to store the data. It’s no longer a given. If you make the wrong decision, you could be looking at a painful migration from one database model to another as a result. To make this decision, you need to carefully examine how the data will be used.

Increased application complexity

Applications also face increased complexity as they now have to interface with two different (potentially very different) data stores. If done correctly, you should be able to isolate this complexity to the persistence layer of your application, freeing the rest of the application from having to know what database specific data is coming from. But, interfacing to multiple data stores could greatly increase the complexity of that data persistence layer. Your application will now need to know:

• How to connect to each of the databases
• What database to use for specific sets of data
• How to handle the different types of errors from each database
• How to map results from each database back to your application’s object model
• How to handle queries for information across databases
• How to mock out the different databases for testing
• Potentially, how to move data from one database to another

Addressing these concerns could result in a bunch of new application code, and with added code usually comes added complexity, and more bugs.

Increased deployment complexity

In addition to the increased application complexity, you will also face increased deployment complexity.

• Will you need to provision new hardware to host the new database?
• How will you backup the data in the new database?
• How will you manage and control changes to the configuration of the new database?

Training for developers and operational staff

Given that this database will likely be radically different from the relational database that your developers and operational staff are comfortable with, how will you bring them up to speed on how to use and manage this new database? And, given that the majority of the NoSQL databases are still very young, how will you keep your developers and operational staff up to speed with the latest developments on the project?

This is a big issue, especially in companies with large development and operations teams, and needs to be thought through carefully.

• Is there an expert you can hire to help you get up and running, and mentor your staff?
• Is there any training available that you can give to your staff?
• Who can you turn to for support when something goes wrong in production?

Summary

I’ve always been a big advocate of using the right tool for the right job. For the past 30 years, the relational database has been the de-facto standard for persistence. Creative people have managed to utilize and manipulate it to serve all sorts of different use cases, quite successfully. But just because you can fit a square peg through a round hole if you hit it with a big enough hammer doesn’t necessarily mean that you should.

NoSQL databases can be great tools for addressing data persistence cases that the relational database struggles with. In addition, each NoSQL database brings its own set of strengths and weaknesses to the table. They are becoming very important tools to have around, and I believe that our industry will see a steady increase in the adoption of these tools going forward.

databases, nosql, polyglot persistence

I got a nice surprise today when upgrading our message processing application from Rails 2.3.3 to Rails 2.3.4, to pull in some important security fixes.

/opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:445:in `initialize_database_middleware': You have a nil object when you didn't expect it! (NoMethodError)
The error occurred while evaluating nil.name
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:182:in `process'
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:113:in `send'
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:113:in `run'
....


Jumping into the Rails source, I found the offending line.

if configuration.frameworks.include?(:action_controller) &&
  ActionController::Base.session_store.name == 'ActiveRecord::SessionStore'

This code assumes that a session store is configured in your Rails app. However, this particular application has no need for a session, so we were disabling it in the configuration by specifying the following:

config.action_controller.session_store = nil

I poked around on the web for a while, trying to find another way to disable the session. No luck. It appeared that the only other way to disable the session was to properly configure a session store in your environment.rb file, and then disable it in your ApplicationController. That seemed lame. Why should I have to configure something that I want to disable?

So, I coded up a simple class to act as the session store for the application that simply raises an error if anybody tries to access the session.

class NilSessionStore < ActionController::Session::AbstractStore
  def get_session(env, sid)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end

  def set_session(env, sid, session_data)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end
end

I then configured the application to use this class as the session store.

config.action_controller.session_store = :nil_session_store

Nice and simple, and it keeps me from having to configure something I never plan to use.

rails

This is the 6th and final post in a series that describes our investigation into CouchDB as a solution to several database related performance issues facing the TextMe application.

<< Part 5: Application Changes

Addressing the remaining issues

We were almost there. After modifying the code to talk to CouchDB, TextMe was successfully pulling data from CouchDB in our development environments. There were just a few remaining issues that needed to be addressed before we could deploy CouchDB to production.

Reducing the view sizes on disk

As I mentioned in a previous post, the amount of disk space consumed by the views was a big problem. If we didn’t do something, we were sure to run out of disk space when migrating our 30 million row messages table to CouchDB.

We determined that it was not what we were emitting from our map functions that was killing us, but how many times we were emitting it. Each of the views emitted a key/value pair for every document in the database. At 30 million documents and 8 views, that ends up being a crap load of key/value pairs.

My colleagues Dave and Jerry took a detailed look at the problem, and came up with a solution. They determined that there was simply no need to be emitting data for each document in the database. While this would give us views that could report statistics by the second, our application only supported presenting statistics by the minute. Even if we were able to support statistics at this level of detail, we doubted our customers would even need it. It was simply not worth the disk space.

So, Dave and Jerry modified the import job described in the previous post to roll up several key statistics by the minute as it was building the documents. When the job finishes processing all of the documents for that minute, it creates a summary document containing all of the rolled up statistics, and adds it to the database. Then, they changed the map functions to only consider these summary documents.

This solution was able to dramatically reduce the sizes of the views on disk, while still supporting the current application functionality. Since we are still persisting all of the original documents to CouchDB, it is possible to add a new statistic to the summary documents should we ever need to.

Oh, and we also picked up two new terabyte database servers, just in case :)

Paginating records in CouchDB

Like many Rails applications, we were using the popular will_paginate gem to paginate results from the database. Given the size of our data sets, pagination was an absolute necessity to keep from using up every last bit of memory.

CouchRest has a Pager class that paginates over view results, but it is in the CouchRest Core part of the library and doesn’t integrate too well with the object model part of the library. It simply returns the view results as an array of hashes. We were hoping to see a solution that would give us back an array of the corresponding ExtendedDocument objects. We were also trying to keep our application from having to know about CouchDB outside of the classes described in the previous post. Having completely different pagination strategies for the two databases would make that more difficult.

So, I decided to write some new pagination code that supported the will_paginate interface and integrated a little better with the object model part of CouchRest. I had a quick solution that same day which fetched view results and handed back an array of the corresponding ExtendedDocument objects. I then spent some time over the next two weeks modifying the code to integrate a little better with CouchRest and add support for CouchRest views, which we weren’t using.

With the new code in place, we can now paginate over a set of contest entries without having to know what database they are coming from.

ContestCampaignEntryDelegate.contest_campaign_entries.paginate(
  :page => 1, :per_page => 50)

This pagination code eventually made it into CouchRest.

Going live

With the remaining issues addressed, it was time to start the production migration. One at a time, we manually started the jobs to move the data from MySQL to CouchDB. When one job completed, we would start the next. As I mentioned before, building the views is very resource intensive. We didn’t want to completely bog down the production machine we were using to do the migration by running multiple jobs at once.

Moving the archived data from MySQL to CouchDB and building all of the views took about a week (a day for this table, a couple of days for that table, etc). Overall, it was a fairly smooth process.

For the initial import, we did not purge any of the data from MySQL. Since we needed to wait until our CouchDB databases were fully populated with all views built before we could start using them, the application needed to continue working with the data in MySQL while the migration was in progress. In anticipation of the eventual switch from MySQL to CouchDB, I added a flag in the application configuration that told the application if it should pull archived data from CouchDB. Once all of the data had been imported and all of the views had been built, we flipped the switch.

With the pouring of a celebratory beer, we watched as our application began pulling data from CouchDB in production. It was time to relax :)

The results

I really wish we had taken the time to record how long our “troublesome” pages were taking to load before the move to CouchDB. Sadly, we did not. All I can say is that pages that used to occasionally time out were now loading in a few seconds. Since the migration, we have also implemented a few new features that would simply not have been possible without CouchDB due to database performance issues.

The database performance issues we set out to address seem to be a thing of the past. If new ones pop up, I’m confident that we could once again utilize CouchDB to address them.

What’s next

This project was focused on addressing database related performance issues that we were facing in production. With these issues out of the way, and our CouchDB infrastructure built-out and proven, we will soon be building even more reporting capabilities that would have simply killed our old database. TextMe customers will soon be able to view their data in more ways than they could have imagined.

I am also working on a project that takes advantage of CouchDB’s schema-less nature to let our customers store and utilize data they collect from their customers. Such a feature, which essentially lets customers define their own schema, would have been a challenge to implement in a relational database. With CouchDB, its just a document.

Thoughts about this project, and CouchDB

I learned a ton while working on this project. While vaguely familiar with “NoSQL” databases before this project, I have just recently become aware of all of the alternatives available. With the enormous amount of data companies are beginning to collect and process, I’m sure that CouchDB and its NoSQL friends will soon become a common component in the operational environments of most companies.

The CouchDB community has been great. The CouchDB and CouchRest mailing lists are extremely active, and have been very helpful. The committers on both of these projects are active, and always eager to help. I’d specifically like to call out Jan Lehnardt and Chris Anderson from the CouchDB project. Jan has commented on a few of these posts, encouraging me to keep writing. He also suggested a more efficient implementation of the CouchRest pagination code I wrote, which I quickly implemented. Chris left a comment on the first post in this series thanking me for writing about CouchDB, and offering his assistance if I needed it. I actually took Chris up on that offer when we were running into issues regarding the sizes of the views on disk. He was quick to reply, offering several suggestions. I’d like to thank Jan and Chris for their support and encouragement.

NoSQL databases are here to stay, and CouchDB is truly unique in this area. The way it handles views, and its support for replication/synchronization set it apart from the others. There are already several large projects, like Ubuntu One, that are relying on CouchDB to deliver what nobody else can. Because of this, I’m sure CouchDB has a very bright future ahead of it.

couchdb, couchdb case study, couchrest, textme

Last night I said goodbye to one of the best friends I’ve ever had.

My wife Beth and I always wanted a dog, and knew that as soon as we got a house, a dog would be soon to follow. On a trip in to see my parents, we stopped by a local animal shelter to take a look at the dogs. We came upon this crate containing 5 puppies, the offspring of what appeared to be a German Shepherd and a Rottweiler. I liked German Shepherds, and Beth like Rotties. We quickly took to the sole black puppy in the litter, and started the paperwork necessary to take her home.

I remember the day I picked her up from the shelter like it was yesterday. I took the day off of work to go and pick her up. The little ball of fur weighed only 10 pounds, and was sleeping soundly in my arms as I filled out the last bits of paperwork. The first thing we did was drive up to see Beth at work. She whined the entire way. We named her Bacardi.

Childless for the first few years of our marriage, Bacardi was our baby. She was a first class member of the family. We did everything with that dog.

A more loyal friend you could not find. Whenever Beth was sick, Bacardi would refuse to leave her side. Those two had a connection like I’ve never seen. It was simply amazing to watch. An eternal puppy at heart, Bacardi was always playful, even in her older years. Affectionate like no other, she’d lick you until your skin came off. Her whip-like tail would start beating against the floor the minute she saw you. Even on my worst days, Bacardi was always able to put me in a good mood. She was always happy to see me, and always put a smile on my face.

When we welcomed our first son Dylan, we were repeatedly warned by friends and family not to let Bacardi too close. I know that many of these feelings stemmed from the fact that she was half Rottweiler, and Rottweilers don’t have the best reputation as a family dog. But Beth and I were confident that we had raised Bacardi to be kind and gentle. Needless to say, Bacardi proved them all wrong. She was both a friend and a guardian to our little boy, like she was for us. I remember one day when Dylan was lying in his bassinet, no more than a week or two old. The bassinet was just a bit taller than Bacardi, and she tried like hell to take a look inside, to see what was making all that noise. She was actually able to tilt the bassinet slightly with the side her head, and look at Dylan with her one eye.

In the following years, she could not have been a better family Dog. She let Dylan do anything to her. He’d ride her like a horse, lie on her side, pull her ears…everything you could imagine, and she just laid there. If she ever had enough, she would simply walk away. Dylan loved Bacardi as much as we did.

Three years later, Beth and I welcomed our daughter Chloe. Chloe was ill quite a bit the first year of her life. We attributed this to Dylan brining home “presents” from pre-school, but started to suspect something else when she was still getting sick over his long breaks from school. We found out that Chloe was an asthmatic, and allergic to Bacardi. We knew we had no choice; we had to find Bacardi a new home. Knowing that this was the right choice didn’t make it any easier. Luckily, Beth’s uncle who lives just 10 minutes from us agreed to take her in. His daughter had always adored Bacardi, and we knew that she would be treated well there. Shortly after agreeing to take her in, Beth’s uncle came to pick her up. That was a terrible day.

In the few short months Bacardi lived with Beth’s uncle, we continued to see her on a regular basis. She was always happy to see us. Even though we knew she was being treated great, a part of us always felt terrible when we left her behind. We wanted so badly to take her home with us, but we knew that we couldn’t. And, it didn’t help to see that she clearly wanted to come with us. Sometimes, it felt like losing her all over again.

Just this past Thursday, Beth’s cousin called to tell us that Bacardi was ill. Beth took her to the vet on Friday, and the vet said it was either a really bad infection or cancer. As Saturday and Sunday passed, her condition quickly got worse, and it became clear that it was not an infection. She was in the late stages of a battle with lymphoma. Early this morning, around 3am, we said goodbye to our friend, and put and end to her suffering.

In Bacardi’s 7 ½ years on this earth, she touched so many lives. We will always cherish the memories of the times we shared with her. She was a great friend, and one amazing dog.

Bacardi, I love you so much, and miss you terribly. Goodbye old friend.

[View with PicLens]

bacardi