Standup Timer version 1.2 has just been released. Per popular demand (2 requests :)), I have added support for meetings of any length, with any number of participants. Prior to this release, Standup Timer restricted you to meetings 5, 10, 15 or 20 minutes long, and a maximum of 20 participants. These restrictions are still enabled by default, but can be disabled via the application’s settings.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

Standup Timer

Now that Standup Timer has a few releases under its belt, I thought it would be a good time to reflect on my experience with the Android SDK, and Android development in general. But before I begin, I should mention a few points about my background, to help you understand my perspective.

First, Standup Timer was my first mobile app. Aside from a very small amount of time I’ve spent playing around with the Blackberry development kit (really too small to even mention), I have had no prior experience developing software for a mobile device. My background is largely web development and distributed applications development. Second, I have been doing Java development for quite some time, so I am very comfortable working in Java. Third, if you’re not familiar with Standup Timer, it is a very simple application. It only interacts with a few components of the Android platform, and only consists of a few screens. No GPS, no web, no multi-touch, etc. So, my tour of the Android SDK was far from complete.

Dealing with multiple devices

Supporting multiple screen sizes

The topic of supporting multiple screen sizes has been a popular one, especially when comparing Android development to iPhone development. Supporting multiple screen sizes in your application does add a little complexity to the development process, but for my app (and I’d imagine most apps), it is very manageable.

This issue did not take Google by surprise. They knew that if Android was to be successful, it would need support devices with a wide range of physical characteristics, including screen size. So, from the beginning, they made it possible for applications to support multiple screen sizes with little effort. The Android platform contains many features to help with this, including the ability to pre-scale images for phones with different resolutions, allowing you to specify your screen component sizes in density independent pixels (dips), and allowing you to easily center and stretch your components to fill the screen. These features, along with a set of best practices for supporting multiple screen sizes, are documented very well at http://developer.android.com/guide/practices/screens_support.html.

For applications that do not use the standard Android view components, or do custom graphics, it may be a different story. I have read blog posts from some Android game developers saying it is a larger issue, and some who say it is not. So, I’m not sure. However, for the vast majority of applications, the standard view components work just fine, making supporting multiple screen sizes a very manageable issue.

One thing that did bite me regarding multiple screen sizes is the fact that views will not scroll by default if they happen to flow off the screen. You need to anticipate this, and wrap any views that may flow off the screen with a ScrollView to enable scrolling behavior.

Supporting multiple devices

To me, this issue is much larger than the multiple screen size issue. Android is an open source project, which gives anybody the ability to modify the code however they wish. Google prevents carriers and cell phone manufacturers from abusing this by holding back a few key applications from the open source release, including the Android Market application. Without the Android Market application, phones would not have access to the 20,000 apps currently available for the Android platform.

However, phone manufacturers do tweak the platform to work for their specific hardware. Android 2.1 on one device is not necessarily the same as Android 2.1 on another device. For an example of this, compare 2.1 on the Motorola DROID with 2.1 on the Nexus One. These seemingly minor modifications can be very troublesome for application developers. The web has been flooded with reports of developers growing frustrated with the complexity matrix of Android versions and the exploding number of phones running them, all potentially containing their own tweaks to the platform. Browsing through the change logs of applications I have installed on my Motorola DROID, I can see that several apps have made changes to fix issues on specific devices.

This issue is forcing developers to not only test on every device they plan on supporting, but also to write device specific code to work around any known issues for a device. As Android continues to grow, this level of support will be unsustainable. Developing for a common platform should mean that your application will run fine on any device running that platform. But, because of these device specific tweaks, this is quickly turning out not to be the case for Android. I worry that if Google doesn’t find some way to control this, developers will continue to abandon the platform.

I have even run into what appear to be device related issues with Standup Timer, which is miniscule compared to the size and complexity some of the other apps available in the Android Market. Standup Timer uses an Android API that prevents the user’s screen from blacking out while a timer is in progress, letting you always see how much time is remaining for a meeting. Just the other day somebody left a comment in the Market indicating that this was not working on their device. So far, all of the other reviews have been positive, leading me to believe that this could be an issue with that user’s particular device. The commentator didn’t provide any information about the device they were using, or how they could be contacted. So, I’m not sure if I’ll ever be able to track down this issue. Even if I knew the device experiencing the issue, I’m not sure I would be able to help. Unless you have access the device in question, reproducing these issues is practically impossible. The Android simulator tool is great, but you cannot create a simulator for a specific Android device, running that device’s flavor of Android. You can only create simulators running the “generic” version of an Android release.

Development

Application life cycle

The Android platform primarily communicates with an application through a series of life cycle events. The platform will let the application know when an activity (a screen) has been created, paused (lost focus), resumed (regained focus), etc. The life cycle events are easy to understand, and are specific enough so that I never need to examine the state of my application or the platform within one of these callbacks. The fact that a specific life cycle method was called tells you what state your application is in.

My largest complaint about how the platform manages an application is how it destroys and re-creates the activity (the object that backs the screen) when the phone rotates from portrait mode to landscape mode, or visa versa. It seems to me that this would best be handled by making the appropriate life cycle callbacks to the same activity instance, giving it a chance to redraw the screen. Instead, you are forced to save all of your activity’s state, and then reload it when the new activity instance takes over. Forgetting to save the state of a particular variable means that value will be lost when the phone is rotated. This was the source of several bugs when writing Standup Timer.

Android APIs

I have no major issues with the Android APIs. They seem complete (for what I needed), and most importantly, behaved as expected. The database API seems a bit archaic, especially in today’s age of sophisticated O/R mapping tools. But, at least I didn’t have to catch SQLException after each operation, like the JDBC API.

However, the lack of decent documentation for some of the APIs is a problem. There were a few instances where I was forced into using trial and error to determine the purpose of a method parameter. I feel this level of unclarity is unacceptable for a public API.

UI Creation

The Android SDK offers developers two methods for creating UI screens: programatically using Java code or declaratively using XML. I did not attempt the programatic approach, as it brought back too many nightmares of building UIs for Java applets. The XML approach to creating screen layouts, which is the recommended approach, is very straight forward. Java developers, especially those working in the “enterprise”, have grown to hate XML over the past few years, due to its verbosity and proliferation. But, I think Android’s use of it is very tasteful. At no point did I consider myself to be in XML-hell. Looking back on the final XML that declares my user interface, it doesn’t seem overly verbose or complicated.

I did however struggle at times to figure out how to get the UI to look exactly the way I wanted it. Again, this seems to be the result of poor documentation of the XML elements and their corresponding attributes. Some more complex examples in the sample code included in the SDK would have also helped.

If you keep your screens simple, and don’t display too much information on any given screen, then you shouldn’t have much trouble dealing with how that screen looks in portrait or landscape mode. Screens with a list of items, for example, will usually look just fine when viewed in either portrait or landscape mode. When in landscape mode, the user will just see a little more blank space in the list. For more complicated screens, the Android platform allows you to specify an alternate UI layout for landscape mode. This allows you to completely restructure the elements on a screen to deal with the the wider, shorter screen size without affecting how the screen looks in portrait mode. Given how I struggled to get the views to look exactly the way I wanted, I tried my best to avoid the need to specify an alternate layout file for landscape views. Only two of the screens in Standup Timer actually needed a landscape specific layout file.

adb

The Android Debug Bridge (adb) is a nifty little tool that allows you to interact with an Android device or simulator from the command line. You can install/uninstall applications, interact with files on the device, or run a shell on the device. It also provides a series of commands that allow you to more easily create scripts to run on the device. Not only is this tool useful by itself, but it also enables the creation of sophisticated tools for Android development. Learning how to use adb is a wise investment of your time.

The Eclipse plugin

The Eclipse plugin for Android development is great. It provides tight integration between Eclipse and the Android SDK. Code completion is available for the Android APIs as well as the different types of XML files used by the Android platform. It has great support for Android resource files. You can easily start your application on a simulator with the click of a button, and running your tests is a breeze. I would not attempt to write an Android application without this plugin, or a similar plugin for a different IDE.

However, I did run into a few issues regarding how Eclipse interacts with the Android simulators. On more than a few occasions, when starting the application or running the tests, the Eclipse plugin started a simulator running a version of the Android platform that did not match my project settings (although, I have not completely ruled out user error here :) ). Also, the Eclipse plugin seems to have a few bugs around interacting with multiple simulators. adb allows you to specify the target device or simulator when issuing a command, which is useful when you have more than one simulator running at a time (which is almost always the case given the current state of the Android platform). However, the Eclipse plugin doesn’t always utilize this capability. Sometimes it will detect multiple simulators running, and ask you which one you’d like to target. Other times it will not. In addition to the simulator interaction issues, there are a few areas that could benefit from a bit of polish. The UI builders, for example, are difficult to work with. It became obvious very quickly that I would be better hand coding the XML for the UI than attempting to use the builders.

Testing

Simulators are awesome

The Android simulators are awesome. Once you have the proper versions of the SDK installed, it is trivial to create simulators with different screen sizes, different amounts of available storage, running different versions of the Android platform. And, as previously mentioned, adb is great for interacting with the simulators. While not perfect (you should always test your app on a real device before publishing it), they are pretty darn close.

The only thing missing, and I’m not sure how feasible this is, is the ability to create a simulator for a specific device. For example, if I know that my app is crashing on the Nexus One for some reason, I’d love to be able to create a Nexus One simulator, so I can find and fix the issue. Currently, debugging and fixing device specific issues is not possible without access to the device in question. I know some large mobile shops purchase the devices they plan on supporting, for testing purposes. And, services like Device Anywhere allow you virtual access to a physical device, which would satisfy this need. But these options, especially the first one, are expensive. A developer trying to fix a device specific issue in their free, non ad subsidized application will not have the resources for either of these options.

Automated (unit and functional) Testing

Automated testing on the Android platform is PAINFUL. It’s hard to decide where to even begin. First off, the tests provided with the sample applications in the SDK are very bare bones. Most of them simply assert that an activity has been created, and that’s it. If there was sufficient documentation describing how to write tests for you application, this wouldn’t be that big of a deal. But, there isn’t. Luckily, guys like Diego Torres Milano are writing blog posts and giving presentations in an attempt to fill this gaping hole in documentation. The slides from Diego’s presentation at Droidcon 2009 are a great place to start. Standup Timer, which is open source, also has a complete set of unit and functional tests for you to check out if you so desire.

Another pain point regarding automated tests for Android applications is that the tests cannot be run inside a standard Java Virtual Machine. The implementation of the APIs in the android.jar file provided with the SDK simply throw exceptions. That jar file is only meant for building your application, not running it. So, all tests must be run on the device (or in a simulator). This dramatically slows down the execution of the tests. One alternative proposed by a non-Android group inside of Google is to mock out all of the Android components using a mocking library, enabling the execution of the tests in a standard virtual machine. Although this would speed up the test execution, it comes with its own drawbacks. Mainly, you end up having to mock out a TON of stuff. I think that mocking out too many components significantly reduces the value of your tests. If you mock out everything, what are you really testing?

The SDK contains a few base classes that you can extend for certain types of tests. In particular, the SDK provides support for functional tests (the testing of a particular activity, or screen) and unit tests (the testing of some underlying support code). Writing unit tests is pretty straight forward. With a little bit of investigation and some trial and error, I was able to create a set of tests for my database layer, which will execute against a test version of the application’s database.

Functional tests were much more problematic. In addition to executing much slower than unit tests, they are also more difficult to write. For functional tests, you write code to perform the operations a user would be performing on their phone (entering text, pressing buttons, etc). The problem is that it is not very straight forward to perform these operations via code. It took me quite some time to figure out how to simply enter some text in a text area and press a button via code. Even when I thought I had functional testing figured out, there was still a use case I was unable to write a test for. Most of the issues revolve around gaining access to the components you wish to interact with. For a simple text box or button, it is very easy. You can simply fetch the view by its id, the same way you do in your application. However, once you start nesting components, it becomes much more complicated. I tried several different ways to invoke the context menu (the long press menu) of a list item, inside of a list, which is inside of a tab. None of them worked.

Instead of writing functional tests for activities, I found it much easier to test activities using the ActivityUnitTestCase class with a mocked out instance of the activity. Test classes that extend ActivityUnitTestCase don’t have as much as access to the platform as functional tests. Several API methods will throw exceptions if called from within a ActivityUnitTestCase, and other API methods simply do nothing. However, this can be addressed with some simple tools that should be in every developers toolbox: mocking and stubbing. It is trivial to contain a method call that would throw an exception during testing within a method of its own. This method can then be overridden by a mock implementation of the activity. The mock can then optionally set a flag when the method is called, so the test can verify that some action was performed. One place I do this is where the activity under test tries to start another activity. Calling startActivity from within a ActivityUnitTestCase does nothing. So, I placed the call to startActivity by itself in a protected method, overrode that method in the mock activity class, set a flag in the mock method implementation, and checked to see if that flag was set in the test case to verify that the activity was started.

Deployment

Publishing to the Android Market

Publishing an application to the Android market could not be any easier. The process is very well documented on the Android developers website, and the Eclipse plugin makes creating and signing an installation package a point and click operation.

A fee of $25 is required to create a Android Developers account. This account gives you the ability to upload your application to the Android Market using the Android Developers website. Once published, the application can immediately be found in the Android Market. There is no approval process. Updating your application is just as easy. Simply upload the new version of your application, and it is available in the Android Market immediately.

It should be noted that there is no fee to install the application on your phone.

Summary

Android is a great platform. It is feature rich, and developer friendly. The devices that run it are powerful and easy to use. I also love the fact that it is built on open source technology, and that Google, for the most part, has continued this tradition of openness.

But, the platform is not without problems. Most of my complaints here are minor, and are very addressable. Android is still a young platform, so there is plenty of room to improve. What worries me most about the future of the Android platform is the complexity matrix I talk about in the Supporting Multiple Devices section.

Android was created to be a common platform for mobile application development. A platform where application developers could write a single application capable of running on numerous devices. However, this dream is at risk of becoming just that, a dream. Unless Google can find a way to minimize the impact of device specific issues, it risks losing more and more Android developers. Last year it was alraedy known that 2010 was going to be a big year for Android. With several new devices slated to hit the market, industry analysts are expecting its market share to jump considerably. However, if each new device that hits the market brings with it its own set of device specific quirks, this could also cause the Android development community to quickly erode. Applications are a very large part of the mobile experience today, and they will continue to be in the future. But, without a development community, there will be no applications.

I really hope Google finds a way to address this very serious issue.

android, Standup Timer

Version 1.1 of Standup Timer has just been released. This new version stores statistics for your stand-up meetings on a per-team basis. Simply create one or more teams using the new Teams menu option, then specify the team holding the meeting when starting the timer. To view statistics for a given team, use the Teams menu option to pull up the list of teams, and select the team you would like to see statistics for. The Stats tab will display the average statistics for that team across all of its meetings. For a list of the meetings that team has held, click on the Meetings tab. Finally, to see statistics for a specific meeting, simply select the meeting from the list of meetings.

Standup Timer is free and open source. The source code can be found at http://github.com/jwood/standup-timer. Standup Timer can be found in the Android Market.

[View with PicLens]

android, Standup Timer

A couple of years ago I bought a new printer for the house; a nice 3-in-one printer/fax/copier. I always research the electronics I purchase, and this particular model had received many very positive reviews. So, you can bet I was surprised, and a little pissed off, when the printer started acting up just a few short months after I purchased it. I tried like hell to figure out the issue, hoping my efforts would save me from a dreaded call to the manufacturer’s customer support hot line. No such luck. After a few hours of troubleshooting, I looked up the support number in the back of the users guide, dialed it, and prepared for the worst.

We’ve all been there. Unfamiliar with the product, we are often unable to explain in detail what is going on, especially over the phone. And usually on the other end of the phone is an under paid customer support representative who simply reads from a script given to him by his manager. Often these representatives know very little about the product they are supporting. And, good luck if you happen to have a question that doesn’t appear on the script.

What happened next I will never forget. On the other end of the line appeared this friendly, knowledgeable technician. I described the problem I was having to him. He listened patiently, and asked a couple of targeted questions to help narrow down the issue. In no time at all, he knew exactly what was going on, and exactly how to fix it. Step by step, he clearly instructed me on what needed to be done to get my printer back online. Before you knew it, I was back in business. That technician, that single person, turned my experience around 180 degrees. I was once again a satisfied customer. Hell, I was more than satisfied.

You cannot underestimate the value of awesome customer support. I will gladly pay more for a product or service if I know that it will come with great support, as I know it will save me from headaches and grief down the road.

Typically, developers don’t serve on the front lines of customer support. However, not directly interacting with and supporting the users of your product means you are missing out on a great opportunity. An opportunity to build customer loyalty, an opportunity to understand how a customer is using your product or service, or simply an opportunity to help somebody.

I think developers should spend at least a few weeks a year directly supporting the users of their product. There are several reasons why.

You will learn how people use your product

You think you know how people are using your product? Think again. You think that user interface you designed makes perfect sense? Maybe it does to you and your co-workers, but odds are it is not as clear to your customers. When on customer support, you’ll see first hand how customers are using your product. If you find that your customers are fumbling around your product, thinking they are doing one thing but actually doing another, it’s a clear indication that your product isn’t as intuitive as it should be. Don’t write off these issues as “user errors”. Instead, get some feedback from your customer, and figure out how to make your product easier to use.

It will become obvious what features are missing, what needs to be improved, or what needs to be taken out

Your sales team, marketing team, and product team may have some great ideas regarding what direction to take your product. But, there is nothing quite like hearing it straight from the customer. This doesn’t mean that you should run out and implement every feature, and make every change requested by a customer. But, if you start to hear similar feedback while on support, it may be an indication of a legitimate need. Take note of it! Or, better yet, just do it!

Developers can fix problems…fast

Developers are in a great position to fix problems, fast. They usually know the product more intimately than anybody else on the team, have easy access to the code, and sometimes have the ability to release a patch. This gives developers the unique ability to provide amazing customer support. There have been times where I’ve fixed an issue while still on the phone with a customer, or shortly thereafter. Think about how you would feel if all of your problems were solved this quickly.

Summary

Happy customers become loyal customers. And, providing awesome customer support is one sure way to keep your customers happy. Providing customer support can be very time consuming, sometimes consuming the majority of a developer’s time (which is why I think a few weeks a year is enough). However, this should be thought of not as an expense, but an investment. An investment in the customer and the developer.

support

Putting my brand new Motorola DROID to work, I just completed my first Android application, which I’m calling Standup Timer.

Standup Timer is a very simple application that helps keep your stand-up meetings focused, and on track. I’ve been participating in stand-up meetings of one form or another for a few years now. One thing that is constantly an issue with stand-up meetings is that they often run longer than they are supposed to. Instead of simply stating their status and moving on, participants will often start to ramble, or engage in conversations with other members of the team. Those who practice any form of the Agile software development methodology know that the value of stand-up meetings drops considerably if the meeting is constantly allowed to overflow its time constraints.

So, I created an app for that :) Standup Timer helps to keep your stand-up meetings on track by allotting each participant an equal share of time, and letting participants know when they are about to exceed, or have exceeded their time. The notification comes in the form of a bell ring for the warning, and an air horn when the time has run out. If any time is left after all participants have presented their status, then Standup Timer will keep the clock ticking, so the remaining meeting time can be used for an open discussion.

Standup Timer is easy to use. Simply provide the number of participants in the meeting, the length of the meeting, and press Start to start the timer. On the timer screen, press Next when a participant is finished presenting their status to reset the timer for the next participant. When the meeting is over, press Finished. Standup Timer will remember the number of participants and meeting length from the last time you used it. It also allows you to enable/disable sounds, and to specify when (how many seconds left in their share of time) to warn participants that they are about to exceed their share of time.

Standup Timer is free and open source. You can find the application in the Android Market, and the source code on GitHub at http://github.com/jwood/standup-timer.

[View with PicLens]

android, Standup Timer

The days of the relational database being a one-stop-shop for all of your persistence needs are over. A new class of application is beginning to emerge with requirements that exceed the capabilities of the relational database. Some of these applications need unlimited scalability or bullet proof fault tolerance, while others may require blazing fast access or flexible data storage. The relational database was simply not designed to meet the needs of this small but growing class. Instead, a new breed of data stores are gaining momentum. These data stores are looking at data persistence with a fresh set of eyes, diverging from the relational model considerably in order to meet these challenges.

What’s wrong with the relational database?

For 99% of the applications out there, absolutely nothing. The relational database has been the industry standard for data storage over the past 30+ years for good reason. It is an incredibly capable piece of software. Although it may not be the best tool for everything it is used for, it certainly satisfies the needs of the vast majority of applications just fine.

However, while not new, the class of applications mentioned above are becoming more common. These applications either handle enormous amounts of traffic, or deal with tremendous amounts of data. The relational database falls short in a few areas when trying to meet the demands of an application like this.

photo credit: bioxid

A single database server is usually not enough to support these requirements. Applications like this need a true database cluster, capable of adding storage space and processing power on the fly without the application even noticing. However, relational databases weren’t designed to operate in a cluster where all machines are capable of reading and writing data. This is largely due to the promises they make regarding data integrity. In order to fulfill these promises, the database needs easy, quick access to all of the data at all times to verify that duplicates aren’t being inserted, constraints aren’t being violated, etc. This quickly becomes a bottleneck when dealing with very large amounts of data.

There are techniques for scaling out relational databases, but they don’t address every concern. One popular technique is to use one or more slave databases for read requests, while continuing to funnel all write requests through the master database. The master database constantly synchronizes with the read only databases, so the data remains consistent between databases. This technique works great for read heavy applications, but does not help applications that perform just as many creates, updates, and deletes. Data sharding is another popular technique, which involves splitting the data up onto several different databases based on some criteria. But this pushes an extraordinary amount of complexity onto the application, as it is now responsible for determining which database to use for specific data sets. Master-master replication can be used to keep multiple master databases in sync, so any database server can perform read or write operations. However, for some applications there comes a point where the replication can’t keep up with the traffic.

Relational databases are also (intentionally) very strict when it comes to the structure of the data being stored. Data must be broken up into a series of rows and columns. Good object/relation mapping tools hide much of this awkwardness from us, but some applications deal with data that doesn’t map well into rows and columns. A simple key/value store is usually a better fit for applications like this.

How does the new breed address these problems?

The new breed of data stores, called NoSQL databases, make very few promises regarding data integrity. In this new model, data integrity becomes the application’s concern. By not having to enforce any complex data integrity rules, NoSQL databases can scale to levels way beyond that of a relational database. Adding more processing power or storage capacity can be as simple as adding a new machine to the cluster. The database can then store and process the data using any machine in the cluster.

In this model, the data being stored is self contained, and does not rely on any other data in the database. Therefore there is no need for one machine to know anything about the other machines in the cluster. This approach is quite different from the relational model, where data is broken up into multiple tables to eliminate duplicate data, and joined back together when being accessed.

Most of these databases subscribe to a theory called eventual consistency. In situations where duplicate information is scattered across different servers in the cluster, it is not feasible for the database to find all instances of that data and update it as a part of the original operation. Instead, the data will be replicated to the other database servers at a later time. Until that replication takes place, the application will be in an inconsistent state, where simultaneous queries fetching the same data could return different results. Although this sounds terrible, it turns out that in practice it is really not too big of a deal for most applications. Do all customers of an online retailer need to see the exact same set of product reviews 100% of the time? Probably not.

Also, because there are few promises regarding data integrity, NoSQL databases can offer data storage that is much more flexible. The database no longer has to enforce the uniqueness of a column, or ensure that the id of some referenced piece of data actually exists in the database. Some of these databases are true key/value data stores, where you can store just about anything. Others require a certain document format to be used (such as JSON or XML), but still allow you to freely change the contents of that document as you wish.

Still no one-stop-shop for persistence

Although NoSQL databases address some issues that can’t be addressed by relational databases, the opposite is true as well. The relational database offers an unparalleled feature set. While some of these features prevent it from serving the needs of the class of applications described above, they are absolutely required by other classes of applications. In some domains, data integrity is the number one concern. You need to look no further than the classic “try to withdraw money from the same account at the same time” example to justify the need for locks and transactions.

For the vast majority of applications out there, relational databases work great. There are a boat load of tools and libraries that support them, and software developers are very familiar with how to use them. It is safe to say that the relational database has secured its spot in IT departments and data centers around the world, and it isn’t going anywhere. It is far from dead.

Polyglot persistence

An increasing amount of case studies are appearing that describe how real world applications are needing the data integrity offered by the relational database in addition to the benefits offered by NoSQL databases. I believe this trend will continue, as companies are storing more data than ever, and processing that data in different ways than previously imagined.

To address these needs, some companies are beginning to run their relational database side-by-side with one or more of the NoSQL alternatives. Extremely large data sets that require scalable storage space and processing power are moved to a NoSQL database, while everything else, especially data that needs its integrity kept in-check, remains in the relational database. The term Polyglot Persistence has been used to describe the use of multiple databases within the same project.

The benefits of polyglot persistence

The benefits are somewhat obvious. By running a relational database side-by-side with a NoSQL database, you get the best of both worlds. Strict enforcement of data integrity from the relational database, and the scalability and flexibility provided by the NoSQL database. This allows you to use the best tool for the job, depending on your use case.

photo credit: jypsygen

There are a few scenarios where I’ve seen systems take advantage of polyglot persistence. The first scenario involves the need to perform some set of complex calculations on an extremely large data set. The data is either copied/moved from the relational database to the NoSQL database, or inserted directly into the NoSQL database by the application. The application can then use a cluster of NoSQL database servers can then divide the work, process the data, and aggregate the results. The more machines you have in your cluster, the less time the processing will take. The resulting data can either remain in the NoSQL database or be inserted into the relational database, depending on what needs to be done with the results.

The other scenario takes advantage of the schema-less nature of some NoSQL databases. While it is certainly possible to store a serialized data structure in a single column of a relational database, interacting with that data can be a bit more challenging than if that data were in a schema-less, document oriented database. This use case, after all, is what the documented oriented databases were designed for. These types of databases simply treat the data as a collection of key/value pairs, identified by a unique ID. The NoSQL databases provide ways in which you can add structure back into the document so the data inside the document can be queried. These databases are great for storing data that can be radically different from document to document, or data whose structure changes constantly.

The challenges of polyglot persistence

Polyglot persistence comes with its own set of challenges. While potentially getting the best of both worlds as far as features go, you get the complexity and hassle of dealing with not only multiple databases, but multiple databases models.

Determining which database to use to store certain data

With more than one database, you now have to decide where to store the data. It’s no longer a given. If you make the wrong decision, you could be looking at a painful migration from one database model to another as a result. To make this decision, you need to carefully examine how the data will be used.

Increased application complexity

Applications also face increased complexity as they now have to interface with two different (potentially very different) data stores. If done correctly, you should be able to isolate this complexity to the persistence layer of your application, freeing the rest of the application from having to know what database specific data is coming from. But, interfacing to multiple data stores could greatly increase the complexity of that data persistence layer. Your application will now need to know:

• How to connect to each of the databases
• What database to use for specific sets of data
• How to handle the different types of errors from each database
• How to map results from each database back to your application’s object model
• How to handle queries for information across databases
• How to mock out the different databases for testing
• Potentially, how to move data from one database to another

Addressing these concerns could result in a bunch of new application code, and with added code usually comes added complexity, and more bugs.

Increased deployment complexity

In addition to the increased application complexity, you will also face increased deployment complexity.

• Will you need to provision new hardware to host the new database?
• How will you backup the data in the new database?
• How will you manage and control changes to the configuration of the new database?

Training for developers and operational staff

Given that this database will likely be radically different from the relational database that your developers and operational staff are comfortable with, how will you bring them up to speed on how to use and manage this new database? And, given that the majority of the NoSQL databases are still very young, how will you keep your developers and operational staff up to speed with the latest developments on the project?

This is a big issue, especially in companies with large development and operations teams, and needs to be thought through carefully.

• Is there an expert you can hire to help you get up and running, and mentor your staff?
• Is there any training available that you can give to your staff?
• Who can you turn to for support when something goes wrong in production?

Summary

I’ve always been a big advocate of using the right tool for the right job. For the past 30 years, the relational database has been the de-facto standard for persistence. Creative people have managed to utilize and manipulate it to serve all sorts of different use cases, quite successfully. But just because you can fit a square peg through a round hole if you hit it with a big enough hammer doesn’t necessarily mean that you should.

NoSQL databases can be great tools for addressing data persistence cases that the relational database struggles with. In addition, each NoSQL database brings its own set of strengths and weaknesses to the table. They are becoming very important tools to have around, and I believe that our industry will see a steady increase in the adoption of these tools going forward.

databases, nosql, polyglot persistence

I got a nice surprise today when upgrading our message processing application from Rails 2.3.3 to Rails 2.3.4, to pull in some important security fixes.

/opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:445:in `initialize_database_middleware': You have a nil object when you didn't expect it! (NoMethodError)
The error occurred while evaluating nil.name
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:182:in `process'
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:113:in `send'
from /opt/ruby-enterprise-1.8.6-20090113/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/initializer.rb:113:in `run'
....


Jumping into the Rails source, I found the offending line.

if configuration.frameworks.include?(:action_controller) &&
  ActionController::Base.session_store.name == 'ActiveRecord::SessionStore'

This code assumes that a session store is configured in your Rails app. However, this particular application has no need for a session, so we were disabling it in the configuration by specifying the following:

config.action_controller.session_store = nil

I poked around on the web for a while, trying to find another way to disable the session. No luck. It appeared that the only other way to disable the session was to properly configure a session store in your environment.rb file, and then disable it in your ApplicationController. That seemed lame. Why should I have to configure something that I want to disable?

So, I coded up a simple class to act as the session store for the application that simply raises an error if anybody tries to access the session.

class NilSessionStore < ActionController::Session::AbstractStore
  def get_session(env, sid)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end

  def set_session(env, sid, session_data)
    raise NotImplementedError, "NilSessionStore: No session configured"
  end
end

I then configured the application to use this class as the session store.

config.action_controller.session_store = :nil_session_store

Nice and simple, and it keeps me from having to configure something I never plan to use.

rails

This is the 6th and final post in a series that describes our investigation into CouchDB as a solution to several database related performance issues facing the TextMe application.

<< Part 5: Application Changes

Addressing the remaining issues

We were almost there. After modifying the code to talk to CouchDB, TextMe was successfully pulling data from CouchDB in our development environments. There were just a few remaining issues that needed to be addressed before we could deploy CouchDB to production.

Reducing the view sizes on disk

As I mentioned in a previous post, the amount of disk space consumed by the views was a big problem. If we didn’t do something, we were sure to run out of disk space when migrating our 30 million row messages table to CouchDB.

We determined that it was not what we were emitting from our map functions that was killing us, but how many times we were emitting it. Each of the views emitted a key/value pair for every document in the database. At 30 million documents and 8 views, that ends up being a crap load of key/value pairs.

My colleagues Dave and Jerry took a detailed look at the problem, and came up with a solution. They determined that there was simply no need to be emitting data for each document in the database. While this would give us views that could report statistics by the second, our application only supported presenting statistics by the minute. Even if we were able to support statistics at this level of detail, we doubted our customers would even need it. It was simply not worth the disk space.

So, Dave and Jerry modified the import job described in the previous post to roll up several key statistics by the minute as it was building the documents. When the job finishes processing all of the documents for that minute, it creates a summary document containing all of the rolled up statistics, and adds it to the database. Then, they changed the map functions to only consider these summary documents.

This solution was able to dramatically reduce the sizes of the views on disk, while still supporting the current application functionality. Since we are still persisting all of the original documents to CouchDB, it is possible to add a new statistic to the summary documents should we ever need to.

Oh, and we also picked up two new terabyte database servers, just in case :)

Paginating records in CouchDB

Like many Rails applications, we were using the popular will_paginate gem to paginate results from the database. Given the size of our data sets, pagination was an absolute necessity to keep from using up every last bit of memory.

CouchRest has a Pager class that paginates over view results, but it is in the CouchRest Core part of the library and doesn’t integrate too well with the object model part of the library. It simply returns the view results as an array of hashes. We were hoping to see a solution that would give us back an array of the corresponding ExtendedDocument objects. We were also trying to keep our application from having to know about CouchDB outside of the classes described in the previous post. Having completely different pagination strategies for the two databases would make that more difficult.

So, I decided to write some new pagination code that supported the will_paginate interface and integrated a little better with the object model part of CouchRest. I had a quick solution that same day which fetched view results and handed back an array of the corresponding ExtendedDocument objects. I then spent some time over the next two weeks modifying the code to integrate a little better with CouchRest and add support for CouchRest views, which we weren’t using.

With the new code in place, we can now paginate over a set of contest entries without having to know what database they are coming from.

ContestCampaignEntryDelegate.contest_campaign_entries.paginate(
  :page => 1, :per_page => 50)

This pagination code eventually made it into CouchRest.

Going live

With the remaining issues addressed, it was time to start the production migration. One at a time, we manually started the jobs to move the data from MySQL to CouchDB. When one job completed, we would start the next. As I mentioned before, building the views is very resource intensive. We didn’t want to completely bog down the production machine we were using to do the migration by running multiple jobs at once.

Moving the archived data from MySQL to CouchDB and building all of the views took about a week (a day for this table, a couple of days for that table, etc). Overall, it was a fairly smooth process.

For the initial import, we did not purge any of the data from MySQL. Since we needed to wait until our CouchDB databases were fully populated with all views built before we could start using them, the application needed to continue working with the data in MySQL while the migration was in progress. In anticipation of the eventual switch from MySQL to CouchDB, I added a flag in the application configuration that told the application if it should pull archived data from CouchDB. Once all of the data had been imported and all of the views had been built, we flipped the switch.

With the pouring of a celebratory beer, we watched as our application began pulling data from CouchDB in production. It was time to relax :)

The results

I really wish we had taken the time to record how long our “troublesome” pages were taking to load before the move to CouchDB. Sadly, we did not. All I can say is that pages that used to occasionally time out were now loading in a few seconds. Since the migration, we have also implemented a few new features that would simply not have been possible without CouchDB due to database performance issues.

The database performance issues we set out to address seem to be a thing of the past. If new ones pop up, I’m confident that we could once again utilize CouchDB to address them.

What’s next

This project was focused on addressing database related performance issues that we were facing in production. With these issues out of the way, and our CouchDB infrastructure built-out and proven, we will soon be building even more reporting capabilities that would have simply killed our old database. TextMe customers will soon be able to view their data in more ways than they could have imagined.

I am also working on a project that takes advantage of CouchDB’s schema-less nature to let our customers store and utilize data they collect from their customers. Such a feature, which essentially lets customers define their own schema, would have been a challenge to implement in a relational database. With CouchDB, its just a document.

Thoughts about this project, and CouchDB

I learned a ton while working on this project. While vaguely familiar with “NoSQL” databases before this project, I have just recently become aware of all of the alternatives available. With the enormous amount of data companies are beginning to collect and process, I’m sure that CouchDB and its NoSQL friends will soon become a common component in the operational environments of most companies.

The CouchDB community has been great. The CouchDB and CouchRest mailing lists are extremely active, and have been very helpful. The committers on both of these projects are active, and always eager to help. I’d specifically like to call out Jan Lehnardt and Chris Anderson from the CouchDB project. Jan has commented on a few of these posts, encouraging me to keep writing. He also suggested a more efficient implementation of the CouchRest pagination code I wrote, which I quickly implemented. Chris left a comment on the first post in this series thanking me for writing about CouchDB, and offering his assistance if I needed it. I actually took Chris up on that offer when we were running into issues regarding the sizes of the views on disk. He was quick to reply, offering several suggestions. I’d like to thank Jan and Chris for their support and encouragement.

NoSQL databases are here to stay, and CouchDB is truly unique in this area. The way it handles views, and its support for replication/synchronization set it apart from the others. There are already several large projects, like Ubuntu One, that are relying on CouchDB to deliver what nobody else can. Because of this, I’m sure CouchDB has a very bright future ahead of it.

couchdb, couchdb case study, couchrest, textme

This is part 5 in a series of posts that describe our investigation into CouchDB as a solution to several database related performance issues facing the TextMe application.

Compared to the challenges we faced with views, modifying TextMe to interact with CouchDB was very straight forward. This post describes how we changed the TextMe code in order to use CouchDB as an archive database. Since TextMe is a Ruby on Rails application, much of the content in this post references Ruby/Rails specific libraries and frameworks. However, I feel the general concepts could be applied to any development platform.

A quick recap

Before I dive into describing how we modified our application to work with CouchDB, I’d like to quickly recap exactly what we were trying to do (see the first post in this series for a more detailed overview). TextMe is a mobile marking application. You can use it to manage SMS powered voting campaigns, contests, subscription lists, and more. The majority of these campaigns have a pre-determined lifespan. Once the campaign is over, the data collected is primarily used to calculate statistics on the campaign. This data is very important to our customers.

A few of our database tables were getting quite large, and starting to affect the performance of the application. Tuning the queries didn’t seem to help much. So, we turned to CouchDB and its views to help us store and aggregate this large amount of data.

While a campaign is still active, we do more than simply calculate statistics on the data. For example, our contest campaign needs to ensure that a winner is properly selected. The winner could be the Nth entry, every Nth entry, the first N entries, etc, depending on how the campaign is configured. Selecting the wrong winner, or more winners than we are supposed to select would obviously be bad. So, we rely on the data integrity features provided by MySQL to help us do this correctly. However, once the campaign is over, the data is only used for statistics.

Based on these requirements, we decided to use CouchDB as an archive database. When a campaign completes, we could move the data out of MySQL into CouchDB. This would make the MySQL tables smaller and more efficient, and allow us to take advantage of CouchDB’s views to handle the statistics. But, this also meant that our application would have to interact two databases instead of one, and for the most part, be ignorant of which database the data was coming from.

Configure the application to access CouchDB

Before our application can talk to CouchDB, we need to tell it a little bit about our CouchDB installation. The CouchRest-Rails plugin aims to make this as easy as possible for Rails applications. CouchRest-Rails provides the necessary hooks that allow you to specify your CouchDB configuration in a couchdb.yml file, which serves the same purpose as the default database.yml file used by Rails. Simply update this file with your CouchDB connection information, and you’ll be able to easily connect to CouchDB from within your application.

CouchRest-Rails also provides a series of Rake tasks that help you manage your databases and views.

Define the documents

The very first thing you need to do when moving data to CouchDB is to figure out what your documents will look like. I talked about this in CouchDB: Databases and Documents, so I won’t cover it again here.

Write code to create documents from relational database backed data objects

Once you know what the documents are going to look like, you need to write some code that will convert your RDBMS backed objects into a document, and store it in CouchDB.

We decided to use CouchRest to help us interact with CouchDB. CouchRest consists of two main parts: code to interact directly with CouchDB via a set of APIs just above CouchDB’s HTTP API (known as CouchRest Core), and code that allows you to create an object model backed by CouchDB. The ExtendedDocument class is the cornerstone of the object model code. ExtendedDocument is like ActiveRecord::Base in Rails. It serves as the base class for CouchDB backed objects. It provides convenient ways to define document properties, access views, define life cycle callbacks, create documents, save documents, destroy documents, paginate view results, and more.

A class extending ExtendedDocument simply needs to define the properties that make up its document.

class ArchivedContestCampaignEntry < ExtendedDocument
  use_database :contest_campaign_entry_archive

  property :campaign_id
  property :user_id
  property :entry_number
  property :winner
end

Then, all it takes to save a document in CouchDB is to create an instance of this class, set its properties, and call the object's create method.

Determine how data will be moved to CouchDB

Now that we have code that can convert RDBMS objects into documents, we need to figure out how to actually get those documents into CouchDB. This step will likely be dependent on how you plan on using CouchDB. For us, we decided it would be best to “archive” records after their corresponding campaigns have been over for 48 hours or more. So, we created a nightly cron job to fetch all non-archived campaigns that have been over for 48 hours or more, and move their corresponding entries to CouchDB. When a campaign's entries have been moved, an “archive” flag is set on the campaign itself, so the application knows to fetch its entries from CouchDB instead of MySQL.

One important item to point out is that CouchDB supports a bulk save operation. This operation allows you to save a batch of documents with a single HTTP request. This is a big time saver, as executing one HTTP request is obviously much quicker than executing several thousand. Our archive cron job takes advantage of this. When archiving entries for a particular campaign, we will build one document for each entry record, and then toss that document into an array. When that array exceeds a certain size, 2,500 in our case, a single bulk save request is sent to CouchDB with the array of documents. This dramatically decreases the number of HTTP requests sent to CouchDB, and the amount of time required to add data to CouchDB.

In addition, our archive job will pause to rebuild all of the views in the database after 100,000 new documents have been inserted, as well as at the end of the job. The final view rebuild is necessary since all of the view queries done from within the application ask for stale data, which will not trigger a view update. We never did any research to determine if this was better or worse than waiting until the end of the job, which could produce over 500,000 new documents, to rebuild all of the views. This step was simply driven by the gut feelings of the three engineers working on the project. I'd be interested in hearing from you if you have done any research to determine if incremental view building is more or less efficient than a big bang view rebuild after a large import has completed.

Replacing SQL queries with CouchDB views

Next, we changed the application to support the substitution of SQL queries with CouchDB views. We did this in several steps.

Identify queries being performed on the data you want to move

The first step in replacing SQL queries with CouchDB views is identifying all of the queries being performed on the data you plan on moving to CouchDB. This took a few hours to do, but was not difficult by any means. We simply searched the code for all instances of the ActiveRecord class name and the MySQL table name for the tables with data being moved. We also tracked down all ActiveRecord associations that were made to that particular table. We then made a note of what the queries did, and how they were used.

Abstract away the query

After the queries had been identified, we moved the execution of all queries to a new class. This freed the rest of the application from having to know if the data being fetched lived in MySQL or CouchDB. The new class would make that decision, delegating to the ActiveRecord class if the data was in MySQL, or the ExtendedDocumenet class if the data was in CouchDB. To start off, we simply delegate to the ActiveRecord class since we have not yet implemented the CouchDB views.

class ContestCampaignEntryDelegate
  def self.find_all_by_campaign_id_and_winner(campaign_id, winner)
    # Delegate to the ActiveRecord object
    ContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
  end
end

Build views to replace the queries

Now that we have the complete list of queries performed on the data that we wish to archive, we can begin building the necessary CouchDB views to support those queries for archived campaigns. I wrote about CouchDB views in previous posts. See those posts for more information.

Add methods to the ExtendedDocument class to query the views

CouchRest gives you a few options when it comes to creating and accessing your views.

One option is to use the view_by method available on all classes that extend ExtendedDocument. view_by not only makes the views easily accessible via the code, but it will also take care of creating and storing the view in the database.

In its simplest form, view_by will generate the necessary map function based on the parameters you specify. This example from the CouchRest documentation shows the map function that will be generated when view_by :date is called inside a class named Post:

function(doc) {
  if (doc['couchrest-type'] == 'Post' && doc.date) {
    emit(doc.date, null);
  }
}

view_by also lets you specify compound keys (view_by :user_id, :date) and any parameters that you wish to be used when you query your view (:descending => true).

If you need to do something a little more complicated, view_by will also let you specify the map and reduce functions to use. Here's another example from the CouchRest documentation:

 # view with custom map/reduce functions
 # query with Post.by_tags :reduce => true
view_by :tags,
  :map =>
    "function(doc) {
      if (doc['couchrest-type'] == 'Post' && doc.tags) {
        doc.tags.forEach(function(tag){
          emit(doc.tag, 1);
        });
      }
    }",
  :reduce =>
    "function(keys, values, rereduce) {
      return sum(values);
    }"

Another option for creating and accessing views is to use CouchRest Core. CouchRest Core, as described above, is a raw, close to the metal set of APIs that let you interact with CouchDB. These APIs let you do basically anything, including creating and accessing views. This example from the CouchRest documentation shows how you can create and query a view using CouchRest Core:

@db = CouchRest.database!("http://127.0.0.1:5984/couchrest-test")
@db.save_doc({
  "_id" => "_design/first",
  :views => {
    :test => {
      :map =>
        "function(doc) {
          for (var w in doc) {
            if (!w.match(/^_/)) emit(w,doc[w])
          }
        }"
      }
    }
  })
puts @db.view('first/test')['rows'].inspect

For accessing our views, we decided to go with a hybrid approach. We didn't really feel comfortable storing our map and reduce functions inside the application code. Doing so made it less clear on how we could gracefully introduce new views or update existing views in production, keeping in mind that some of these views could take hours or days to be built for the first time. Instead, we stored our map and reduce code outside of the application, and used CouchRest-Rails to help us get those views into the database. This allows us to push new or updated views independent of the application, giving us time to build the views before anything tries to access them.

Since the views are already in the database, we decided to use CouchRest Core to access them. We created a class called ArchivedRecord to make working with CouchRest Core a little easier. ArchivedRecord contains methods that do type conversions, manage bulk save operations, incrementally regenerate the views, and more. It also contains a series of methods that help with executing views with similar behavior. For example, there are methods that will simply return the number of rows returned by a view, execute a view for a specific timeframe using the dates stored in the documents, etc. These abstractions also handle any errors that could pop up when accessing a view. Our application code uses the abstractions provided by ArchivedRecord to access the views.

Change the delegate class to call the ExtendedDocument class for archived data

Now that our views can be accessed via the application code, we can modify the delegate class to call the ExtendedDocument object's query method to fetch data for campaigns that have been archived.

class ContestCampaignEntryDelegate
  def self.find_all_by_campaign_id_and_winner(campaign_id, winner)
    campaign = ContestCampaign.find_by_id(campaign_id)
    if campaign.archived?
      ArchivedContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
    else
      ContestCampaignEntry.find_all_by_campaign_id_and_winner(campaign_id, winner)
    end
  end
end

Deal with the ActiveRecord associations

The last piece of the puzzle is to deal with the ActiveRecord associations. ActiveRecord associations are magic little things that make a record's associated data accessible via methods on an instance of the ActiveRecord class. For example, if I wanted to declare an association between a contest and its entries, I would simply declare the following at the top of my ContestCampaign class:

has_many :contest_campaign_entries

ActiveRecord takes care of joining the contest_campaign_entries table with the contest_campaigns table, and making all of the related campaign entries available via a call to some_contest_instance.contest_campaign_entries.

This will not work for us, as the contest_campaign_entries table will not contain any data for archived contests. So, we need to handle associations differently.

Instead of using the above code to create the association, we use the following:

has_many :active_contest_campaign_entries,
  :foreign_key => 'contest_campaign_id',
  :class_name => 'ContestCampaignEntry'

This more verbose version tells ActiveRecord that we want to setup an association, named active_contest_campaign_entries, on the class. Since we're circumventing the convention of naming the association after the foreign key to the associated data (which is in turned named after the associated table), we need to specify the foreign key to use, and the name of the class that backs that table.

To keep from breaking the existing code that uses the contest_campaign_entries method to obtain related entry data for a contest, we define a new method on the class with that name to fetch the associated data. The new method simply calls the corresponding method on the delegate class, which will pull the associated entries from MySQL or CouchDB, depending on if the campaign has been archived.

class ContestCampaign
…
  def contest_campaign_entries
    ContestCampaignEntryDelegate.contest_campaign_entries(self.id)
  end
…
end

class ContestCampaignEntryDelegate
…
  def self.contest_campaign_entries(campaign_id)
    campaign = ContestCampaign.find(campaign_id)
    if campaign.archived?
      ArchivedContestCampaignEntry.find_all_entries(campaign_id)
    else
      campaign.active_contest_campaign_entries
    end
  end
…
end

ActiveRecord supports other associations besides has_many. These other associations also add methods to the class that will fetch associated data from the database. In our case, some of this associated data is going to remain in MySQL. CouchRest will not (and should not) automatically fetch the corresponding data from MySQL, so we needed to handle this ourselves.

In our documents, we store the ids of the associated data still in MySQL (see campaign_id and user_id in the document snippet below). Because we have associations setup between the ContestCampaignEntry class and the ContestCampaign and User classes, ActiveRecord adds methods named campaign and user to ContestCampaignEntry that will fetch the associated objects. We need to do the same in our ExtendedDocument class.

class ArchivedContestCampaignEntry < ExtendedDocument
  use_database :contest_campaign_entry_archive

  property :campaign_id
  property :user_id
  property :entry_number
  property :winner

  def user
    @user ||= User.find_by_id(user_id)
  end

  def campaign
    @campaign ||= ContestCampaign.find_by_id(campaign_id)
  end
end

The user and campaign methods in the class above will take the ids stored in the document and fetch their corresponding objects from MySQL. In our case, these values will never change for an archived entry, so we hold on to the objects as instance variables to avoid doing additional queries when they are referenced again.

Make the ExtendedDocument class act like the ActiveRecord class

As I stated above, one of the goals was to make it so the application code does not need to know which database the data is coming from. Since the data can be returned as instances of two different classes, ContestCampaignEntry or ArchivedContestCampaignEntry, we need to make sure that both of these classes implement the same methods, and behave the same way. Failing to do so could result in hard to find bugs, or straight up exceptions.

One group of methods to pay extra attention to are the convenience methods that ActiveRecord adds to the class based on the attribute types in the database. An example of this is the attribute? accessor method that is added for boolean attributes. All attributes get an accessor named after the column in the database, but boolean attributes get an additional accessor, containing a “?” at the end. I personally use the “?” variation of the accessor method all of the time, as I feel it makes the code easier to read and understand.

CouchRest on the other hand is not able to determine in advance the data types of the properties you have stored, since CouchDB is a schema-less database. So, it is not able to do anything special for properties of a given type unless you specifically tell it what the type is. CouchRest does allow you to specify a type when you declare the property, but the current release (version 0.32) only uses this to cast property values into their proper type after they are fetched from the database. I've submitted a patch that will generate “?” accessor methods for properties with a type specified as :boolean. However, this is just one example of how your ExtendedDocument class could be subtlety different from the corresponding ActiveRecord class.

Summary

As I stated at the beginning of this post, changing the application to work with CouchDB was much more straightforward than getting the views to work as expected. Perhaps this is because I'm a developer, and not a DBA. But, great libraries like CouchRest and CouchRest-Rails certainly go a long way in helping to write clear and concise code that interacts with CouchDB. I can only hope that other programming languages have, or soon will have, libraries like these. The fact that CouchDB has a great API built on a protocol that everybody can talk, HTTP, certainly makes it possible.

couchdb, couchdb case study, couchrest, rails, textme

This is part 4 in a series of posts that describe our investigation into CouchDB as a solution to several database related performance issues facing the TextMe application.

In the previous post, I wrote about the many features of CouchDB views. In this post, I will describe the challenges we faced when replacing MySQL queries with CouchDB views.

Map/Reduce

One of the largest challenges with CouchDB views is simply wrapping your brain around the map/reduce model. If you’ve spent any significant amount of time in the relational model, this can be quite a task. Do not underestimate it. Give yourself plenty of time to make this adjustment. In my opinion, setting aside one or two full weeks to read about and experiment with views would not be excessive. It really is a whole new world. After several weeks, I’m still not 100% sure how to use the map/reduce model to its fullest potential. In fact, there were a few queries that I could not figure out how to implement as views. Because of that, I had to keep around an archive table in MySQL, and I run those few queries against that table.

Javascript

If you don’t know Javascript, you may want to tack on another week to the learning curve. Javascript is an incredibly powerful language. However, in its raw form it is fairly basic and can take some getting used to. It does help that you don’t need to write too much Javacript to implement most map and reduce functions. However, if you’re new to Javascript, get ready to do some research.

There are view servers available in other programming languages, and it is easy to configure CouchDB to use them. But, CouchDB is still young and under heavy development, and these alternative view servers are not supported by the CouchDB team. So, use them at your own risk.

SQL

As I mentioned in the previous post, views are powerful and flexible. But, views are not nearly as flexible as SQL. SQL has been in development for decades. Even today, it continues to evolve as a language. You can do a ton with SQL. As of right now, views simply cannot rival this flexibility. The CouchDB team continues to add built-in Javascript functions to help write map/reduce code, and there is even talk about supporting map/reduce/merge. But as of right now, the feature sets are not even close. It is very difficult for any new technology to enter the game with the same, or even a comparable feature set to such a battle-hardened veteran. And to be honest, I highly doubt that the CouchDB team is even trying to match SQLs feature set. After all, CouchDB is not meant to be a replacement for the relational database. However, this is an important point to consider if you think your current relational database backed application might be a good fit for CouchDB.

Multiple views, one design document

Views live in documents called “design documents”. Views within the same design document share a B-Tree data structure. This means that when one view in the design document is built, they all are built. So, careful planning is required to make sure unrelated views do not live in the same design document. You would not want the re-building of one view to delay the accessibility of another, totally unrelated view.

Building/Indexing views

Views can take a L…O…N…G time to build from scratch. The view building process is also very resource intensive. This becomes less of an issue once the views have been built, as views are updated incrementally. It really only comes into play when you are adding many, many documents to a CouchDB database between view builds. However, one place where this is an issue is ad-hoc queries. Every week or two, I’ll get a request from a customer for data that is not available via our web application. While we’ll throw that request onto the product backlog so it is eventually available via the application, it doesn’t change the fact that our customer needs that information now. We usually satisfy such requests by firing up the MySQL client, and running one or more ad-hoc queries. This simply is not feasible using CouchDB views, especially if you are working with a large database containing millions of documents. CouchDB does support “temporary views”, which are ad-hoc views that you can build and execute on the fly. However, temporary views are not recommended for production use, because they need to be built before they can get you the data you need. This could take hours, or days depending on the size of you database and the processing power of your database server. Temporary views are meant as a way to test new views in development which will eventually be saved into a design document, and not for running ad-hoc queries.

View sizes on disk

I’ve already mentioned that each design document is stored in its own B-Tree, completely separate from the B-Tree that holds the documents in the database. These data structures can become quite large, especially if you have a ton of documents in your database. A large database combined with several design documents can take up quite a bit of disk space. For example, our main messaging database consisting of around 30 million documents is 20GB on disk. The 8 design documents for that database combine for a total size of 35GB. This brings the grand total, for the documents and the views, to 55GB. That’s a whole lot of disk space. CouchDB sacrifices disk space for performance, which is a good tradeoff, as disk space is cheap and virtually limitless these days. But sadly, this is not the case for everyone. To add enough storage capacity to handle this database, the other large databases, a mirror/backup of these databases, and still have room to grow, we were looking at an additional several hundred bucks a month in charges from our hosting company for the additional disk space. That can be a lot to handle for a small company. Some larger companies use SANs or similar storage devices that offer unmatched redundancy and reliability. However, these devices will often have a fixed maximum amount of storage, and can be very expensive to upgrade one the storage capacity is maxed out. Justifying the use of so much disk space on such an expensive resource could prove difficult.

Summary

Views were without a doubt the hardest part of this project to get right. My colleagues Dave and Jerry are still hard at work trying to find creative ways to reduce the size of the views on disk. We’re very happy with the performance boots that we’ve seen using views in our testing environment. But unless we can find a way to reduce the disk usage, or find more affordable storage options, we may never see these performance boosts in production.

couchdb, couchdb case study, textme