My goal this week was to find useful and preferably large data sets. Unfortunately, the kind of data I was hoping work with is not available to the public. I am still hoping to work with this type of data in the near future. In the meantime, one of my colleagues gave me access to a nice data set that was immediately useful.
This data set contains temperature readings for our office. Temperature readings are added to the data set every five minutes. My colleague created a nice RESTful interface to access this data set. In this post, I will describe how Node.js and CouchDB were used together with my colleague’s temperature data set to create and host the pages we see in Figure 1 and Figure 2.
First, I’ll describe is how the temperature data is collected and shared. We have some power meters available from a previous project. These can be seen in Figure 3. These power meters are able display how much power a certain power outlet consumes. They also have the ability to display the current temperature. All this data can be accessed in electronic format over a serial connection to the power meter.
My colleague, being the hardware guru that he is, connected the power meter (seen in Figure 3) to an mbed device (seen in figure 4). The mbed device is a tool for rapid prototyping with microcontrollers. It is programmed to read the data on the power meter using a serial connection and post it over HTTP to a MySQL database. The mbed device is a stand alone device. It just needs a serial connection to the power meter to collect the data and a network connection to post it. The REST resource that processes and stores the power meter data was written with Java using the Jersey framework. It is hosted in Tomcat.
Once the power meter’s data is in the MySQL database, it’s made available via a RESTful query interface. Using this interface, I can collect the data as it comes in by hitting the server about every five minutes and posting the results to my CouchDB instance. I do this with a Node.js application that I scheduled as a cron job.
This solution is a bit over-engineered. I’m pulling data from a MySQL database using REST and posting it to a CouchDB document store using Node.js. The objective is to get more familiar with CouchDB and to load some semi-useful data so a few extra steps are OK. Anyway, using Node.js to move data around is just fun. In a more realistic scenario, the mbed device could post directly to CouchDB.
Every 5 minutes, the Node.js application posts a JSON document to my CouchDB instance which looks like :
This is a plain vanilla JSON object that Node.js posts to CouchDB. Remember that we interact with CouchDB using REST. Adding a document is done with a simple HTTP post to a particular URL. In this case, doing an HTTP POST of the JSON object described in Figure 5 to a URL like http://<CouchDBServer>/<CouchDBDatabase> will store the document in the <CouchDBDatabase> database. Once I have some data in CouchDB, I can create views to slice and dice the data.
To get the daily average of my temperature readings I created a MapReduce view and saved it as avg_temp in CouchDB. The web page in Figure 1 uses this view. The view’s details can be seen in Figure 6. The map function code creates date/temperature key/value pairs. This is a series of dates with individual temperature readings. The reduce function calculates the average of the values for each key.
This view can be queried for the daily average using Couch’s built in query mechanism for views. An HTTP request query like : http://<CouchDBServer>/<CouchDBDatabase>/_design/avg_temp/_view/avg_temp?key=“1/15/2012” should come back with a JSON document containing a value of 71.751185… for the average.
CouchDB is quite interesting. I am glad I found a semi-useful use case to explore CouchDB a little more. Now can I point people to my office temperature web page and graph if it gets too hot in my office.