Panorama Gallery Here: The 360 Gallery

A few weeks ago I started see http://occip.it/ links in my twitter stream. I always clicked on the links when I saw a new one and thought that there were some very interesting photos being posted. But that was it, I only got to look through a few that were in my stream, and perhaps if I remembered to run a search on twitter once in a while I would see a few more. I wanted a better way to browse them. Then Josh Fraser tweeted: “i really hope @occipital makes a gallery of all their pubic panos”. Why wait? Introducing The 360 Gallery.

A few of my choices of how to build this gallery were based on using new tech. I’m not new to CouchDB, but I havent played around with views as much as I should have in the past. I’ve been writing a few things in node in the past month, but nothing web facing so I wanted to see what if any frameworks were available and if they are any good. And thanks to the AppSumo bundle last month I have some Linode hosting that I’m trying out as well (so far so good).

Getting some Data

So from what I can tell there is not public list of panoramas from Occipital. That’s ok because most people tweet them. In fact once they tweet the image using the Occipital 360 Panorama App then the image is also posted to yfrog or twitpic depending on the user’s choice. To get a list of images I tried to use the streaming track api from Twitter, however there are limitations on their allowed characters. (Their track search parser must not be very feature complete compared to search.twitter.com) I ended up having to ditch that and poll from the search once a minute for any new occip.it links, a little regex later and I have all the data need. I store the data + original tweet in couchdb. It’s great to work with an API that speaks JSON, a scripting language that is JS, and a DB that speaks JSON, flows together very easy.

CouchDB

CouchDb makes apps like this very simple to setup. Couch is a JSON document store that allows you to write views using Map Reduce functions written in Javascript. Then you can query those views(there are plenty of query options). After creating the view I wanted to proxy access to the view query with nodejs. I setup a route in express (more about express later). I did this to expose only a subset of the available query possibilities. You can see my recent feed for yourself The last integer in the url is the page number, super easy to setup in node:

I didnt want to host my own couchdb. I could but why should I? I picked Cloudant, only because I’ve used them before and already have an account. CouchOne is also a very good option for CouchDB hosting (several core committers to the couchdb project work there) A quick test with siege shows that Node + CouchDb should hold up ok. (I think Cloudant does some caching on top of views which keeps them faster.)

Siege shows a fairly consistent response without adding any further optimization. It could be a lot faster if I cached the results locally on the node server.

So lets cache some of those calls locally.

Since there are only a couple of requests this node app accepts we can cache the response it in memory.

Now look at the response time, that’s better just over 10 times faster on average.

Notice I handle fewer concurrent requests now, not because I cant, but because the server is responding so fast that many concurrent requests arent ever happening (at least with running siege from my laptop). Also those response times are over the net, so I’m really happy with them.

So all of the360gallery.com just uses those few simple JSON calls to nodejs/couchDB over XHR. This made the site easier to write, and provided a simple way to add paging. Overall I really like CouchDb and views. I already like JavaScript so Node.js is very promising even if it lacks some maturity. There are many unfinished libraries and frameworks… but some promising ones are emerging.

A few notes

I dont like the way the Connect or ExpressJS handles static files. The paths get all messy and a there doesnt seem to be a very well thought out file structure for projects in general. I’m hoping to work on this. Also, I never thought I would have said this, but I missed django like templates express has several options and maybe I need to look around more but I wasnt happy with what was available so I used my own solution. Any ideas?

Nginx does a great job is usual. Did really mention it but it worked out very well. If you are setting up to Nginx + node this tutorial can be helpful. There is plenty of info out there for node + nginx.

I used the twitter @anywhere JS lib for the site, and I don’t really like it. The lib seems heavy and has a lot of requests back to twitter. I might be replacing it. Anyone have thoughts about @anywhere?

I would like to see more structure to the espressjs framework (or any nodejs web framework) even it isn’t strictly enforced some norms that everyone expects and relies on would be nice. Any suggestions for using a web framework in node?

I’ve used several lightbox like plugins before, but I really liked the look and simplicity of this one: fancybox

The End

I’m impressed if you read all the way through this post. Sorry for the length, just wanted to document most of this. I’ll post a little more about the code on the site when I post all the source on github later.