Calling the last few months here at Iconfinder “exciting” would probably be an understatement. Starting from September, we’re now a four man army working hard to take Iconfinder to the next level. The first step in this journey is swapping out our code base from 2009 with a whole new setup.
Iconfinder originally started out as a fun side project for my co-founder, Martin, who just wanted a simple way to search for icons when doing his everyday job. This, of course, meant that the focus from the beginning wasn’t on orchestrating a super scalable Web 2.0 architecture but simply to make the core of what is Iconfinder — searching — as usable and as fast as possible with the smallest development overhead.
In other words, the Iconfinder that you know and use has not been a NoSQL driven, framework religious academic exercise. Rather, to this day, it’s your ultra pragmatic, run of the mill LAMP stack. To help speed things up, we use XCache for PHP byte code caching and mod_pagespeed to help speed up assets. The most exotic bit is arguably that we added memcached as the first level cache this spring, but apart from that, this is LAMP at its earnest.
Since 2008, Iconfinder has been running on various size dedicated virtual machines at Media Temple who’s got a reasonable price point for medium size workloads. Currently, Iconfinder is running on their top of the line (dv) Dedicated Virtual with 16 GB of RAM and 16 CPU cores.
What wasn’t anticipated back in 2007 or even 2009, when the code base was rewritten, was that Iconfinder would grow to be as popular as it is today. Ranking well within the top 1000 most used websites in the world, we now serve more than 7,250,000 searches to 1,250,000 people every single month — and that’s not even counting the heavy API use. Kind of cool for a project that between school and work couldn’t get more than a few hours’ of attention every week, huh?
With great growth comes great challenges
The site’s growth means that we now have the opportunity to take Iconfinder to the next level and tackle some of the bigger challenges than just search. You may have noticed that a few weeks ago, we announced the first big news for Iconfinder in its new form: our coming marketplace. We’re really excited about this addition, which I personally feel is a natural extension of how I myself have always used Iconfinder even in the years before I joined the company. Of course, this is just one in a line of ideas that we have cooking, so there’ll be a lot of goodies coming your way in the future!
But, with an aggressive road map and the growth of the site itself comes a heap of technical challenges, which we need to tackle as part of leveling up Iconfinder.
The first, and probably biggest, challenge is scalability. Serving as much dynamic content as we do does put quite a strain on a system. To be honest, the LAMP stack isn’t the single fastest solution out there for a search engine. So, in order to keep Iconfinder running, the current code base is therefore highly optimized for caching. Even so, handling the current levels of traffic out of a single box is becoming painful. Also, anyone who’s done ops of any “real” web application probably cringed when they realised that we’ve run Iconfinder on just one server until now — no redundancy and no partitioning of workloads is not The Right Way™ to do things. It was clear that we needed to start fanning out onto multiple servers.
Another big area of concern was that the heavy caching was hard to combine with development of more dynamic features, as cache busting takes its toll on the service. So, to be able to handle an increasing number of searches while providing the dynamic user experience we want, we’d need to change our approach to caching and searching.
The last big challenge is less technical. Having watched companies like GitHub, Etsy and 37signals pump out features and fixes at a staggering rate on the back of Continuous Integration and extensive monitoring, I really wanted that too. I wanted to us to be able to ruthlessly build and deploy without that lurking fear of the site breaking at any point. That would require us to add testing and monitoring to the code base, something that’s currently limited to a Pingdom test alerting us when all hell breaks loose.
In the end we were faced with one of the toughest choices for any startup at a technical cross road; should we continue working on the current code base or rewrite the site from scratch? Either way, there was a lot of work to be done.
Despite the Internet overflowing with advice to never rewrite your code, we eventually opted to do just that. Why? Regardless of whether we were to continue with the current code base or write a new code base, a lot of work would have to be done. Rewriting on the basis of a framework and swapping out a series of components gave us the opportunity of latch on to larger road maps and get a lot of features handed to us for free.
So, for the last two months, we’ve been hard at work on what’s internally been known as “Iconfinder Next” or simply just “Nextfinder.” The name is of course a good luck charm and an homage to the successful rewrite of 37signal‘s Basecamp which was codenamed Basecamp Next.
Iconfinder Next is a fully fledged Django based web application which uses a PostgreSQL database rather than a MySQL database for anything but searching. For search we’ve opted for the rapidly evolving elasticsearch in part due to the awesome community behind it and also how insanely fast it is while being really easy to shard. True to the Web 2.0 generation, we’re using memcached for caching and the impressive redis for jobs such as our task queues, API rate limiting etc.
Behind the scenes, everything is fed to a Jenkins Continuous Integration server through GitHub’s Janky and code is deployed to a fully redundant setup running across multiple availability zones on Amazon Elastic Compute Cloud using Fabric. A mixture of Sentry, Graphite and the excellent Server Density cactches all mishaps and makes it possible for us to back track problems. Now, how’s that for a modern application.
Today, we’re opening it up for the rest of the World to start playing with! Because this is a whole new code base, there are guaranteed to be quirks that need to be worked out, so we’d love it if you would join us in beta testing it. How? Just go to beta.iconfinder.com, sign up and start using it like you would normally use Iconfinder. We’d love it if you used the Feedback link in the menu bar to tell us how it’s working out.
Thanks for helping us out!
Nick Bruun, CTO