On 3/13/2015, we set Imagineer loose as one of our newest and most cutting edge open source projects. I'll explain why we built it, how it helped us, and where it's headed.
Images are important. At Roadtrippers, we use images to show our users that we know where we're taking them. We provide a lot of guides and blog posts. We also drive and decorate other content types with images.
Designs change, and new image sizes are required often as we shuffle our layout and inch toward the perfect UX. As our internal image library grew, the time it took to create a new version grew. Generating resized images was no small task for a few million place records with any number of images!
A little about us
Our back end is built with Ruby on Rails, and we use the rightfully popular CarrierWave gem to manage images for our models. Prior to Imagineer, we used two different integrations with CW:
These have their own merits. Fog is useful, but new sizes require loading each record, generating, and resaving. Cloudinary takes much of that pain away, but we ended up up paying for features we didn't need and feeling limited by their API.
We set out to create a solution for ourselves. We had a list of goals:
- Simple code to create new images, no extra tasks need to be ran
- Keep image processing off our app servers
- Able to handle a flood of identical requests, typical for new image versions
With that in mind, we created Imagineer. Imagineer produces images based on the request URL. If you request
/-resize_50x50/filename.png, you recieve a copy of
/filename.png that is 50px wide and 50px tall.
Imagineer is a simple Node app and Lambda function. The Node app handles the image requests, calls the Lambda function, and redirects the connection to the image after creation. The Lambda function downloads the image from S3, creates the new version, and finally stores the new version in S3.
What this means for us
When someone requests an image that doesn't exist, that request is forwarded to Imagineer. The application doesn't have to worry about anything other than knowing how to encode the request. We extended CarrierWave in our app to translate processes to Imagineer paths.
Processesing of images is handled entirely in Lambda. One of our goals here was just to get familiar with Lambda and see how we could make it work for us. By doing so, we were able to remove image processing from our app while making it flood resistant. We pushed the optimization even further by placing a CDN—for us, that's MaxCDN—in front of the whole thing to add an extra caching layer and all the benefits of geographically distributed assets. The app server and the Imagineer server can safely defer to Lambda, S3, and the CDN.
Using Node and Lambda, we were able to accomplish everything we set out to do in about 215 lines of code! We began using some version of this in production months ago, and haven't looked back.
Challenges and caveats
Configuration is open ended, and therefore potentially not straightforward. You will need to host the Node app somewhere. For that, we chose Elasticbeanstalk.
You will need to forward image requests to Imagineer. For that, we use a redirect rule on the static web hosting option that S3 provides. Our configuration is posted in the Imagineer README.
Resources for Lambda functions are limited. We use the maximum of 1024mb memory for each process, but things like large image files and changing interlacing can cause the process to overload and die.
Some key features that we'd like to work on are expanding the supported image manipulations, cleaning up the path encoding, and adding more automated configuration.
Our hope is that other folks might benefit from using Imagineer, and lend their feedback or development skills to continue improving the project. It's a tool that we rely on, but also a gift to the open source community that has given us the tools to build Roadtrippers.
Holler if you've got any questions about Imagineer or anything else that we do here at Roadtrippers. If you're interested in more about our experience with Lambda, check out Lambda: Bees With Frickin' Laser Beams. Thank you for reading!