Relay on Rails

Last week the engineering team at Image Relay (6 people strong!) attended RailsConf in Detroit, and at least for myself it was the final push needed to get the wheels in motion on creating an engineering blog for us devs at Image Relay.

It's something we've talked about casually over the years, but more recently I have either said to myself "I really need to do this" or have heard "why don't we have a blog?". It's felt increasingly silly to say "eh we'll do it later".

Thankfully the reality is that all we've got to do is stand up and say "let's make a blog and really do it!". It's a small commitment that is easy to make with a tiny push. The idea will gather momentum as needed, or it could fizzle out. Either outcome is totally fine, but it's better to learn a few lessons along the way, right? What was I actually waiting for, after all?

Going to conferences is nice like that, regardless of the technical content, just seeing fellow professionals discussing professional things can be just the kick in the ass you were looking for. It might be short-lived, but we start seeing our day-to-day operations through a new lens that seems clearer for some reason.

So here we are. Blog post #1 in our attempt at trying to talk about the day-to-day tech operations, interesting problems we've faced, and just about anything else in the realm of running a cloud-based Rails app (plus auxiliary support services) for 15+ years.

Every day seems to present a new problem, or a potential problem we could try to solve. Writing down our thoughts on problems we've faced is a form of rubber duck debugging and it helps us reflect on yesterday's work and better prepare us for success on the new unknown problems of today. Readers or not, it's a win win for the team.

I'll go into the technical details in a follow-up post, but I will say that the plan we've developed to run, build, design, & maintain this blog is maybe the most important factor in running it aside from the actual content we produce of course. Rails devs are supposed to run their blogs on Rails, obviously! I don't make the rules, I just try to follow them.

Aside from the blog intro, I also wanted to use this as an opportunity to introduce myself and tell a bit of my backstory.

I'm in my 40s (maybe late 40s, or mid, who really knows) living in rural Michigan, doing remote work as the CTO of Image Relay. I've been with the company since 2017, and was bequeathed the CTO position in 2018 when our previous CTO Buffy Miller headed over to YNAB (congrats on 6 years with YNAB Buffy!).

I studied computer science in Sacramento in the late 90s and received my bachelor's in the early 2000s. Web development wasn't being taught at that time in core curriculum work, but it was  interesting me. My school offered a "web developer certificate" (it sounds just as silly now as it did then, but it worked) that anyone could take. The instructors (2001-2002) were primarily post-dot com 1.0 bubble bursters and they taught Microsoft tech. I learned ASP while finishing coursework, then after college I worked as a contractor, first in classic ASP, then .NET. I primarily worked for a large healthcare company, working on backoffice web-based tooling for claims approvals and what have you. It was extremely uninteresting but it gave me a taste for what cubicle corporate development culture was like, and all the bureaucracy that came along with it. I wore a suit to work every day and had to show a badge to a security guard at at gate! It was very important stuff.

If you were part of the web scene at that time, you know that the HTML that systems like ASP.NET spit out was utter garbage. Anyone remember Dreamweaver? Markup was bloated, wrong, ugly, and oftentimes very difficult to style in any meaningful way. I found myself increasingly passionate about clean HTML and building layouts that were based on CSS vs. tables.

There was a world in the web development community of CSS-based design that was starting to blow up, and sites like CSS Zen Garden were showcasing the extremely impressive powers of combining semantic HTML with modern CSS. Everyone joked back then that CSS was no longer "just for fonts" but that was real. In 2000, that's about all everyone used CSS for. It was a totally different world back then.

I'd say by 2005 or so, the HTML/CSS community seemed to be sprinting in the direction of gaining more and more control over the HTML that backend systems were spitting out, but if you were working within a framework you had to choose your battles. While it was always an option to override controls, the juice wasn't always worth the squeeze. Some things were getting hard.

It was around this time I learned about the SXSW Interactive conference in Austin, and depending on what blogs you were reading in those days it felt like the conference to attend. Everybody who was anybody in the HTML/CSS community was going to be there. Nowhere else could you attend panels put on by the folks at A List Apart, and rub shoulders with one another at the Gingerman while Breaking Bread with Brad.

Although a very separate community, more and more people were talking about this thing called Ruby on Rails. Apparently it was an application framework written by a Danish guy that lived in Chicago while he was working on some weird project management software coded in some new language called Ruby that was written by a Japanese guy. If that's not the antithesis of what it feels like to be doing .NET work in corporate healthcare, I don't know what is. Sign me up!

Around June/July 2005 I preordered the first Ruby on Rails book (also the Pickaxe book because it seemed mandatory) and excitedly started working through the examples of pre-V1 Rails in the pre-release PDF of the book.

It's hard to capture what it felt like to thumb through the pages of the Rails book for the first time, maybe take the word "fun" and multiply it by 10 or possibly a larger number. Something like that. It was addictive, but in all the good ways something could be addictive. I loved it.

Coincidentally and conveniently, I moved to Austin in late 2005 (still doing .NET, remotely) and by the time SXSW 2006 rolled around I had taken a job as a full-time Rails developer at a brand new startup called Spiceworks. It was there I primarily worked on their distributed application, which was a free tool for IT teams to manage their inventory. If you're inferring something and scratching your head, yes your gut is correct. We created and distributed a Ruby on Rails application hidden inside an EXE that ran on Windows systems. In 2006!

I don't know who at Spiceworks has discussed this accomplishment publicly, nor do I claim to have any large involvement in the complexity of accomplishing that task (I was just a Rails dev), but it's a statement that still to this day makes me say WHAT????

Austin had an awesome Ruby & Rails scene to say the least. By 2006 there was already an active Ruby enthusiast group called Arctan (I can't find a decent link to them now, but I assure you we existed) and Austin on Rails was growing. Dave Thomas (Pickaxe Dave) lived in Dallas at the time and would attend our meetups on occasion. I was lucky enough to work with one of the founders of Austin on Rails and we became great friends. He was already a senior engineer by this point, and was always willing to help a young buck with a seemingly never ending stream of questions. If it wasn't for that person I don't know if I'd be here today talking about Rails. But that's how the Rails community is – excitement is fostered and encouraged. It was contagious and I loved it.

I also worked with Do512 off and on while that platform was launching, and eventually took a full-time job with them in 2013 (I think??). Behind the scenes the Do512 platform was also serving music festival line ups and schedules in a Rails monolith on EC2 classic – like Image Relay, Do512 was an early adopter of AWS.

Do512 eventually spun out to a franchise-style business with DoStuff Media as the parent organization and they launched similar sites across the United States. The Rails app for Austin morphed into a multi-tenant app where each city was a tenant. If you live in a large city, there's a good chance a Do Site exists (or did exist) for your area code. At the very least, the domain name probably is registered and redirects to DoStuff's marketing site.

By 2016 we were running several Rails apps performing varying states of production workloads. We were trying to separate the Do City traffic from festival traffic, and also have a separate backoffice tool for internal staff, as well as auxiliary apps to help with business needs. We were a small dev team with a lot of pressure on our shoulders, and an increasingly outdated Rails application that seemed to attract emergencies like flies. I learned a lot during that time about performance, N+1s, dealing with latency spikes, and cloud instance sizing & scaling. I also learned a great deal about how to maintain a Rails app that seemed to work great for development and testing, but was an absolute bear to keep healthy when real traffic loads rolled in (and there was nothing that you could do about it). We did some wild stuff to keep ourselves from having a heart attack when Lollapalooza, ACL, Jazzfest, etc decided to do an email blast without letting us know. Or even when we did know and couldn't do anything about it.

Mid-2016 found me moving again, this time to Vermont. Looking for contract work led me to Image Relay, a Vermont based company that created a DAM app. What's a DAM app? 2017 me had no idea, but it was a Ruby on Rails shop and they had been running on AWS cloud since the beta days in 2008 or so. Nice! I can do that.

The 2-second pitch I give people on DAM: It's like Dropbox or Google Drive, but for a company.

I started as a contractor, then went to full-time, then a year later I was offered the CTO position. I absolutely loved the company and who I was working with, and by that time I came to understand quite a bit about the DAM space and the common problems that our users faced. It was a scary & stressful time, filled with many moments of imposter syndrome feelings. But, we grew the team and just chipped away at our tech debt, and tried our best to prioritize the business critical work above all else.

We grew the infrastructure to support an increasing volume of traffic from our growing customer-base, we moved services that needed to scale under load to AWS Lambdas and tried to rely less and less on custom written code that performed tasks that off-the-shelf software could already do reliably. It's common for 1-person dev shops to try to do everything themselves, both due to budgeting reasons as well as superhero mentality, but it was time to accept that we didn't need to do it all and shed that young skin.

As the customer-base grew we increasingly faced compliance-related questions from both prospective and existing customers. Our larger customers started sending in annual security questionnaires and the distractions from day-to-day operations started piling up. I knew the infrastructure was not in a position to achieve compliance, anyone who was launching services on AWS in the late 2000s knows that security and compliance best practices was not exactly top of mind, let alone possible. When you run on that same stack for 10+ years, the effort to get out from under it can feel like being at the bottom of a dog pile. It was paralyzing to say the least.

Pursuing compliance (SOC2 Type 2 for those wondering) was clearly the correct path to choose for a B2B SaaS in 2020, but getting there was like trying to see into a black hole. I had no idea where to even begin. Lucky for us, we weren't the only company facing this problem and we were connected with the fine folks at Converge. Together we collectively worked our butts off for the better part of a year to modernize the entire infra stack and swapped it right out from under our user's noses without anyone noticing. It was a huge accomplishment that simplified all aspects of running the business. Sure, maintaining compliance takes work, but at least it's predictable, known work as opposed to the common distractions and emergencies that arise from no planning and no compliance. Compliance work can be divided out and distributed across a team of people. We can create public compliance dashboards for our prospective and paying customers so they can self-serve their security questions as much as possible.

We now maintain 2 core Ruby on Rails applications, one is an older & classic monolithic app that 99% of our customers use, and then our new modern Rails stack that is built with all the lessons of past Rails apps baked into it. We're using View Components for UI, using Tailwind for design, Stimulus for interactions, & containers for ease of development and infrastructure best practices. We are in-flight on sunsetting the old application, with the replacements being a variety of more specialized tools. Moving away from the monolithic app that does everything is resulting in a much smaller blast radius when things go awry and allows us to scale out specialized services in parallel when increased loads occur.

The Image Relay engineering team loves Ruby on Rails, and we are using the lessons and experiences from our senior engineers over the past 20 years to help drive the choices we make. Our app should be fun to work on just like it was back in 2005 when Rails was brand new, and that philosophy drives nearly every decision we make.

Cheers to Rails and the community we're all a part of 🎉