Deploy to 40+ sites more quickly and with less room for error? Yes, please.

Managing a single site is pretty simple, by modern web agency standards. Develop locally, host the code on a server, commit and update it, apply security patches, and backup regularly. But what happens when your client has more than 40 affiliate sites all running on the same code base, each on a different server? The simple deployment of a codebase on one site suddenly becomes much more complicated.

We have this exact situation with several of our clients here at Capellic.  Easterseals is one of those, and we are fortunate to have a great partnership with them (read the case study). So when we approached them with an idea to upgrade to a continuous integration process, they were happy to try it out.

What is “continuous integration”?  I’m glad you asked.  To understand continuous integration, it’s helpful to look at what a site development process looks like. 

Website development 101

Websites are built on code, which is stored in a central repository. Developers can check out code from the repository to their local computer, which they can use to run a copy of the site. They can then make changes to the code to fix bugs or add features, and then they can check in their changes. Once the code is checked in, a merge request can be generated - a request to integrate the changes into the main code base. The new code can then be copied, or deployed, to a test site, which is another copy of the site that is accessible to more than one person on their team. It gets tested, merged, and then deployed to the publicly-available site, called production.

You probably knew all that, but it’s always good to cover the basics.

In the olden days, this whole process was manual, with a developer having to log in to various computer systems and copy files around and run update commands. And until recently, that is more or less how we were managing the Easterseals sites - but on a much larger scale. A developer had to remotely execute multiple similar-but-different commands on 40+ separate sites individually. We assembled all those commands into a JavaScript-driven page internally, which combined them with a list of variables and output them for copying and pasting in 40+ terminal tabs.

Sound tedious, time-consuming, and prone to errors? It was!

Enter Continuous Integration

Continuous integration (CI) is the process of automating all of that. A central computer takes in a configuration, a number of variables, and some login information, and runs all of that in the background, in a regular, predictable, and best of all, simultaneous fashion. We can even run different sets of commands for different needs.

For our purposes, we’ve identified 5 separate use cases:

1. Build a multi-dev: Pantheon, our web hosting partner, allows for the creation of multiple copies of each site, based on different code, for testing. We now do this automatically on each merge request.
2. Run a site backup: Backup and archive the site’s content and files.
3. Clone site data: Pull the production site’s content and files into the testing environments
4. Dev build: Build the new code to the “dev” server, which is the primary testing environment. This is usually done towards the end of the sprint.
5. Site build: Build the new code to the “test” server, which is the final testing environment, or to “live”, the public-facing production environment.

Each of these tasks can be run on one or more of the 40+ sites, so no more logging into multiple servers to paste commands!

We’ve now run several full deployments on the Easterseals sites, and are quite pleased with the results.

Benefits of CI

Time savings

It takes a lot of time and effort to run a series of commands on 40 or more websites. Even if you are working in multiple windows, there is still significant time required to manage that process. Our new CI process required a lot of up-front developer time, but now, kicking off a full deployment for the whole suite of sites is a matter of clicking a couple buttons and entering a single line of data. Did we spend too much time on it? Well, Randall Munroe of xkcd.com compiled this handy chart about routine tasks and efficiency.

We’re right about where the red dot is, so I think we’re ok.

Those of you that like facts and figures are probably asking, “How much time, exactly, did you save?” Well, running the full process on Easterseals–backup, site clones, test deploy, production deploy–used to take approximately 45 minutes, and you couldn’t do much of anything besides click between terminal windows to monitor the results and input the next command.

With automation, kicking off each of those tasks takes about 90 seconds, including the time to log into CircleCI, and then you can go and do other things while they run. Circle tracks each build as it runs, outputting status messages right in the app. Reviewing this output takes maybe another minute (if nothing fails). With the four tasks we counted * 2.5 minutes… your actual time spent is only ~10 minutes. 

Risk factor mitigation

Cutting and pasting commands from a document into many different windows is an incredibly error-prone process. It’s easy to click the wrong window, or copy the wrong command, or mess up in a myriad of other ways. By automating the practice, we are running the exact same process, in order, every time, on multiple instances, with the precision of a computer. This means fewer problems, less downtime, and more reliable deployments.

Error monitoring

Bugs happen. Even in the most well-scripted automated deployments, sometimes a communication glitch or out-of-memory error occurs. CircleCI, our choice of service providers, has robust error reporting, with real-time monitoring on all active processes through their application. This makes it easy to see if something goes wrong and take corrective action immediately. It even allows you to rerun failed jobs from within the application, which often makes fixing failures a breeze!

Ease and access

The process of logging in to multiple sites on a command line, pasting commands, and deploying sites is an intimidating one. It’s generally only a task that developers are willing or able to do. But now? Each of those deployment processes outlined above is described in one paragraph in our client documentation. Pretty much anyone with a login to our CircleCI instance - including our clients! - can do it.

Able to push to a subset of sites

One of the clients on our framework has two sets of sites in their portfolio - some of which deploy in the morning and some of which deploy after the close of the workday in their timezone. By utilizing some cool grouping, we are able to run backups, clone site data, and run deployments to test environments to all the sites, then run production deployments to their “AM” group, all in the morning. Then, later in the afternoon, we can run production to their “PM” sites, and make sure that we don’t disrupt active use of those critical sites.

How we did it

The inspiration to finally tackle this challenge came from DrupalCon Atlanta, where we took in Lullabot and the American Booksellers Association’s session Leveraging Drupal SaaS to Power 400 Websites as Unique as Independent Bookstores. They are managing hundreds of websites using a continuous integration process that is extremely sophisticated. It led to this fever dream scrawl of notes, which was so insistent in my head that I couldn’t sleep until I wrote it down.

Our CI stack is built around CircleCI, one of the industry leaders in the Continuous Integration space. Our process uses the Docker container that Pantheon uses for their own Circle integration, which comes pre-installed with a bunch of command-line tools that are useful for our purposes.

We then use a combination of composer, Robo scripts, bash scripts, and Pantheon’s Terminus tool to assemble the site’s files, push them to Pantheon, update databases, and import new configuration.

I could spend an entire blog post detailing each line of code, but this ground has been covered pretty well elsewhere. Our key highlights are noted below, with a couple things that tie all the various parts together.

Pantheon Tags

The sites list in Pantheon allows you to apply tags to sites, individually or in bulk. This allowed us to specify several subsets of sites, for different purposes. On Easterseals, the tags include `es` (all 46 public sites), `es_devonly` (all sites, including one that’s development-only), and `es_canary` (2 sites used for final regression testing). This is also how we manage the split release I wrote about earlier - the morning group has tags suffixed with `_am`, and the evening group with `_pm`.

Tagging the sites would be fairly pointless for CI purposes, were it not for Terminus, which has a query command that will return a list of sites by tag. Our integration can take a tag as a parameter, query Pantheon, retrieve the site list, then run processes on each of the returned sites.

CircleCI Continuation Orb

One of the nice features about CircleCI is that reusable configuration or functionality can be packaged up into what they call “Orbs”, which are similar to plugins or modules. The most important one for our purposes is the “continuation orb”, which allows for one Circle Workflow to dynamically switch to another workflow.

Yeah, this seemed really useless to me, too. At first.

What I was looking for was a way to define, at run time, a whole bunch of jobs to run in parallel, passing them a site name and ID from a list. This blog post by Circle Developer Advocate Angel Rivera was the key to doing just that.

Using it as a springboard, we developed a set of template files, one for each of the jobs we wanted to automate.  Each template comes with one section, designed to repeat, that has tokens for site name and id. A Robo script then ingests both the list of sites from the tag query and the template, loops on the repeating section, replaces the tokens, smooshes it all back together, and outputs a new Circle configuration file. Then, the continuation orb is put into play, and Circle jumps to the new, dynamically-generated configuration file, and runs the jobs on each of the sites.

This process is repeated each time one of the automations is run, based on the tag entered at run time and the dynamic list of sites from Pantheon. It takes about 2 seconds to generate that additional configuration, and then Circle runs all of the defined jobs in parallel. Our last full deployment from dev to production for all 46 Easterseals sites took 9 minutes 59 seconds to complete, start to finish. 

CircleCI Caching

Circle allows you to cache any number of things. In this case, we build the entire codebase– `composer install`, `npm install`, `gulp build`–once. Then, we cache that entire top-level directory.  That cache can then be loaded in each of the 46 workflows, saving us having to build an identical codebase 46 times.

What’s next?

There’s one critical use case we didn’t put into CI - testing. The complexity of some of our clients' sites means we have extremely lengthy and comprehensive test suites. Right now, we are still running those manually prior to deployments. One of the next tasks I’ll be taking on is automating those tests, probably as a separate workflow. It will require a bit more infrastructure work on the Circle environment, but the investment will be well worth it. There are, of course, always refinements that can be made, so we’ll also be keeping an eye on the process as we move along to see where there’s friction and clean things up.

Thoughts?

There is so much more we could talk about on this topic. If you’re interested in discussing it, please feel free to contact me at joshua @ capellic [dot] com or on Drupal Slack - I’m @srjosh there.