Developing Drupal websites in the cloud using Gitpod

At Capellic we’re building and maintaining Drupal websites using a cloud-based development environment. My last writing on this topic gets into the reasons why. This tutorial is a guide for getting you rolling with developing Drupal websites using Storybook with Emulsify in the cloud with Gitpod, DDEV, Bitbucket and hosted on Pantheon. It’s based on this specific set of technologies because this is the stack we use at Capellic. We’ve been using this configuration for about 6 months. We’ve been able to create and restart hundreds of ephemeral workspace environments during that time without any significant hitches. And while we haven’t made any changes to this configuration in several months, by no means does that mean it's perfect, but it is good enough to get the work done.

This guide is still relevant to you if you’re using GitLab or Github, Gitpod supports them as well. 

This guide is likely still helpful to you if you're not using Storybook or not using it with Emulsify. You’ll know better than anyone if you need to compile the theme and how. You’ll be able to take what we’ve done here and modify it for your needs.

And again, it is likely still very helpful if you’re using a hosting provider other than Pantheon as DDEV comes with provider configurations for Platform.sh and Acquia and there's also a one for Amezee.io (Lagoon).

This guide won’t be helpful if you’re using Lando. In fact, we use Lando for our local development environments but my experimentation with it in Gitpod resulted in me concluding that I’m going to wait until Lando 4 before trying again. And I will try again, largely due to the tight integration with Pantheon’s stack which DDEV, at this moment, and as far as I know, is lacking. If I’m wrong here, let me know!

Step 1: Gitpod account and project configuration

A: Create Gitpod account

Create an account on Gitpod if you don’t already have one. If you’re an agency managing several websites you may want to consider adding an Organization so that users attached to that organization are spending time against the agency account instead of their own.

B: Add project in Gitpod

Then add a project and, if you’ve added an organization, add that project to the organization.

C: Create Pantheon Terminus machine token and add it to Gitpod

Create a Pantheon Terminus machine token to use with Gitpod. Name it “Gitpod” or something similar so you know where the token is being used. Be sure you click the “I understand” button, not until then is the machine token functional. In Gitpod, add the machine token you just created as an environment variable. the name needs to be TERMINUS_MACHINE_TOKEN (to work with snippets provided later) and set the scope to */*.

Step 2: Install browser extension to enable launching Gitpod for Bitbucket

This step is optional, but it is slick and it improves our workflow. We create Bitbucket branches from Jira tickets and so it’s a natural extension of that flow to be able to click over to the branch in Bitbucket and then launch that branch from within Bitbucket. This Chrome browser extension makes this workflow frictionless, I recommend it highly. (You can also launch a branch in Gitpod via some slick URL hackery.)

The browser extension isn’t perfect, Gitpod gets confused if initiated from some Bitbucket contexts. If you run into trouble be sure you’re launching from the “Source” context.

More on this later.

Step 3: Create a new branch for main to add the config files for Gitpod

Following best practices, create a new branch from main for the configuration work you’re about to do.

You will add the configuration files in this step before attempting to launch Gitpod – it simply won’t work. You could add these configuration files locally but I don’t have this repo on my local machine so I’m going to edit directly in Bitbucket.

Here’s a screencast that covers this step:

A: Create file `.gitpod.yml`

Be sure you update the file so that you replace “THEME_NAME” with your theme name. Then commit the file your repo.

Here’s a high-level overview of the file.

image: This is the gitpod docker image file that we’re using. You haven’t yet added it to your repo, we’ll do that below.

tasks: There are three tasks defined. They are set up this way for efficiency. The “Build Theme” task doesn’t have a dependency on the “Build Application” task so they can run in parallel. The last task, “Open Browser” waits until “Build Theme” and “Build Application” are done and ends with launching the preview browser and delivering a one-time login link. 

vscode: This is a list of the VS Code extensions that we use. Modify as you see fit.

ports: Port 8080 is the most important port, that’s the one you’ll use to view the website running in Gitpod. Note that the “visibility” flag is set to “open”. This means that anybody in the world can visit that URL while your workspace is running and see what you’re working on. This is very handy for collaborating with colleagues.

image:
  file: .gitpod/.gitpod.dockerfile
tasks:
  - name: Build Application
    before: |
      ddev config global --instrumentation-opt-in=true --web-environment="TERMINUS_MACHINE_TOKEN=$TERMINUS_MACHINE_TOKEN"
    init: |
      ddev start -y
      ddev composer install
      ddev pull pantheon --skip-confirmation --skip-files
    command: gp sync-done build_application
  - name: Build Theme
    init: |
      cd web/themes/custom/THEME_NAME
      nvm use
      npm install
      npm run build-storybook -- -o storybook
      node -v
    command: gp sync-done build_theme
  - name: Open Browser
    command: |
      gp sync-await build_application
      gp sync-await build_theme
      ddev start -y
      ddev drush cr
      gp preview $(gp url 8080)
      ddev drush uli 
# VScode extensions 
vscode:
  extensions:
    # Twig language support
    - mblode.twig-language-2
    # YAML language support
    - redhat.vscode-yaml
    # PHP documentation block support
    - neilbrayfield.php-docblocker
    # PHP code intelligence
    - bmewburn.vscode-intelephense-client
    # Gherkin support for Behat test writing
    - alexkrechik.cucumberautocomplete
    # Drupal API support
    - andrewdavidblum.drupal-smart-snippets
    # Git Extension Pack including Git Lens
    - donjayamanne.git-extension-pack
ports:
  # Used by ddev - local db clients
  - port: 3306
    onOpen: ignore
  # Used by MailHog
  - port: 8027
    onOpen: ignore
 # Used by phpMyAdmin
  - port: 8036
    onOpen: ignore
  # Direct-connect ddev-webserver port that is the main port
  - port: 8080
    onOpen: ignore
    visibility: public
  # Ignore host https port
  - port: 8443
    onOpen: ignore 
 # xdebug port
  - port: 9003
    onOpen: ignore

B: Create file `.gitpod/.gitpod.dockerfile`

Using the Bitucket interface I create this file which will also create the `.gitpod` directory in the project root.

You’ll want to update the NODE_VERSION environment variable to be whichever you use for your website. Then commit the file to your repo.

Honestly, it would be great if this script consumed up the node version from the .nvmrc file in our theme, so if you know how to do that, let me know!

FROM drud/ddev-gitpod-base

ARG NODE_VERSION 
ENV NODE_VERSION=16.14 

ENV PNPM_HOME=/home/gitpod/.pnpm 
ENV PATH=/home/gitpod/.nvm/versions/node/v${NODE_VERSION}/bin:/home/gitpod/.yarn/bin:${PNPM_HOME}:$PATH 

RUN curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | PROFILE=/dev/null bash \    
    && bash -c ". .nvm/nvm.sh \
        && nvm install v${NODE_VERSION} \
        && nvm alias default v${NODE_VERSION} \
        && npm install -g typescript yarn pnpm node-gyp" \
    && echo ". ~/.nvm/nvm-lazy.sh"  >> /home/gitpod/.bashrc.d/50-node 
# above, we are adding the lazy nvm init to .bashrc, because one is executed on interactive shells, the other for non-interactive shells (e.g. plugin-host) 
COPY --chown=gitpod:gitpod .gitpod/nvm-lazy.sh /home/gitpod/.nvm/nvm-lazy.sh

C: Create file `.gitpod/nvm-lazy.sh`

Here are the contents of that file, no updates are needed. Then commit the file to your repo.

#!/bin/bash 

export NVM_DIR="$HOME/.nvm" 

node_versions=("$NVM_DIR"/versions/node/*) 

if (("${#node_versions[@]}" > 0)); then
  PATH="$PATH:${node_versions[$((${#node_versions[@]} - 1))]}/bin"
fi

if [ -s "$NVM_DIR/nvm.sh" ]; then
  # load the real nvm on first use 
  nvm() {
    # shellcheck disable=SC1090,SC1091 
    source "$NVM_DIR"/nvm.sh
    nvm "$@"
  }
fi 

if [ -s "$NVM_DIR/bash_completion" ]; then
  # shellcheck disable=SC1090,SC1091 
  source "$NVM_DIR"/bash_completion
fi

Step 4: Configure DDEV

DDEV has a “config” and “start” command that will generate configuration files that you will then need to commit to your repo, but we’re not using that workflow in this tutorial. Why? We need a functioning container for running DDEV in order to run those commands. We used to simply launch Gitpod to get the functioning container for configuring DDEV. But what we’ve found is that this gums out Gitpod’s image and task caching system. When restarting a workspace or even creating a new workspace from the same branch, we’ve seen all sorts of strange errors. 

The good news is that our manual configuration workflow only requires three changes to the Git repository.

Here’s a screencast that covers this step:

A. Create file `.ddev/config.yaml`

Using the Bitucket interface I create this file which will also create the `.ddev` directory in the project root.

You’ll want to update the name variable to match the application machine name on Pantheon.

If you’re not using Drupal 10 for this project you’ll want to update the type variable to match your version to drupal9, drupal8, etc.

Our sites are configured with their docroot in the “web” directory. I believe this is fairly standard on Pantheon these days so you won’t likely have to make any changes here.

Updated nodejs_version to match the version of Node you’ve defined in the .gitpod.dockerfile above.

name: PANTHEON_APPLICATION_NAME 
type: drupal10 
docroot: web 
nodejs_version: "16" 
php_version: "8.1" 
webserver_type: nginx-fpm 
router_http_port: "80" 
router_https_port: "443" 
xdebug_enabled: false 
additional_hostnames: [] 
additional_fqdns: [] 
database: 
  type: mariadb
  version: "10.3"
nfs_mount_enabled: false
mutagen_enabled: false
use_dns_when_possible: true
composer_version: "2" 
web_environment: []

B. Create file `.ddev/providers/pantheon.yaml`

While DDEV does support the ability to pull files from Pantheon, push files up to Pantheon, and push the database up to Pantheon, we only authenticate with Pantheon to pull the database. Therefore we have limited our Pantheon providers file to this activity. We use Stage File Proxy to pull the content files we need for our Gitpod workspace, it’s a lot more efficient.

You’ll want to update the “project” variable in the snippet below to match your application’s machine name as it is on Pantheon.
Note that I’m pulling from the “live” environment. You may want to adjust this based on your convention. Note that this provider script isn’t pulling from the Live database, it’s pulling the most recent backup of the Live database.

environment_variables:
  project: PANTHEON_APPLICATION_NAME.live 

auth_command:
  command: |
    set -eu -o pipefail
    if ! command -v drush >/dev/null ; then echo "Please make sure your project contains drush, ddev composer require drush/drush" && exit 1; fi
    if [ -z "${TERMINUS_MACHINE_TOKEN:-}" ]; then echo "Please make sure you have set TERMINUS_MACHINE_TOKEN in ~/.ddev/global_config.yaml" && exit 1; fi
    terminus auth:login --machine-token="${TERMINUS_MACHINE_TOKEN}" || ( echo "terminus auth login failed, check your TERMINUS_MACHINE_TOKEN" && exit 1 )
    terminus aliases 2>/dev/null

db_pull_command:
  command: |
    set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null
    terminus backup:get ${project} --element=db --to=db.sql.gz

C. Update file `web/sites/default/settings.php`

Open your existing settings.php and paste this code snippet at the bottom. No updates are needed here.

// Automatically generated include for settings managed by ddev. 
$ddev_settings = dirname(__FILE__) . '/settings.ddev.php';
if (getenv('IS_DDEV_PROJECT') == 'true' && is_readable($ddev_settings)) {
  require $ddev_settings;
}

Step 5: Launch Gitpod!

The Gitpod and DDEV configuration files are now in place and its time to launch Gitpod!

Here’s what that looks like:

You can launch this within Bitbucket or paste the Bitbucket URL into the Gitpod new workspace workflow.

Within Bitbucket, get into the branch context and click on the “Open” button with the Gitpod logo.

You’ll be ushered over to Gitpod to create a new workspace. You should verify that the URL in the first option contains your branch name. If not, you’ll want to get back to Bitbucket, get in the right context and then try again.

You could work on configuring a different code IDE here, but I recommend just going with the VS Code default and making a note to come back and explore other options after you’ve got the complete workflow nailed down.

Click the “Continue” button. You may not be required to click on Continue, it may be that Gitpod will just move you along automatically.

Step 6: Merge into the main branch, you’re done!

If all looks good and you’re full of joy, merge into the main branch. From there you can start using Gitpod for all your development needs.