Atomic deployment with Deployer

Michael Walsh
By Michael WalshPublished 8 years ago

Deployer is a PHP based open source tool for performing atomic deployments of websites and web applications.

With the arrival of the much awaited release of Craft CMS 3 in April this year, which is heavily integrated with Composer, it was evident that we needed to update our approach for deploying our projects.

Our current approach essentially boiled down to running a git pull on the production server instances and whilst it wasn’t perfect it did serve us well. Granted a git pull does take a few seconds to complete and during which time the website’s files may be inconsistent, this was somewhat mitigated by our use of Varnish Cache, which is cleared after the deployment has succeeded, but even so there was still scope to improve things.

With Composer-based projects such as Craft CMS 3 and Laravel a “git pull” no longer suffices as we have to run a “composer install” to pull any new or updated dependencies into the vendor folder, and that’s a process that likely takes minutes rather than seconds to complete. Therefore we needed to look at an atomic deployment solution.

I will politely sidestep the argument around whether the vendor folder should in fact be committed into Git. I think that really depends on the project and for some of our larger projects maybe the case for that is greater but for our smaller projects any benefits seem very negligible in my opinion.

Atomic deployment takes its name from database terminology but essentially the idea is that either the deployment has executed completely or not executed at all, and a failed deployment does not leave your project in an inconsistent state. In practice this means a different sort of folder structure for your project on the web servers. Rather than just updating the currently running files, a brand new folder is created where the new files are placed, vendor folders are updated, any other tasks are completed and then once everything has successfully completed this new folder is symlinked to the folder your web server (Nginx, Apache, etc.) is running from. So regardless of however long it takes for a “composer install” to run the files that the web servers are using are always in a consistent state.

We wanted to find a solution that we could use across the board for our PHP projects, so from Craft CMS sites (version 2 and 3), Laravel web apps and even WordPress, Magento and any bespoke/legacy sites.

Of course there are plenty of software-as-a-service tools that fill this void and I’m sure they do an excellent job but the costs for these would soon become quite eye watering when you take into account a large number of projects.

Initially we had a look at a Node.js tool called ShipIt, this looked good but in the end we selected Deployer as it seemed to have more recipes for the sort of projects we wanted to deploy. One slight caveat to our production environments is that we use autoscaling at AWS so at the point a deployment is actually happening we need to query AWS to ascertain how many servers are running and their addresses. We already have code that accomplishes that both in PHP and Node.js, but it seemed clearer as to how that could be accomplished in Deployer than ShipIt, and from my investigations it seemed there was a more active community around Deployer than ShipIt.

So we set about getting Deployer integrated into a Craft CMS 3 project and a Laravel project, and on the whole this was a very smooth operation. The example recipes it provided were very useful and it didn’t take long until we had parallel deployments working. A parallel deployment means the deployment is executed simultaneously across all server instances, again a plus for our load-balanced production environments.

The only slight hiccup we found related to our Nginx config, which meant that PHP-FPM’s OpCache was not cleared when the new deploy folder was symlinked, with a quick change to the FastCGI config settings as described in this Server Fault post the issue was soon resolved and PHP-FPM’s OpCache was cleared with each deploy.

To help others get started with using Deployer, here are some example configurations for both Craft CMS 3 and Laravel projects:

Deployer recipe for Craft CMS 3 projects

<?php
namespace Deployer;
require 'recipe/common.php';


// Project name
set('application', 'enovate.co.uk');


// Project repository
set('repository', 'git@githosting.com:enovatedesign/project.git');


// Shared files/dirs between deploys
set('shared_files', [
    '.env'
]);
set('shared_dirs', [
    'storage'
]);


// Writable dirs by web server
set('writable_dirs', [
    'storage',
    'storage/runtime',
    'storage/logs',
    'storage/rebrand',
    'public/cpresources'
]);


// Set the worker process user
set('http_user', 'worker');


// Set the default deploy environment to production
set('default_stage', 'production');


// Disable multiplexing
set('ssh_multiplexing', false);


// Tasks


// Upload build assets
task('upload', function () {
    upload(__DIR__ . "/public/assets/", '{{release_path}}/public/assets/');
    //upload(__DIR__ . "/public/service-worker.js", '{{release_path}}/public/service-worker.js');
});


desc('Execute migrations');
task('craft:migrate', function () {
    run('{{release_path}}/craft migrate/up');
})->once();


// Hosts


// Production Server(s)
host('110.164.16.59', '110.164.16.34', '110.164.16.50')
    ->set('deploy_path', '/websites/{{application}}')
    ->set('branch', 'master')
    ->stage('production')
    ->user('someuser');


// Staging Server
host('192.168.16.59')
    ->set('deploy_path', '/websites/{{application}}')
    ->set('branch', 'develop')
    ->stage('staging')
    ->user('someuser');


// Group tasks


desc('Deploy your project');
task('deploy', [
    'deploy:info',
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'deploy:update_code',
    'upload', // Custom task to upload build assets
    'deploy:shared',
    'deploy:writable',
    'deploy:vendors',
    'deploy:clear_paths',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
    'success'
]);


// [Optional] Run migrations
after('deploy:vendors', 'craft:migrate');


// [Optional] If deploy fails automatically unlock
after('deploy:failed', 'deploy:unlock');


// Run with '--parallel'
// dep deploy --parallel

View the Gist on GitHub.

Deployer recipe for Laravel projects

<?php
namespace Deployer;
require 'recipe/common.php';


// Project name
set('application', 'enovate.co.uk');


// Project repository
set('repository', 'git@githosting.com:enovatedesign/project.git');


// Shared files/dirs between deploys
set('shared_files', [
    '.env'
]);
set('shared_dirs', [
    'storage'
]);


// Writable dirs by web server
set('writable_dirs', [
    'storage',
    'bootstrap/cache'
]);


// Set Laravel version
set('laravel_version', function () {
    $result = run('{{bin/php}} {{release_path}}/artisan --version');
    preg_match_all('/(\d+\.?)+/', $result, $matches);
    $version = $matches[0][0] ?? 5.5;
    return $version;
});


// Set the worker process user
set('http_user', 'worker');


// Set the default deploy environment to production
set('default_stage', 'production');


// Disable multiplexing
set('ssh_multiplexing', false);


// Helper Tasks


desc('Disable maintenance mode');
task('artisan:up', function () {
    $output = run('if [ -f {{deploy_path}}/current/artisan ]; then {{bin/php}} {{deploy_path}}/current/artisan up; fi');
    writeln('<info>' . $output . '</info>');
});


desc('Enable maintenance mode');
task('artisan:down', function () {
    $output = run('if [ -f {{deploy_path}}/current/artisan ]; then {{bin/php}} {{deploy_path}}/current/artisan down; fi');
    writeln('<info>' . $output . '</info>');
});


desc('Execute artisan migrate');
task('artisan:migrate', function () {
    run('{{bin/php}} {{release_path}}/artisan migrate --force');
})->once();


desc('Execute artisan migrate:fresh');
task('artisan:migrate:fresh', function () {
    run('{{bin/php}} {{release_path}}/artisan migrate:fresh --force');
})->once();


desc('Execute artisan migrate:rollback');
task('artisan:migrate:rollback', function () {
    $output = run('{{bin/php}} {{release_path}}/artisan migrate:rollback --force');
    writeln('<info>' . $output . '</info>');
})->once();


desc('Execute artisan migrate:status');
task('artisan:migrate:status', function () {
    $output = run('{{bin/php}} {{release_path}}/artisan migrate:status');
    writeln('<info>' . $output . '</info>');
})->once();


desc('Execute artisan db:seed');
task('artisan:db:seed', function () {
    $output = run('{{bin/php}} {{release_path}}/artisan db:seed --force');
    writeln('<info>' . $output . '</info>');
})->once();


desc('Execute artisan migrate:fresh --seed');
task('artisan:migrate:fresh:seed', function () {
    $output = run('{{bin/php}} {{release_path}}/artisan migrate:fresh --seed');
    writeln('<info>' . $output . '</info>');
})->once();


desc('Execute artisan cache:clear');
task('artisan:cache:clear', function () {
    run('{{bin/php}} {{release_path}}/artisan cache:clear');
});


desc('Execute artisan config:cache');
task('artisan:config:cache', function () {
    run('{{bin/php}} {{release_path}}/artisan config:cache');
});


desc('Execute artisan route:cache');
task('artisan:route:cache', function () {
    run('{{bin/php}} {{release_path}}/artisan route:cache');
});


desc('Execute artisan view:clear');
task('artisan:view:clear', function () {
    run('{{bin/php}} {{release_path}}/artisan view:clear');
});


desc('Execute artisan optimize');
task('artisan:optimize', function () {
    $deprecatedVersion = 5.5;
    $currentVersion = get('laravel_version');
    if (version_compare($currentVersion, $deprecatedVersion, '<')) {
        run('{{bin/php}} {{release_path}}/artisan optimize');
    }
});


desc('Execute artisan queue:restart');
task('artisan:queue:restart', function () {
    run('{{bin/php}} {{release_path}}/artisan queue:restart');
});


desc('Execute artisan storage:link');
task('artisan:storage:link', function () {
    $needsVersion = 5.3;
    $currentVersion = get('laravel_version');
    if (version_compare($currentVersion, $needsVersion, '>=')) {
        run('{{bin/php}} {{release_path}}/artisan storage:link');
    }
});


/**
 * Task deploy:public_disk support the public disk.
 * To run this task automatically, please add below line to your deploy.php file
 *
 *     before('deploy:symlink', 'deploy:public_disk');
 *
 * @see https://laravel.com/docs/master/filesystem#the-public-disk
 */


desc('Make symlink for public disk');
task('deploy:public_disk', function () {
    // Remove from source.
    run('if [ -d $(echo {{release_path}}/public/storage) ]; then rm -rf {{release_path}}/public/storage; fi');
    // Create shared dir if it does not exist.
    run('mkdir -p {{deploy_path}}/shared/storage/app/public');
    // Symlink shared dir to release dir
    run('{{bin/symlink}} {{deploy_path}}/shared/storage/app/public {{release_path}}/public/storage');
});


// Tasks


// Upload build assets
task('upload', function () {
    upload(__DIR__ . "/public/js/", '{{release_path}}/public/js/');
    upload(__DIR__ . "/public/css/", '{{release_path}}/public/css/');
    upload(__DIR__ . "/public/mix-manifest.json", '{{release_path}}/public/mix-manifest.json');
    //upload(__DIR__ . "/public/service-worker.js", '{{release_path}}/public/service-worker.js');
});


// Hosts


// Production Server(s)
host('110.164.16.59', '110.164.16.34', '110.164.16.50')
    ->set('deploy_path', '/sites/{{application}}')
    ->set('branch', 'master')
    ->stage('production')
    ->user('someuser');


// Development/Staging Server
// Note: Overrides Composer options to include development dependencies
host('192.168.16.59')
    ->set('deploy_path', '/sites/{{application}}')
    ->set('branch', 'develop')
    ->set('composer_options', '{{composer_action}} --verbose --prefer-dist --no-progress --no-interaction --optimize-autoloader')
    ->stage('staging')
    ->user('someuser');


// Group tasks


desc('Deploy your project');
task('deploy', [
    'deploy:info',
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'deploy:update_code',
    'upload',
    'deploy:shared',
    'deploy:vendors',
    'deploy:writable',
    'artisan:storage:link',
    'artisan:view:clear',
    'artisan:cache:clear',
    'artisan:config:cache',
    'artisan:optimize',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
    'success'
]);


// [Optional] Run migrations
after('deploy:vendors', 'artisan:migrate');


// [Optional] If deploy fails automatically unlock
after('deploy:failed', 'deploy:unlock');


// [Optional] Symlink the public disk.
//before('deploy:symlink', 'deploy:public_disk');


// Run with '--parallel'
// dep deploy --parallel

View the Gist on GitHub.

related posts

Distributed GitLab Runner Caching with MinIO

How to use MinIO as GitLab runner cache storage to help increase job concurrency

An introduction to structured data

In this blog post, I will be discussing what structured data is, how developers can implement it on their websites, and what effects using structured data can have to their sites and end users.

Vim: Demystifying the Beast

While being very powerful, Vim is notorious for it's steep learning curve and perceived user-unfriendliness. This article will attempt to clarify Vim's peculiarities and demonstrate it's efficiency.

Distributed GitLab Runner Caching with MinIO

How to use MinIO as GitLab runner cache storage to help increase job concurrency

An introduction to structured data

In this blog post, I will be discussing what structured data is, how developers can implement it on their websites, and what effects using structured data can have to their sites and end users.

View all posts

This website uses cookies. Find out more about cookies.