Gulp and AngularJS, a love story. OR: The old wheel was terrible, check out my new wheel! | NetEngine

Gulp and AngularJS, a love story. OR: The old wheel was terrible, check out my new wheel!

Dan Thursday, 20 March 2014

tldr: steal our gulpfile.js. It’s at the bottom.

We build a lot of AngularJS apps that sit in front of JSON APIs. Sometimes they’re deployed together, sometimes they’re not. I think there’s a benefit to treating the HTML + JS as just the first of many potential clients of your API, so I want to decouple its development / deployment as much as is possible. Above and beyond the benefits for treating this as an independent client which may be wrapped up as a mobile / desktop client, it also offers protection for the knowledge being gained by my team.

No one working on the client will throw away knowledge if we switch the JSON API implementation from Rails to Grape, Koa, HAPI, Express, Dynamo, Martini, etc, or if the API is outside the scope of our project.

We’ve been building apps this way for about 6 months, and we’ve learned a lot. I hope to find the time to discuss more of these lessons-learned, but today I’m on about our new build chain. Our existing Grunt build process was getting complicated, and it was getting slow. Before I go further, I’d like to make something clear.

Grunt is awesome. I’ll continue to use it where appropriate. For example, it’s still building our fonts.

That said, I’ve read all the hype around Gulp, and I was curious, so I tested the water. I found it warm and inviting, so last week I jumped in. On most of our apps, I want users to download an index.html that looks something like this.

<!doctype html>
<html ng-app="clientApp" ng-controller="ApplicationCtrl">
  <head>
    <meta charset="utf-8">
    <meta name="language" content="english">
    <title ng-bind="pageTitle + ' | clientApp'">clientApp</title>
    <link rel="stylesheet" href="/styles/app-5016b1a8.css">
  </head>
  <body>
    <header ui-view="header"></header>
    <main ui-view="content"></main>
    <footer ui-view="footer"></footer>

    <!--[if lt IE 9]><script src="/scripts/shims-hba3kca8.js"><![endif]-->
    <script src="/scripts/vendor-fdf7ec99.js"></script>
    <script src="/scripts/app-54577144.js"></script>
    <script src="/scripts/templates-3f9f5b17.js"></script>
  </body>
</html>

When it comes to optimising the impression of speed, I’d probably revise this to include initial content and to give the browser a chance to fetch images before processing my JS bundles, but it ticks some important boxes.

The index.html is the only file we’re serving that shouldn’t be cached as long as possible, and it’s the only file we intend to invalidate on the CDN, so everything else gets a SHA fingerprint.

If we were including a lot of vendor CSS, there might be another stylesheet up in the head, but that isn’t the case today. You’ll note the 4 JS bundles. Not all browsers get our JS shims (es5-shim, json3, Modernizr, give or take) so that’s one file. Vendor JS tends to be the slowest to build, and changes the least often, so we split it out from our own JS.

Angular fetches any HTML template that isn’t already in the $templateCache service, so we pre-populate the cache with a fingerprinted JS file – otherwise our users would keep downloading the same templates over and over, and we’d have to worry about invalidating mulptiple files.

To insert these fingerprinted assets into index.html, we use gulp-inject.

<!doctype html>
<html ng-app="clientApp" ng-controller="ApplicationCtrl">
  <head>
    <meta charset="utf-8">
    <meta name="language" content="english">
    <title ng-bind="pageTitle + ' | clientApp'">clientApp</title>
    <!-- inject:app-style:css --><!-- endinject -->
  </head>
  <body>
    <header ui-view="header"></header>
    <main ui-view="content"></main>
    <footer ui-view="footer"></footer>

    <!--[if lt IE 9]><!-- inject:shim:js --><!-- endinject --><![endif]-->
    <!-- inject:vendor:js --><!-- endinject -->
    <!-- inject:app:js --><!-- endinject -->
    <!-- inject:templates:js --><!-- endinject -->
  </body>
</html>

To achieve this, our gulpfile has ..

function indexHtml(cb) {
  plugins.util.log('Rebuilding index.html');

  function inject(glob, path, tag) {
    return plugins.inject(
      gulp.src(glob, {
        cwd: path
      }), {
        starttag: '<!-- inject:' + tag + ':{{ext}} -->'
      }
    );
  }

  function buildIndex(path, cb) {
    gulp.src('app/index.html')
      .pipe(inject('./styles/app*.css', path, 'app-style'))
      .pipe(inject('./scripts/shim*.js', path, 'shim'))
      .pipe(inject('./scripts/vendor*.js', path, 'vendor'))
      .pipe(inject('./scripts/app*.js', path, 'app'))
      .pipe(inject('./scripts/templates*.js', path, 'templates'))
      .pipe(gulp.dest(path))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  }

  buildIndex(expressRoot, cb || function(){});
  buildIndex(publicDir, function(){});
}

A couple of takeaways from this function.

  1. I’m using functions, rather than gulp tasks. Orchestrator, the task runner for Gulp, has some cool features, but there’s nothing to stop you combining it with vanilla javascript.
  2. The plugins object comes from the gulp-load-plugins module. It’s handy.
  3. I have 2 destinations in mind for the post-inject index.html.

One is for deployment / testing the minified versions. The other is for development (unminifed, with source-maps and livereload.) For us, this generally means writing to a .gitignored tmp folder served by express on port 4000, while also writing the production-ready files to a rails public folder, to be served up by nginx.

Everything we’re doing in this build chain simultaneously streams to 2 destinations. This makes people more likely to test the production distribution, and eliminates any ‘build’ step on deployment, because the tested, minified output is always checked-in.

Starting at the top of the injected files, let’s look at where our CSS is coming from.

function clean(relativePath, cb) {
  plugins.util.log('Cleaning: ' + plugins.util.colors.blue(relativePath));

  gulp
    .src([(publicDir + relativePath), (expressRoot + relativePath)], {read: false})
    .pipe(plugins.rimraf({force: true}))
    .on('end', cb || function() {});
}

function styles(cb) {
  clean('/styles/app*.css', function() {
    plugins.util.log('Rebuilding application styles');

    gulp.src('app/styles/app.scss')
      .pipe(plugins.plumber())
      .pipe(plugins.sass({
        includePaths: ['app/bower_components'],
        sourceComments: 'map'
      }))
      .pipe(gulp.dest(expressRoot + '/styles'))
      .pipe(plugins.minifyCss())
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(publicDir + '/styles'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}
  1. We kick off most tasks by cleaning up their previous output.
  2. We’re using gulp-sass -> node-sass -> lib-sass -> sassC. We have to give up a couple of SCSS features from the latest version, and there’s still a couple of gotchas before the ruby and C implementations produce identical output, but the speed is incredible. Gulp + SassC + Livereload gives designers superpowers.
  3. There’s a couple of good helpers around to make streaming play nice, like plumber and streamify. Size is there just to keep an eye on what we’re asking users to download.
  4. Development doesn’t get a SHA fingerprint, nor is it minified.

Our vendor and shim JS bundles are treated identically in both development and production.

function vendor(cb) {
  clean('/scripts/vendor*.js', function() {
    plugins.util.log('Rebuilding vendor JS bundle');

    gulp.src(require('./app/scripts/vendor'))
      .pipe(plugins.concat('vendor.js'))
      .pipe(plugins.streamify(plugins.uglify({ mangle: false })))
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(expressRoot + '/scripts'))
      .pipe(gulp.dest(publicDir + '/scripts'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}
  1. We don’t try to use a module loader like Browserify (which is awesome) on 3rd party code. Like it or not, it’s still the wild west out there, and most libraries expect to find their dependencies attached to the global object.
  2. Not all angular libraries survive minification (ie. they rely on the variable-name syntax for dependency injection.) { mangle: false } is your friend.

Now for one of the big wins. Browserify / Watchify for our application JS.

function scripts(cb) {
  var bundler = watchify('./app/scripts/index.js');

  function rebundle() {
    clean('/scripts/app*.js', function() {
      plugins.util.log('Rebuilding application JS bundle');

      return bundler.bundle({ debug: true })
        .pipe(source('app.js'))
        .pipe(gulp.dest(expressRoot + '/scripts'))
        .pipe(plugins.streamify(plugins.uglify({ mangle: false })))
        .pipe(plugins.streamify(plugins.size({ showFiles: true })))
        .pipe(gulp.dest(publicDir + '/scripts'))
        .on('end', cb || function() {})
        .on('error', plugins.util.log);
    });
  }

  bundler.on('update', rebundle);
  bundler.on('error', plugins.util.log);
  rebundle();
}
  1. We’re not using Gulp’s watch, it (and caching of partial bundles) is built into watchify.
  2. That 'source’ function is coming from 'vinyl-source-stream’, and lets gulp consume the text-stream API from watchify.

Finally, we bundle the templates.

function templates(cb) {
  clean('/scripts/templates*.js', function() {
    plugins.util.log('Rebuilding templates');

    gulp.src('app/views/**/*.html')
      .pipe(plugins.angularTemplatecache({
        root:   'views/',
        module: 'clientApp'
      }))
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(gulp.dest(expressRoot + '/scripts'))
      .pipe(gulp.dest(publicDir + '/scripts'))
      .on('end', cb)
      .on('error', plugins.util.log);
  });
}
  1. We could probably do some HTML minification to ditch whitespace before pushing them into JS strings.
  2. Obviously this won’t be suitable if you’re generating these HTML templates dynamically on your API. This goes against my rant about modularity of client versus API, but I certainly imagine cases where this would be desirable.

The rest of the details of our new set up should be clear when you see the package.json and gulpfile.js in their entirety. Vendor.js and shims.js export arrays of files to be concatenated.

package.json

{
  "name": "client",
  "description": "AngularJS client",
  "repository": "http://github.com/net-engine/client-app",
  "version": "0.0.0",
  "dependencies": {},
  "devDependencies": {
    "connect-livereload": "^0.3.2",
    "express": "^3.4.8",
    "grunt": "~0.4.1",
    "grunt-webfont": "~0.2.2",
    "gulp": "^3.5.2",
    "gulp-angular-templatecache": "^1.1.1",
    "gulp-concat": "^2.1.7",
    "gulp-imagemin": "^0.1.5",
    "gulp-inject": "^0.4.1",
    "gulp-livereload": "^1.2.0",
    "gulp-load-plugins": "~0.3.0",
    "gulp-minify-css": "^0.3.0",
    "gulp-plumber": "^0.5.6",
    "gulp-rev": "^0.3.0",
    "gulp-sass": "^0.7.1",
    "gulp-size": "~0.1.2",
    "gulp-streamify": "0.0.4",
    "gulp-uglify": "^0.2.1",
    "gulp-util": "^2.2.14",
    "node-sass": "~0.7.0",
    "tiny-lr": "0.0.5",
    "vinyl-source-stream": "^0.1.1",
    "gulp-rimraf": "0.0.9",
    "watchify": "^0.6.3"
  },
  "engines": {
    "node": ">=0.10.0"
  }
}

gulpfile.js

'use strict';

var connectLr         = require('connect-livereload'),
    express           = require('express'),
    app               = express(),
    expressPort       = 4000,
    expressRoot       = require('path').resolve('./.tmp'),
    gulp              = require('gulp'),
    liveReloadPort    = 35729,
    lrServer          = require('tiny-lr')(),
    permitIndexReload = true,
    plugins           = require('gulp-load-plugins')(),
    publicDir         = require('path').resolve('../server/public'),
    source            = require('vinyl-source-stream'),
    watchify          = require('watchify');

function startExpress() {
  app.use(connectLr());
  app.use(express.static(expressRoot));
  app.listen(expressPort);
}

function startLiveReload() {
  lrServer.listen(liveReloadPort, function(err) {
    if (err) {
      return console.log(err);
    }
  });
}

function notifyLivereload(fileName) {
  if (fileName !== 'index.html' || permitIndexReload) {
    lrServer.changed({ body: { files: [fileName] } });

    if (fileName === 'index.html') {
      permitIndexReload = false;
      setTimeout(function() { permitIndexReload = true; }, 5000);
    }
  }
}

function clean(relativePath, cb) {
  plugins.util.log('Cleaning: ' + plugins.util.colors.blue(relativePath));

  gulp
    .src([(publicDir + relativePath), (expressRoot + relativePath)], {read: false})
    .pipe(plugins.rimraf({force: true}))
    .on('end', cb || function() {});
}

function scripts(cb) {
  var bundler = watchify('./app/scripts/index.js');

  function rebundle() {
    clean('/scripts/app*.js', function() {
      plugins.util.log('Rebuilding application JS bundle');

      return bundler.bundle({ debug: true })
        .pipe(source('app.js'))
        .pipe(gulp.dest(expressRoot + '/scripts'))
        .pipe(plugins.streamify(plugins.uglify({ mangle: false })))
        .pipe(plugins.streamify(plugins.size({ showFiles: true })))
        .pipe(gulp.dest(publicDir + '/scripts'))
        .on('end', cb || function() {})
        .on('error', plugins.util.log);
    });
  }

  bundler.on('update', rebundle);
  bundler.on('error', plugins.util.log);
  rebundle();
}

function styles(cb) {
  clean('/styles/app*.css', function() {
    plugins.util.log('Rebuilding application styles');

    gulp.src('app/styles/app.scss')
      .pipe(plugins.plumber())
      .pipe(plugins.sass({
        includePaths: ['app/bower_components'],
        sourceComments: 'map'
      }))
      .pipe(gulp.dest(expressRoot + '/styles'))
      .pipe(plugins.minifyCss())
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(publicDir + '/styles'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function templates(cb) {
  clean('/scripts/templates*.js', function() {
    plugins.util.log('Rebuilding templates');

    gulp.src('app/views/**/*.html')
      .pipe(plugins.angularTemplatecache({
        root:   'views/',
        module: 'clientApp'
      }))
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(gulp.dest(expressRoot + '/scripts'))
      .pipe(gulp.dest(publicDir + '/scripts'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function shims(cb) {
  clean('/scripts/shims*.js', function() {
    plugins.util.log('Rebuilding shims JS bundle');

    gulp.src(require('./app/scripts/shims'))
      .pipe(plugins.concat('shims.js'))
      .pipe(plugins.streamify(plugins.uglify({ mangle: false })))
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(expressRoot + '/scripts'))
      .pipe(gulp.dest(publicDir + '/scripts'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function vendor(cb) {
  clean('/scripts/vendor*.js', function() {
    plugins.util.log('Rebuilding vendor JS bundle');

    gulp.src(require('./app/scripts/vendor'))
      .pipe(plugins.concat('vendor.js'))
      .pipe(plugins.streamify(plugins.uglify({ mangle: false })))
      .pipe(plugins.streamify(plugins.rev()))
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(expressRoot + '/scripts'))
      .pipe(gulp.dest(publicDir + '/scripts'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function images(cb) {
  clean('/images', function() {
    plugins.util.log('Minifying images');

    gulp.src('app/images/**/*.*')
      .pipe(plugins.imagemin())
      .pipe(plugins.size({ showFiles: true }))
      .pipe(gulp.dest(expressRoot + '/images'))
      .pipe(gulp.dest(publicDir + '/images'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function fonts(cb) {
  clean('/styles/fonts/icons', function() {
    plugins.util.log('Copying fonts');

    gulp.src('app/styles/fonts/icons/*.*')
      .pipe(gulp.dest(publicDir + '/styles/fonts/icons'))
      .pipe(gulp.dest(expressRoot + '/styles/fonts/icons'))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  });
}

function indexHtml(cb) {
  plugins.util.log('Rebuilding index.html');

  function inject(glob, path, tag) {
    return plugins.inject(
      gulp.src(glob, {
        cwd: path
      }), {
        starttag: '<!-- inject:' + tag + ':{{ext}} -->'
      }
    );
  }

  function buildIndex(path, cb) {
    gulp.src('app/index.html')
      .pipe(inject('./styles/app*.css', path, 'app-style'))
      .pipe(inject('./scripts/shim*.js', path, 'shim'))
      .pipe(inject('./scripts/vendor*.js', path, 'vendor'))
      .pipe(inject('./scripts/app*.js', path, 'app'))
      .pipe(inject('./scripts/templates*.js', path, 'templates'))
      .pipe(gulp.dest(path))
      .on('end', cb || function() {})
      .on('error', plugins.util.log);
  }

  buildIndex(expressRoot, cb || function(){});
  buildIndex(publicDir, function(){});
}

gulp.task('vendor', function () {
  vendor(indexHtml);
});

gulp.task('default', function () {
  startExpress();
  startLiveReload();
  fonts();
  images();
  styles(indexHtml);
  templates(indexHtml);
  shims(indexHtml);
  scripts(function() {
    indexHtml(function() {
      notifyLivereload('index.html');
    });
  });

  gulp.watch('app/scripts/shims.js', function() {
    shims(function() {
      indexHtml(function() {
        notifyLivereload('index.html');
      });
    });
  });

  gulp.watch(['app/styles/**/*', '!app/styles/fonts/**/*'], function() {
    styles(function() {
      indexHtml(function() {
        notifyLivereload('styles/app.css');
      });
    });
  });

  gulp.watch('app/styles/fonts/**/*', function() {
    fonts(function() {
      styles(function() {
        indexHtml(function() {
          notifyLivereload('styles/app.css');
        });
      });
    });
  });

  gulp.watch('app/images/**/*', function() {
    images(function() {
      indexHtml(function() {
        notifyLivereload('index.html');
      });
    });
  });

  gulp.watch('app/views/**/*', function() {
    templates(function() {
      indexHtml(function() {
        notifyLivereload('index.html');
      });
    });
  });

  gulp.watch('app/index.html', function() {
    indexHtml(function() {
      notifyLivereload('index.html');
    });
  });
});

This is still early days in our Gulp adventure, but it’s already been a productivty boon. I’d love to hear about any suggestions for improvements, or experiences people have trying this out, so please don’t hesitate to hit me up on dan@netengine.com.au

Further reading, and some articles on Gulp I’ve enjoyed.

Finally, should it prove useful to understand the relative folder structure used above, here’s an example project layout.

.
├── Gruntfile.js
├── .tmp
├── app
│   ├── 404.html
│   ├── images
│   ├── index.html
│   ├── scripts
│   │   ├── controllers
│   │   │   └── index.js
│   │   ├── directives
│   │   │   └── index.js
│   │   ├── filters
│   │   │   └── index.js
│   │   ├── helpers
│   │   │   └── index.js
│   │   ├── index.js
│   │   ├── routes
│   │   │   └── index.js
│   │   ├── services
│   │   │   └── index.js
│   │   ├── shims.js
│   │   └── vendor.js
│   ├── styles
│   │   ├── app.scss
│   │   ├── fonts
│   │   ├── layout
│   │   └── modules
│   └── views
│       ├── directives
│       ├── layouts
│       │   ├── _footer.html
│       │   ├── _header.html
│       │   └── _search.html
│       └── users
│           └── sign_in.html
├── bower.json
├── grunt
│   └── webfont.js
├── gulpfile.js
└── package.json
comments powered by Disqus

Stay up to date by subscribing to our mailing list.