Skip to main content

Fancy FTP Deployment with Grunt

I recently dove into Grunt.js at work for automating our build process and I haven’t looked back. It’s an awesome tool with a plug-in for just about anything. I expected the usual would be there like JavaScript minification and concatenation, but I was surprised at a few others that I found, one being for FTP file deployment. Just shows how popular and community supported Grunt is.

There are a few FTP plug-ins available for Grunt. I didn’t do an analysis of all of them but ran across grunt-ftp-push which seemed to do what I needed so I decided to try it out. A simple ftp-push setup to upload an entire project via FTP could look like this:

grunt.initConfig({
  ftp_push: {
    all: {
      options: {
        host: 'example.com',
        port: 21,
        dest: '/project/path/',
        username:'user',
        password:'pass'
      },
      expand: true,
      cwd: 'dist',
      src: ['**/*', '!**/*.zip']
    }
  }
});

Some details here: I opted to put the username and password in the main config rather than using an .ftpauth file. The example on the Github page uses the file-array format for selecting the source files. I didn’t need that level of expression and I need to be able to easily modify the config dynamically later on so I switched to the compact method. One of the “Coming Soon” features is the ability to list files for exclusion in the upload. While such an ability would probably be a bit cleaner, this is technically already possible by using ‘!’ at the beginning of a pattern to negate it from the result set.

Now, a typical project at my job consists of standard web client code (HTML/CSS/JS) as well as some bigger type files, images, audio and sometimes video. The server we deploy to is off-site and our upstream is limited to around 10Mbps. The video and audio files are the largest and rarely change, whereas the code files are the most likely to change between deployments. So, it would be nice to have a separate ability to upload just code folders. Let’s create a second target in our ftp_push config:

grunt.initConfig({
  ftp_push: {
    options: {
      host: 'example.com',
      port: 21,
      dest: '/project/path/',
      username: 'user',
      password: 'pass'
    }.
    all: {
      expand: true,
      cwd: 'dist',
      src: ['**/*', '!**/*.zip/**']
    },
    minimal: {
      expand: true,
      cwd:  'dist',
      src: [
        '*',
        'css/*',
        'js/*'
        '!**/*.zip/**'
      ]
    }
  }
});

Noticed I moved the options object out of the ‘all’ target. Both targets will share the same options so this saves us from defining them twice. The first src item is for getting just the root level files, this is where our html files can be found. Next grab the css and js folders but keep skipping any zip files. We can now choose between either upload target with ‘grunt ftp_push:all’ or ‘grunt ftp_push:minimal’.

Now, what if an audio file changes and we want to upload the change but we still prefer to skip uploading video? We could go ahead and create yet another target for this specific case but this may eventually lead to a huge config list. What if instead we just offer an option to specify a folder for ftp deployment? Let’s define some task aliases:

grunt.registerTask(
  'ftp_all',
  'Build and deploy everything',
  ['default', 'ftp_push:all']
);

grunt.registerTask(
  'ftp_minimal',
  'Build but only deploy main project code',
  ['default', 'ftp_push:minimal']
);

These two new task aliases are primarily for joining the build task “default” with the FTP deployment tasks. Next we’ll write up a custom task to allow for manual file/folder deployment as mentioned above:

grunt.registerTask(
  'ftp',
  'Specify files or folders for FTP upload after build process.\n' +
  'Ex: grunt ftp:”js/*”:”css/*” ',
  function() {
    if (this.args.length > 0){
      grunt.config.set('ftp_push.all.src', (['!**/*.zip']).concat(this.args));
    }
    grunt.task.run('ftp_all');
  }
);

This will allow the user to specify folders or files for FTP deployment using Grunt’s glob patterns. I modify the config setting for ftp_push:all dynamically. The initial setting from the config is replaced via grunt.config.set(). This doesn’t get saved to the Gruntfile and no other tasks call ftp_push so this seemed like a pretty simple method as opposed to creating a third target. If no folder is specified, ftp_push:all is not modified and we simply default to uploading all the files.

I eventually want to try out the rather new plug-in grunt-diff-deploy, which would basically render all I talked about above useless. The diff-deploy is designed such that only files that have changed since the last FTP upload are transferred. In reading the code, it appears this is handled by computing a hash of every file and then uploading this file/hash list as it’s own file along with the specified files. When an FTP upload is later initiated by the user, the file/hash list is downloaded, hashes are computed again for local files and then the hashes are compared to identify the files that have changed and need to be uploaded. By the time I ran across this plug-in I had already started down the ftp-push road so I decided to finish up there. Also, ftp-diff is rather new (just a couple days old on Github when I first saw it) so it quite possibly has some rough edges that need to be fixed. There could be other issues too, perhaps computing the hash on 10’s of MB’s of video takes an inordinate amount of time. I don’t want to assume though and pass it up, I’ll likely try it out soon as it sounds like the Holy Grail of FTP deployment.

Comments

Olli Aapro said…
Lemme know how it ends up w/ u, I had some trouble since the task wouldnt run w/o the your_target pairs. When left empty, loading hashes was only state I got :)
XHR said…
You're referring to diff-deploy? I keep getting connection errors, even on a stable FTP on the local LAN. It loads the hashes and gets quite a few transfers complete but ultimately fails on one (not always the same) file. I think ultimately that the underlying JSFTP library needs a re-try file transfer option. Note also I manually upgraded JSFTP for diff-deploy to the latest release in order to get as far as I did.

Popular posts from this blog

Accessing other HTTP servers on Cloud 9 IDE

If you're using Cloud 9 to do development, you'll quickly realize that only ports 8080 through 8082 are available to the outside world from your development box. This is generally not an issue as you can set your application to bind to the $PORT environment variable when in development mode. However, there are sometimes other servers that we want to make use of that host on different default ports.

I recently had to setup a Neo4j server which defaults the admin interface of port 7474. Unfortunately, I could not access the admin interface even through the IDE based web browser window. So, what to do? I could change the default server settings so that it runs on a different port. However, the app I'm working on with a team has 7474 hard-coded and I currently don't feel like writing a local only work-around.

After some searching, I ran across a neat Linux tool called socat. This allows us to easily forward one port to another. After a quick install via apt-get, I ran the …

Moving to Babel 6 on the Server

Cross post from my employer's development blog: http://rootinc.github.io/2016/03/14/babel-6-server/

Decided it was time to upgrade my server-side code to run on Babel 6. Below is a synopsis of all the issues I ran into and resolved while upgrading my 0.11.3 SailsJS server to run with Babel 6 transpilation.

The upgrade to Babel 6 itself is easily achieved in Sails by upgrading the `sails-hook-babel` package.

### Missing preset
`couldn't find preset "stage-0" relative to directory`
Just because a preset is on the official Babel preset page, doesn’t mean that Babel comes with it. Simple fix by installing the [package][1] from npm. Read more [here][2].
[1]: https://www.npmjs.com/package/babel-preset-stage-0
[2]: http://jamesknelson.com/the-six-things-you-need-to-know-about-babel-6/

### Need strict mode everywhere
`Block-scoped declarations (let, const, function, class) not yet supported outside strict mode`
I didn’t have to worry about this before, and I don’t feel like …