Limitless file uploading to Amazon S3 with Node & Express

Last time we’ve looked at uploading files locally.

Uploading files locally is useful, but it has way too many limits for most applications.

You have to think about back ups of your files, then you have to implement the back up strategy. In addition, your disk have a finite space and from time to time you have to add more disks.

Handling many files from different disks is also a complex task. You shouldn’t forget that uploading and serving large files also requires a lot of memory and processing power.

Thankfully, Amazon has a service which does all this for us and does it at a very reasonable price.

How It Works

The user selects a file to upload.
You sent with AJAX the file name and type to your server.
Your server contacts AWS securely and generates a temporary signed URL.
Your server sends back the URL to the browser.
The browser uploads the file directly to S3 by using the temporary signed URL.

Requirements

Before you start you need to have access to an account at Amazon Web Services (AWS).

Bucket Configuration

S3 stores your data in buckets. Let’s create a specific bucket for this task and name it ‘images_upload’. You can do this from your AWS console.

Then select the ‘Properties’ tab in your bucket. Click on “Permissions” and then “Add CORS Configuration” and use the following configuration.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
  </CORSRule>
</CORSConfiguration>

It configures your bucket to allow the usual HTTP requests sent from any computer. However, each request still needs to be authenticated or it will fail.

Now that your bucket is ready, you are ready to write some code.

Starting Point

Let’s assume that we have the following application structure.

project/
  controllers/
    index.js
    pictures.js
  models/
  public/
    uploads/
  views/
    index.html
  app.js

It’s a classical MVC structure. Controllers contain our request handlers, models handle our data and business logic, and views define all page templates. Public contains all publicly accessible static files like images, front-end JavaScript and stylesheets.

In our case we are going to work with only two files controllers/index.js and views/index.html

Next, you need the AWS SDK. It is just one line to install with NPM.

$ npm install aws-sdk --save

Signing Securely S3 Uploads

With your current configuration S3 will not allow uploading of files unless they are authenticated.

For this purpose we need to use a temporary signed URL for each file upload to S3.

Let’s see how your controller at controller/index.js should look like.

  var express = require('express')
    , router = express.Router()
    , aws = require('aws-sdk')

  router.get('/', function(req, res) {
    res.render('index')
  })

  var AWS_ACCESS_KEY = 'your_AWS_access_key'
  var AWS_SECRET_KEY = 'your_AWS_secret_key'
  var S3_BUCKET = 'images_upload'

  router.get('/sign', function(req, res) {
    aws.config.update({accessKeyId: AWS_ACCESS_KEY, secretAccessKey: AWS_SECRET_KEY});

    var s3 = new aws.S3()
    var options = {
      Bucket: S3_BUCKET,
      Key: req.query.file_name,
      Expires: 60,
      ContentType: req.query.file_type,
      ACL: 'public-read'
    }

    s3.getSignedUrl('putObject', options, function(err, data){
      if(err) return res.send('Error with S3')

      res.json({
        signed_request: data,
        url: 'https://s3.amazonaws.com/' + S3_BUCKET + '/' + req.query.file_name
      })
    })
  })

  module.exports = router

Let’s look what we have hear.

There are two request handlers. The first renders our page, which we will look at in a minute.

The second handler uses the AWS SDK to request that S3 generates a temporary URL where you can upload your file. This URL is valid only for 60 seconds, as per the Expires option.

Once the request is successfully returned you send back the generated URL.

Let’s now see how our page at views/index.html looks and works

<!DOCTYPE html>
<html>
  <head>
    <title>Limitless Image Upload</title>
  </head>
  <body>
    <h1>Welcome!</h1>
    <br>Please select an image
    <input type="file" id="image">
    <br>
    <img id="preview">

    <script>
    function upload(file, signed_request, url, done) {
      var xhr = new XMLHttpRequest()
      xhr.open("PUT", signed_request)
      xhr.setRequestHeader('x-amz-acl', 'public-read')
      xhr.onload = function() {
        if (xhr.status === 200) {
          done()
        }
      }

      xhr.send(file)
    }

    function sign_request(file, done) {
      var xhr = new XMLHttpRequest()
      xhr.open("GET", "/sign?file_name=" + file.name + "&file_type=" + file.type)

      xhr.onreadystatechange = function() {
        if(xhr.readyState === 4 && xhr.status === 200) {
          var response = JSON.parse(xhr.responseText)
          done(response)
        }
      }

      xhr.send()
    }

    document.getElementById("image").onchange = function() {
      var file = document.getElementById("image").files[0]
      if (!file) return

      sign_request(file, function(response) {
        upload(file, response.signed_request, response.url, function() {
          document.getElementById("preview").src = response.url
        })
      })
    }
    </script>
  </body>
</html>

It’s quite a lot of code but don’t worry, it is very simple.

First at the beginning of the page we have a simple file field which the user can use to choose a file.

The more interesting part is in the JavaScript section at the bottom of the page. First, we listen to the change event of the file field with onchange which is fired when the user selects a file.

When fired we use the sign_request function to get a signed temporary URL where the browser can upload the file. Then again using AJAX you upload the file with the upload function to the temporary URL.

Once the file is uploaded we display it to the user.

This is everything that you have to do. As you can see the file never reaches your server. Only tiny little bit of data is actually handle by it. Almost the entire workload is handled by S3.

Don’t forget, S3 is great not only for uploading and storing files, but it is also very capable of serving files. It can easily server files of all sizes to millions of users simultaneously.

As you can see it is very simple to upload files directly to S3. However, sometimes you need first to process a file before uploading to a server. This is what you can try to do next.

Other articles that you may like