Unit testing is great, but it doesn’t show you the whole picture. Integration tests are also nice, because you know that your components will work together as expected, but it is still missing some pieces.
The fact is, that when you code is in production, your users still encounter bugs no matter how many unit and integration tests you have.
You are still worried that when a real user hits your app she might immediately find a new bug.
Your integration and unit tests are just not interacting with the app the same way people do.
That is why there exist end-to-end tests, also known as functional tests, scenarios tests or simply as web or browser tests when it comes to web apps.
Imagine what would be
- Testing your work just like a real person will interact with it
- Knowing that every component from end to end will work together nicely
How it will all work?
To make it a reality you will need to setup several components before even writing the tests.
You are going to run your browser tests in a real browser. They can work even when your testing machine does not have GUI, like a Linux server for example.
We are going to use Chrome. The full blown version that so many people love and use, not the open source variant Chromium. We want to get as close to the real users as possible.
Then you are going to use ChromeDriver. It is a component which is developed by the same people who create Chrome & Chromium.
On one hand, it can control Chrome just like a real user will do. On the other hand it provides a standard API to which you can connect from Node or any other platform.
This is the WebDriver API. It’s a standard API with available libraries for almost any language out there.
You are also going to use webdriverio to interact with ChromeDriver from NodeJS. It implements the WebDriver API and makes interacting with it a breeze.
When you want to run Chrome on a Linux machine without GUI, like a server, you will need one more component XVFB.
To run on Linux, Chrome (and any other GUI app) needs an X server. However, you don’t want a full blown X server just to run a few tests. You need something fast and light.
XVFB is such fast and light X server, which actually doesn’t display anything. Instead it writes all commands to a buffer. It is developed by the same people who create the full blown X server.
You are NOT going to use Selenium
Whenever you read something about browser tests on the Internet with a real browser it always uses Selenium. People don’t even think about it twice.
However, when you are using ChromeDriver, you don’t require Selenium at all. ChromeDriver has everything you need and supports the complete the WebDriver API.
Let’s install and configure everything you need.
XVFB
If you are going to run your tests on a Mac with OS X or on a Linux with GUI like Gnome, you don’t need XVFB and you can skip this part.
You only need it when your testing machine does not have a GUI.
To install XVFB on Debian or Ubuntu do the following
$ apt-get install xvfbOn CentOS or Red Hat you should
$ yum install XvfbOnce installed start it like this
$ Xvfb :99This command runs the fake X server and creates a display for your programs to connect to at :99
Installing Chrome
You are not going to use Chromium but a full version of Chrome, just like your users.
If you are on a Mac, it is easy to install it from Chrome’s website.
Let’s see how it is done on Linux even without a GUI. You have to add the appropriate repository and then just install it as usual.
On Debian or Ubuntu
$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
$ sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
$ apt-get update
$ apt-get install google-chrome-stableOn CentOS or Red Hat
First create a file /etc/yum.repos.d/google-chrome.repo
[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/$basearch
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pubThen install it with just one line
$ yum install google-chrome-stableChromeDriver
ChromeDriver will not only control Chrome but it will also start and stop it.
One instance of ChromeDriver can control multiple instances of Chrome simultaneously. It also supports multiple clients to connect to it and each of them to run its own independent tests.
On OS X you can install it from here http://chromedriver.storage.googleapis.com/index.html?path=2.20/
On Linux you can install it like this
$ wget http://chromedriver.storage.googleapis.com/2.20/chromedriver_linux64.zip
$ unzip chromedriver_linux64.zipThen run the following to start it
$ export DISPLAY=:99
$ ./chromedriverThe exported variable instructs ChromeDriver which display to use to run Chrome. This is the fake display provided by XVFB. You don’t need to export this variable when you are not using XVFB.
Your Project
You have everything in place to run your tests. Let’s have a look what your project might look like.
tests.js
app.js
views/
funny.jade
sad.jade
package.jsonThe project is very simple, just three files. app.js is the entry point of your application. tests.js contains all of your tests and views houses a few of your templates.
This structure is so simple because I want to you to focus on the web tests. To learn how a real Node & Express project can be set read my article on Best practices for Express app structure
Let’s install the project dependencies.
$ npm install express jade --save
$ npm install mocha should webdriverio --save-dev- express is the web framework that we are going to use to build the app.
- jade will provide us with a decent templating language.
- mocha will help us structure and run our tests.
- should is a great library for testing whether result is what you expect.
- webdriverio will help you control Chrome like a real user.
Let’s have a look at how each of the project files looks like.
Here is app.js
var express = require('express')
, app = express()
app.engine('jade', require('jade').__express)
app.set('view engine', 'jade')
app.get('/funny', function(req, res) {
res.render('funny')
})
app.get('/sad', function(req, res) {
res.render('sad')
})
app.listen(3000, function() {
console.log('Listening on port 3000...')
})This is a simple Express web app. It defines two routes /funny and /sad, which just render the two templates.
Let’s see how each template looks like. This is funny.jade
doctype
html
head
title Funny Title
body
div(id='funny-id') This is funny
a(href='/sad', class='sad-link') Go to SadLet’s now look at sad.jade
doctype
html
head
title Sad Title
body
div(id='sad-id') This is sad
a(href='/funny', class='funny-link') Go to FunnyThe two templates are almost identical with only a few differences.
Now let’s run the web app. You need it running to be able to access it from a real browser like when running the web tests.
$ node app.jsWriting tests
Everything is ready, so let’s create your first tests.
var should = require('should')
, webdriverio = require('webdriverio')
, options = {
host: '127.0.0.1',
port: 9515,
path: '/',
desiredCapabilities: {
browserName: 'chrome',
chromeOptions: {
"args": [
"window-size=1366,768",
"no-proxy-server",
"no-default-browser-check",
"no-first-run",
"disable-boot-animation",
"disable-default-apps",
"disable-extensions",
"disable-translate",
],
},
},
}
describe('Testing a few pages', function() {
before(function() {
this.browser = webdriverio.remote(options).init()
return this.browser
})
after(function() {
return this.browser.end()
})
it('funny page', function() {
return this.browser
.url('http://127.0.0.1:3000/funny')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Funny Title')
})
.element('#funny-id', function(err, element) {
should.not.exist(err)
should.exist(element)
})
.element('#sad-id', function(err, element) {
should.exist(err)
should.not.exist(element)
})
.click('a.sad-link')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Sad Title')
})
})
it('sad page', function() {
return this.browser
.url('http://127.0.0.1:3000/sad')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Sad Title')
})
.element('#funny-id', function(err, element) {
should.exist(err)
should.not.exist(element)
})
.element('#sad-id', function(err, element) {
should.not.exist(err)
should.exist(element)
})
.click('a.funny-link')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Funny Title')
})
})
})There are a lot of things going on here. First, the options object contains the configuration for our browser.
By default webdriver expects Selenium which listens to http://127.0.0.1:4444/wd/hub. However ChromeDriver listens to http://127.0.0.1:9515/ and this is part of the configuration above.
You may also notice that there are many Chrome specific options. Their purpose is to make Chrome behaviour more predictable and faster, especially when starting.
Once configured you have to start the browser. This is the slowest operation, even with the parameters from above, and you want to do it inly once in a file.
The place for such operations is in the before function.
before(function() {
this.browser = webdriverio.remote(options).init()
})Once Chrome is started, it will not be closed automatically at the end of your tests. What’s more, when you run your tests with XVFB you will not even notice it but it will still eat your memory.
That is why you are using the method after to close it after all your tests have
finished executing.
Finally, there are the two tests. Each of them begins on one of the two pages in the app, then checks whether an element specific to the page exists, then clicks a link to go to the other page and at the end checks the title to see whether it arrived at the right page.
return this.browser
.url('http://127.0.0.1:3000/funny')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Funny Title')
})
.element('#funny-id', function(err, element) {
should.not.exist(err)
should.exist(element)
})
.element('#sad-id', function(err, element) {
should.exist(err)
should.not.exist(element)
})
.click('a.sad-link')
.getTitle(function(err, title) {
should.not.exist(err)
title.should.eql('Sad Title')
})You can see that we are using the url, getTitle, click and element methods.
- url loads a new page in the browser.
- getTitle reads the title in the browser.
- element looks for a specific element on the page.
- click clicks on an element.
I am using standard CSS selectors to select the elements I want, but there is also XPath if you are in that sort of thing. It is actually more powerful than CSS.
Underneath they all use the WebDriver API. It has many more capabilities like filling forms, selecting items from drop down menus, moving the mouse. You can even inject some JavaScript, for example to scroll the page or check some JavaScript state.
You can check the documentation at webdriveroi.
With the way the tests above are written you should not worry about any asynchronious actions.
As you can see we return the result from all the methods chained on this.browser.
The returned result is a Promise. Mocha will wait for this promise to be resolved
before moving to the next test.
To run your tests, all you have to do is
$ ./node_modules/.bin/mocha tests.jsGetting Feedback
There are mainly two ways to get feedback when you are running browser tests.
When you are running on a machine with with GUI you can just see how the page looks. When you are running on healdless machine you can still take a screenshot.
this.browser.saveScreenshot('./screenshot.png')The other way to get some useful feedback is to look at the page source code.
this.browser.getSource(function(err, source) {
console.log(source)
})Problems that you may encounter
There are a few problems which you might encounter when creating web tests. First,
sometimes the browser is not closing properly after the tests end. Even when you
call end in the after method.
This can be fixed by restarting ChromeDriver. It will clear any window which is left and eating your memory.
Another problem that I’ve seen is the stability of the tests. Even when the tests are correct and the browser seem to work correctly, sometimes it might not succeed clicking on an element, or clicking might not produce the expected JavaScript event.
There are many more similar problems that might happen, so be careful.
It is not all roses
You have just seen how wonderful web tests are. Unfortunately, they have a dark side, too.
The first problem is that they are slow, significantly slower than any other kind of tests.
The second problem is that their feedback is poor, just some element missing on a page, a screenshot and a source code. It may sound a lot, but it is not.
It is often hard to understand what really went wrong.
To make it worse, all your components, databases and services are taking part. Nothing is mocked and any of them can produce a problem. This makes it even harder to identify what went wrong.
Next
Go on and create your first end-to-end tests. They might take a little bit more time to write, but the satisfaction of knowing that this is how the user will experience your site is great.
However, don’t make too many of them. They are hard to debug, and very slow. You should really on unit and integration tests to catch 99% of your problems and only then on web tests.
Once you have seen how it works with Chrome, you can check with other browsers. For example, you can try running Firefox with Selenium, or even Chrome on iPhone or Android.
The last thing I would suggest is to try to automate your browser testing with Jenkins.
Other articles that you may like
