Part I and part II of this series reviewed the tools that we can use in order to access the system under test — a browser-based a desktop or a mobile app. But accessing the application is not sufficient to be able to execute the tests. For that we will need a full testing environment that will help us run the tests.

The E2E Testing Environment

As described in the first part of the series, the E2E test is mimicking a user actions on the system at hand. It will usually work as follow:

• Perform a series of actions similar to what a user will do. Such as go to a specific url, www.google.com, enter some text “ [Meghan Markle](google.com/search?q=Meghan+Markle "Search for "Meghan Markle"")” into the input field and click the button labeled “Search”.

• Define the results that you expect: title will contain the search text and you will see 10 results.

• Run the test on your application with specific capabilities, such as the chrome browser on Android 8 device.

The description of the actions and the assertions are defined in test files called Test Specs. Their format depends on the specific tool you use. The expected results (assertions) are also defined there. The specific environment requirements are usually described in a configuration file. The results will also be extracted into a file (or on the console).

To make all of this happen, we need a testing environment. Here is what the testing environment will know to do:

• Find all the files where your tests are described

• Execute all the tests on the requested configurations

• Spit out the results in a format we can read.

So here is what we want it to look like:

Testing Environment Parts

Testing Environment Parts

Just like a car needs an engine, wheels and a steering wheel to make it move from one point to another, your testing environment need few parts, all connected, to run the tests:

Test Framework

The notion of whether an assertion result is good or bad is defined in the test spec. The test framework is reading the tests and translating them to browser commands. The responses received from the language bindings can also be asserted to understand if they are correct. Popular test frameworks (in JS world!) include Jasmine and Mocha with Chai for assertions.

Language Bindings

The language bindings are the part that is talking to your Selenium Server (or a Selenium like server) to execute the commands. The language bindings have no notion of good or evil. They can only interrogate the browser if an item exist or not, what is the displayed value etc. Is the fact that that an element with id #welcome is visible is a good or a bad thing? Language bindings would not know. They can only report on the element state, or perform an action.

Reporter

The reporter is collecting all the results from the assertions and outputs them in a friendly format.

Test Runner

Test runner is responsible for making everything play nicely together. It will gather all the test specs, send them to the test framework to start running them (one by one or in parallel). It will know how to trigger the tested environment and eventually it will activate the reporter for the results.

Testing Frameworks in NodeJS World

As the series is titled “The hitchhiker’s guide to E2E Testing in Javascript world”, we will focus on NodeJS based testing frameworks. Here is the list of the popular ones.

A specific attention should be given to Selenium Webdriver which is the Selenium official language bindings for Nodejs. It is included in the Selenium Github repository and was used as the basis for some other frameworks. But being an open source and standardized , additional language bindings emerged, dealing with different aspects (such as Node’s asynchronous nature) in different manners. (Testcafe is probably additional name worth mentioning that was omitted Above.

So which one is the right one for you? As with any design decision, there is no one decisive answer. It depends on your needs, on the company you work for, and on the tools you are familiar with. [spoiler alert: Webdriverio is my tool of preference] Let’s go over some of the considerations you may want to weigh into your decision:

Popularity

Popularity is a not just about having many likes and shares. In the Open Source community, popularity is directly translated to tangible benefits:

• More people using the software which means lot of bugs were already identified and fixed (or worked around).

• More people are likely to encounter an issue you are struggling with and already posted a github issue or answered a stack overflow question regarding the issue.

• A live community maybe up and kicking on slack or on Gitter to respond to the most noob or complex questions and guiding you on best practices.

A common measure for NodeJS packages’ popularity is the number of npm downloads. Check out this graph:

Comprehensibility

The speed at which you will be able to absorb the mental model of the testing library, the better your tests will be. Comprehensibility is achieved by having well organized and complete documentation, but also from the inherent complexity (or simplicity) of the tool itself.

The ability to find answers quickly within the documentation is a key for moving fast with the tool.

Synchronous Support

Node is asynchronous in nature. But tests are much more comprehensible when written in a synchronous manner. Almost all popular libraries have taken some approach in trying and hide the async complexity behind a sync-like syntax. The devil, as always, is in the details: does that work for errors as well? what if I need to include server requests?

Because the browser (or application) tests are also asynchronous in nature, a specific support that is worth noticing is for waits and retries. It will take a while until a page will render on screen and you will want your test to wait until the page is ready. This can be built into the library or you may need to manage waiting and retrying yourself.

Configuration Flexibility

As shown in the testing environment structure above, the configuration file is a major input to the tool. The tool flexibility is derived from the number of configuration options it has. Can you specify the wait time? the number of retries before it fails? What hooks are available to run your own code — before each session, or can you run a function before each spec file, or even each line of code? What parameters are available at each hook? Flexible tool will provide with lot of hooking and configuration options.

Frameworks Support

The testing framework may dictate a use of specific test language such as Mocha and provide assertions via Chai, and reporting with a specific report. Alternatively, it may provide you the flexibility to select your preferred stack. Using Jasmine with Allure reporting.

Mobile Support

If you plan to run tests also on mobile devices, it is worth checking upfront what are the capabilities provided by the tool, based on your needs. Supporting mobile device means first of all — integration with Appium to access the mobile device. But it also means access to mobile specific capabilities: mobile gestures (swipe, pinch), mobile actions (rotate, turn network on and off), file access, setting language and so on.

Why Webdriver.io?

As noted in the spoiler alert above, I ended up with Webdriver.io. Here is why:

• Best mobile support of all other tools. Easy integration with Appium and a set of mobile specific commands.

• Easy configuration with an excellent plug in systems that allows you to pick your testing framework of choice (Mocha, Jasmine or Cucumber)

• A comprehensive model, including the support for async backed by good documentation and a lively Gitter channel.

• Framework flexibility. I ended up with Cucumber, and will explain the reasons later on in the series.

Need someone to help you around the E2E testing world? Read thewhole series. If you are happy and joyful — you may clap!