In the JavaScript Galaxy

No code snippets here! But if you are overwhelmed by the E2E testing various tools and no longer sure who is who and what is what — this one is for you. Towel is optional.

A little while ago we decided to add E2E tests to our Ionic app. I embraced the task heartily. I have a thing for testing. Running your tests and waiting for the red or green line to appear can give you an adrenaline rush similar to that of watching a spinning roulette.

If the project was a plain web-only-angular-cli project, this guide would have probably never be written. But an Ionic project, based on the Ionic cli which does not have E2E tests wired in, has made the whole journey a more exciting one.

Soon enough I found myself drowning in terms like selenium-webdriver, selenium standalone server, Chromedriver, Geckodriver, Appium, xcuitest and many more. Examples were written in Java and c# and Ruby. I know JavaScript, do I need to learn Java to write tests for Ionic app? I was looking for something that will make some sense in all this mess, but none was to find. Sure — everything is out there, but with different timing and different versions, I could not find THE one guide that will lead me through everything. So, more research and here is a go on writing this guide. Belts on!

What is E2E Testing?

The purpose of end-to-end (shortened as E2E) testing is to test (ah!) your system as close as possible to the way the user will interact with it. E2E testing is aimed in testing all of your system. If your system has a UI side, that will of course include testing the UI by imitating a set of actions that are similar to the actions the user will do.

In unit and integration tests, a part of your code is taken and is wrapped by a test case. The testing framework, e.g. Karma or Jest, is running both the wrapping code and the tested code.

In E2E (or more accurately — user testing), the testing framework is responsible for dispatching the test specs to the system under test. The test framework will not run the application code and it is the tester’s responsibility to make sure the tested code is running: i.e. building a test app or firing an app server that will serve the website. The test framework is automating A USER.

Unit vs E2E testing

What does an E2E test look like? It is a set of commands that simulate the user actions:

• Find an element by some attribute: e.g. find a button by className
• Perform an action on the element: e.g. click a button.
• Find an element that responded to the action and get its status or value.
• Evaluate the status or value against an expectation.
• Report the result.

E2E Test Framework

The user actions that are being tested are written in test spec files. The role of the E2E testing framework is to translate the spec files to the actions on the the tested platform and query the platform for the elements that are being evaluated. Eventually, it will report all of the results to a testing results report. The other part of the test framework is the configuration file that gives additional information to the testing framework, such as what platform is being tested and where to find the test specs.

Test Framework architecture

Selenium Architecture

Web testing world is being dominated by the Selenium architecture. Let’s go on a time travel back to 2004. This is the beginning of the second browser war (the first one was in the 90s between Microsoft and Netscape). And although a major player, Chrome, is not yet there, it is clear that web is the thing to develop for.

In Chicago, a company called Thoughtworks is developing an internal tool to automate browsers to perform web testing. The tool is first used internally and later on is being open sourced. The purpose of the tool is the automation of web browsers, with testing being the major use of it (others are performing repetitive tasks or scraping information).

At the time, Mercury (later acquired by HP) tools were a major player in testing automation. Selenium was named because its creator casually said that element selenium is a cure for mercury poisoning.

Selenium Webdriver

The tool has evolved multiple phases, with people from various companies (such as Google), and later became Selenium RC (Remote Control) and eventually Selenium Webdriver, a.k.a Selenium 2.0. The support for running simultaneously against multiple platforms was also developed and named Selenium Grid. The long Selenium’s history and changing architectural considerations is still somewhat reflected in the SeleniumHQ website that is a reminder to what websites looked like a decade ago.

But a major outcome of this effort is a set of Standard APIs, called Webdriver APIs. A Selenium server receives a set of HTTP calls and is executing them on the browser(s) under test. This standard is now under w3c and is still evolving.

Webdriver APIs

The APIs are built as REST APIs with data that is relevant to the requested action. The other part of the Webdriver APIs spec is a definition of capabilities: A set of features that are requested in order to spin the correct platform to be tested upon.

Language Bindings

The standardization of the APIs made it relatively simple to build translators that will read testing specs in multiple languages and will translate them to those APIs. Those are called language bindings. The official Selenium Webdriver project has bindings for Java, Ruby, Python, .net and NodeJS.

Selenium language bindings

But the language bindings are not limited to the official one. Being an open source project, additional language bindings emerged, dealing with different aspects (such as node’s asynchronous nature) in different manners. Some of those tools in the nodejs area include:

• Nightwatch
• Testcafe
• Protractor which is built on top of the original Selenium Webdriver with specific support for Angular
• Webdriver.io which replaced a deprecated project called webdriverjs
• wd.js

(Did I miss your favorite nodejs Selenium language bindings? Scroll all the way down and leave a comment).

Although some language bindings use similar conventions, such as the use of a jquery-like $ to find an element, the specific syntax that will be used in your test specs is derived by the The language binding tool that you are using. E.g, in webdriver.io (version 4), your tests can look like this:

browser.run();
browser.url('/');
browser.element('.input-email').setValue('user@example.com');
browser.$('.input-password').setValue('123456');
browser.element(‘#login’).click()

Selenium Standalone Server

So, in order to test on a browser, we need to set a language binding tool that will translate specs into APIs, and a selenium server that will receive the webdriver API requests and will dispatch the requests to the right browser.

But there is also a shortcut: certain browsers (read: Firefox, Chrome and lately Safari), have the Selenium Server “embedded” within them so they can be interacted with directly. That means, that in order to test chrome you can simply install Chromedriver locally and send the requests to the default port 9515.

Beyond the Desktop

The Selenium architecture was originally meant for testing desktop browsers. But, hey, it’s 2017 and web technologies are everywhere:

Platforms matrix

We have mobile and desktop running browser based and native applications. And one of the most important things is hybrid applications that use web technologies but are wrapped in native. For mobile — this is mainly provided by Phonegap / Cordova, and for desktops by Electron. Can we use the Selenium architecture to test those as well? (spoiler: yes). On a side note: Since we are focusing on web, we will ignore native apps for desktop and mobile, and focus on browser and hybrid apps.

Thanks to the fact that the Selenium Webdriver API have evolved as standard for browser automation / testing, more tools could evolve leveraging the standard to develop tools for testing hybrid applications.

Selenium for different platforms

So for desktop browsers we can use the Selenium standalone server or a direct driver. Appium is currently leading the race for becoming the standard for mobile app testing and Atom has developed Spectron to enable testing on their Electron platform.

The Mobile Experience

Appium is dated back to 2012, and has been re-written 3 times in 3 different languages. It was originally developed to support iOS testing automation but then was extended to support Android and recently also Windows. It is currently the only test framework that is cross mobile platform.

Appium is built on the idea that testing native apps shouldn’t require including an SDK or recompiling your app. And that you should be able to use your preferred test practices, frameworks, and tools. Appium is an open source project and has made design and tool decisions to encourage a vibrant contributing community.

Appium provides a selenium standalone like server that receives the webdriver APIs requests from your favorite testing framework and negotiates them with the mobile native platform so they can be executed on the mobile device.

Appium for Mobile platforms

When Android is tested, Appium will call Android Debug Bridge (adb) commands and Android Studio UIAutomator. When iOS is tested, the Xcode command line tools and XCUITest is used to act on the application.

Appium can be used to test mobile web, hybrid apps and native apps on multiple platforms. That is a big promise, but the actual experience might be a little bumpy…

Other tools besides Appium include Google’s Espresso and Robotium, both support only Android.

Reaching The Clouds

Let’s finish our E2E Galaxy travel by discussing the cloud services. There are 2 reasons you might want to consider using a cloud service:

• Easy setup — as with any cloud service, a lot of the hard wiring of things is done by someone else and all you need to do is register and use (uh, and pay).
• Devices & OS variety — you might have 2, 3, 4 different types of devices available to you in your office. Cloud services provide you with a wide range of devices and OS versions installed on them, so you can test your software against a much larger combination of devices that will imitate your real users.

E2E testing cloud services

The good news is that connecting existing local tests to cloud service is relatively easy. Instead of pointing your tests to call your local HTTP server (selenium standalone, browser or Appium), You just need to point the Webdriver API server to the provider’s HTTP endpoint and specify your key in the required capabilities, and your tests will (attempt to) run on the cloud. The lesser news is that theory and practice do not always overlap.

The 2 leading cloud services for testing are Saucelabs and BrowserStack. Saucelabs provides browser based testing for variety of desktop, mobile and tablet platforms. With the recent acquisition of TestObject it also provides manual and automated native and hybrid mobile testing. BrowserStack is other dominant player but is currently for browser testing on a large set of devices.

Other names include bitbar, Kobiton.

Reaching our Destination

Made it all the way down here? Kudos. I hope you learned something new and things are slightly better sorted out now. Like, clap, share or leave a comment telling me where I was wrong.