Cross Platform and Visual Regression Testing

When movies are made, one of the most demanding roles on set is the script supervisor, also known as the continuity supervisor. The major objective of the continuity supervisor is to keep the continuity of the wardrobe, props, hair, set dressing, makeup and actors actions during the filming of each scene. This sounds like an impossible task for one person, and it almost always is. A lot of mistakes slip through the cracks and show up as the movie is edited together but generally movie studios don’t mind as they have a psychological trick up their sleeve.

Change blindness is a phenomenon we all experience in our lives. It occurs when large changes in a visual scene go unnoticed by ourselves. According to some amatuer ‘movie mistake spotters’, Apocolypse Now is the movie with the most mistakes, 558 to be exact. I don’t remember seeing a single one the first time I saw the film. And this is the crucial point, as humans our visual perception is not as good as we think it is. Change blindness is also not just confined to movies, but anything in our visual field - new haircuts, a change of furniture, new clothes and those amazing childhood games we used to play to try and spot the difference.

As it turns out, machines are far more accurate and efficient at detecting visual changes then humans. This is crucial when developing cross-platform applications. When working on the CSS of one platform it might change or break some parts of the CSS on another platform which can be difficult and time consuming to detect. This is why at Webscope we don’t just do automated acceptance, unit and integration tests, we also automate cross platform tests and test for visual regressions.

By automating tests for visual regressions, we let the machines do what they do best while letting our developers do what they do best - developing rather than manually testing any changes they have made. When we run these tests we can define what pages or parts of the user interface are tested as well as which browsers and devices we test on. Depending on the test, this can take up to 30 minutes to complete which is far quicker than testing every device and browser manually, not to mention more accurate.

Tests are written using webdriverCSS, run using mocha, and utilize Browserstack’s Automate API for taking the screenshots. They can be run daily or after every commit, against either the development copy or the production website. If a test fails, a visual difference has been detected, of which a screenshot demonstrating the difference is then provided. Most websites change, and that is why we go back to them, for new stories, pictures, videos and other updates. The visual regression tests we run account for these expected differences and only show the problematic differences we need to have detected.

It may help to think of a visual regression test as similar to a script supervisor on a film set, but one that never lets a mistake slip through. It is by running automated cross-platform and visual regression tests that we can work on different areas of an app or website without the fear of accidentally breaking the UI. And if anything unexpected does happen, we are quickly notified and can resolve the issue immediately. This increases the accuracy and the speed of development work as well as assuring the digital product is fully designed and compatible on all browsers and platforms.


More blog posts by Katie Graham

I’ve just landed back home after spending a few days in New Orleans for the annual North American DrupalCon. As usual, I mainly attended Business and DevOps sessions, as well as a number of sessions in a new track called Horizons which covers the edges of Drupal. Horizons is an interdisciplinary track that discusses and showcases the biggest challenges and greatest opportunities in Drupal.
Why we’ve switched from Vagrant to DockerWe’ve recently started moving away from using our Homestead based Vagrant setup when developing Laravel apps due to the fact Docker is lightweight, quicker and we’ve found that many projects we work on, require additional PHP extensions or server configuration which Docker simplifies the management of.
The two pairs of the Knox V2 Google Cardboard we’d ordered had arrived in the office in time for Webscope Labs, so we thought it would be fun to have a play and see what we could do in terms of Virtual Reality. We created a VR tour of our office, it was pretty disconcerting to remove the headset after seeing one part of the office and adjust to being in the area you’re actually in!