Visual Regression Testing: The Best Tool (With the Worst Name)
Quick, I’ll give you a dollar if you can spot all the differences in the two images above! Better yet, I’ll give you a hundred dollars if you can spot all the differences in one minute. For every second it takes over one minute, I’ll deduct a dollar. And if you miss anything, that’ll cost your team two hundred bucks.
Does that pressure sound familiar?
“We want to make the text in the copyright statement a little bit bigger. It shouldn’t affect anything but the footer. Quick, make the change, double-check the site, then publish it!”
If you’re a “move fast and break things” kind of organization, you’ll make the change, trust that your CSS is well-structured, and have a quick look at the site. You might miss that the “mouse text” class got improperly re-used only in the hamburger menu at the tablet breakpoint, but don’t worry — your Executive Vice President will find that problem two weeks later when they discover that the navigation is broken on their iPad Pro. Whoops!
If you’re a “belt and suspenders” kind of organization, you’ll prepare the change, and then you’ll kick off a round of Visual QA. Your QA team has lots of automated tools at their disposal, but the tools they use only test to see if items are present or missing on a page, or that the items do the right thing when clicked. The only way to see if lines are wrapping awkwardly due to an unexpected text size change is to have a tester look at every page of the site. And every time they find something that might not be right, they’ll raise an “observation” to be discussed with design at the next standup. You deploy the text size change with no errors… a month from now.
There’s a much better way to catch visual changes
I’ve fed both versions of the image to a computer and instructed it to highlight all the differences in magenta. Did you catch all of the changes?
- The missing desk lamp?
- The addition of text on the closest blank page?
- All the color changes, including the blouse that changed from beige to a light pink?
Did you catch the fact that the entire second image has been shifted one pixel to the left, which will be completely undetectable to a human… until the user shifts to a dark theme, and suddenly the image has an unexpected white hairline all along one side?
A visual regression testing tool catches all those changes in one step, and makes them easy to spot.
Here’s how we use this tool at Think Company
First, we use a tool called Backstop to take pictures of entire web pages, and then compare those pictures to a different set of pictures. Backstop provides a “cool scrubber thingy” to show you how those snapshots are different. Here’s YouTube user Garris Shipon demonstrating how you can “scrub” across a page to make changes crystal-clear:
Backstop can turn a week-long hunt for visual differences into a ten-minute “push the button” exercise. No magenta? No changes. No changes? No problem — SHIP THAT SITE.
Second, we make visual testing a part of our development workflow at the component level. Here’s the set of tools we’re using right now — Jest for writing tests, and Puppeteer for making screenshots — so that every pull request compares every component on every commit.
Using visual regression as a part of development means that “Whoops, I accidentally made a visual change seventy-five pages away, in a breakpoint that shouldn’t exist” is no longer something we worry about. Those (inevitable!) problems surface right away, while the developer is working, and they’re surfaced right in front of the person who can do something about it… instead of waiting two weeks for an observation to come from the QA team.
We’d be happy to show you how we do it
Using visual regression both as a part of your QA process — and as a part of your continuous integration flow — reduces stress and errors, makes Pull Requests easier to review, and (dare we say it?) puts a lot of the fun back in hammering on the front-end. We offer workshops and hands-on clinics to show you how we do it, and to teach your team to customize these tools to match your flow. Let us know if you want to learn more!
Originally posted on the Think Blog by John Young.
