A lot of time, effort, and hours go into creating a product that users love, and the quality of the software should reflect that. With Lyearn being a fast-paced, tech-first startup, test automation plays a very important role.
We started off with Selenium to automate our tests. Once you have architecture and process in place, it’s difficult to switch tools. But we still took a leap and went ahead with Playwright. This is the story of our transition and how switching to Playwright made our lives easy!
1. The Setup Tour
We’ve heard this before: “Selenium automates browsers. That’s it!” And that’s actually it. One needs much more than that for an E2E test framework.
As we can clearly see from the above image, Selenium as a framework is not a complete and one-stop test automation solution. It requires multiple third-party libraries, language bindings, and APIs to be usable.
On the other end, everything comes out of the box with Playwright. So, all we had to do was npm init playwright and this single-handedly eliminated the tedious initial setup and creation of various components. Needless to say, an all-in-one solution always works more seamlessly when compared to multiple third-party solutions combined.
From the above image, one can understand how drastically we managed to save lines of code, time, and effort. That too with improved results.
❗Our challenges: Complex setup, high initial cost, steep learning curve
💡Playwright’s Solution: Everything comes out of the box hence no complex setup or learning curve
2. Finding your perfect match — The Selector Confusion
And we do mean a perfect match for the web element — careful use of words here because choosing an inappropriate locator can lead to the breakdown of the entire test suite.
There are so many ways to identify the same element, but there isn’t really a good guide on which selector to use. When and if you make a typo, there is no error indication at all. Heard of ID, Name, Link text, Partial Link text, CSS, or XPath? And if you ever went by the XPath way, you would be a genealogy expert by encountering child, parent, descendant, ancestor, preceding sibling and whatnot.
Ideally, our code should resemble how users interact with our web page. Reading the test should reflect the experience of visual/mouse users.
With Selenium, there is no out-of-the-box feature that can help us identify a locator.
Playwright provides its own Test Generator which generates tests from scratch without even leaving your code editor (VS Code).
Test Generator uses built-in Playwright locators created using Testing Library API that gives us the most resilient and readable locators. Some of them are getByRole, getByText, getByLabel and getByPlaceholder.
We generally want our locators to be:
- Reliable
- Immune to failure
- Unique
But user-facing attributes and script readability are hardly taken into consideration. Playwright recommends we use getByRole locator which searches through the accessibility tree and hence enables us to perform Accessibility Testing hand in hand. It is the closest way to how users and assistive technology perceive the page. Thus we get early feedback if the ARIA guidelines are correctly followed. Our locators now have high readability and UI accessibility.
Not just that, we can record the entire test without leaving the code editor. It saved us the hassle of opening developer tools and finding XPath. Thus, we were able to save authoring efforts and focus more on optimising the code.
❗Our challenges: Time-consuming locator finding strategies, no reliable locators
💡 Playwright’s Solution: Test Generator (with high readability and UI accessibility)
3. Traitor Selector
“My locator identifies the correct element in DOM but it doesn’t always work when I run tests.” 😢
You worked so hard to find the best match but eventually, it didn’t work.
The problem lies with how Selenium interacts with an element. If an element is not visible, we see an error which is why we use waits in Selenium — Implicit, Explicit, or Fluent. Again, when to use which waits depends on the use case.
These waits are entirely eliminated in Playwright. Each and every action in Playwright has multiple Actionability checks. It auto-waits till all relevant checks pass. In case of failure, we see TimeoutError.
Get Lyearn Inc.’s stories in your inbox
Join Medium for free to get updates from this writer.
An example of Selenium click vs Playwright click:
❗Our challenges: Locator failures, implementing various waits
💡 Playwright’s Solution: Auto-waits
4. Error Horror
Stuck somewhere? Here are the options we use to identify the issues in our code when working with Selenium: logging utility, screenshots, session recording, adding breakpoints, and analysing variables.
Error Timeline: Seeing the bug in action
What if there was a way to see what happened before, after, and at the time of error — all on this in a timeline format?
Playwright has its in-built Trace Viewer which is a one-stop solution for analysing results and debugging tests. Also, our automation script doesn’t interact with the webpage as we do. This is why Trace viewer also shows us where exactly Playwright clicked, typed, cleared, etc.
Testers helping Developers
Imagine how helpful it would be for devs to fix a bug if testers provided them with the exact root cause!
Instead of simply providing steps to reproduce, Playwright enables us to see console logs and network requests that were made in each step. Without manually checking the bug, one can instantly provide all the issue details and the turnaround time would drastically decrease.
One tab fits all
Wouldn’t it be helpful if we could see script actions — snapshots — source code — logs all at the same time?
Tired of switching between test reports and source code? See the source code for your entire test. Very helpful to see the previous and next steps and run through the overall code.
❗Our challenges: Complex debugging, creating separate modules for logging, screenshot, and reporting.
💡Playwright’s Solution: Trace Viewer
5. Visual Comparison
Automation tests cover the functional aspect of our application. With the increasing variety of devices, browsers, and screen resolutions, it is necessary to ensure nothing breaks visually.
We earlier used Selenium with Percy to achieve this. It involved the following steps:
- Installing Percy for Java with Selenium
- Importing the Percy Selenium library
- Creating a new Percy object with a WebDriver instance as a parameter
- Calling and configuring Percy snapshots
- Analysing results on Percy
All these steps were replaced by just one command in Playwright: await expect(page).toHaveScreenshot().
Playwright Test uses the pixelmatch library that supports features like anti-aliased pixels detection and perceptual colour difference metrics. It can be used in any environment (Node or browsers).
We can easily tweak settings like image difference sensitivity or masking certain parts of a page by passing respective locators in the above function. Test results can be analysed directly from Playwright’s test report with differences highlighted as shown below.
❗Our challenge: Separate module for visual tests with third-party dependency
💡Playwright’s Solution: Out of the box pixel perfect tests
Conclusion
We were able to see a major improvement in our Automation Metrics after adopting Playwright. For us, this meant an overall productivity increase of ~40%!
Both Selenium and Playwright are open-source tools. One may assume both are free to use. But that’s not the case. The Total Cost of Ownership is quite high when we work with Selenium.
QA team cannot contribute to product improvement if they are spending hours writing each test. They are not innovating if they spend time fixing flaky tests. They are not planning the next feature release if they are spending time on root cause analysis from their reports.
Playwright is the youngest among the best functioning tools in the Testing world. We are heavily benefited by adopting it and it’s worth a try!
