Monitoring as a testing approach at Trustpilot
We’re not afraid to run tests on production
Before I worked at Trustpilot I had no idea how micro-services across multiple development teams can be organized. I also didn’t know how testing the product can be executed during constant independent releases in production. But then I started to work in Trustpilot’s Product Quality team, which is not responsible for testing the product itself, but for organizing the testing process and for the quality of that testing.
So how exactly does the monitoring testing approach work? What challenges do we face and what advantages can I see with this way of testing?
This article will dive into how the monitoring testing approach works here at Trustpilot, where we have 17 product teams, 600+ micro-services, more than 200 releases on production per week and only one Product Quality team consisting of just 5 people.
Responsibilities
A common way to do testing, not utilized at Trustpilot, is the regression testing approach, in that case developers are responsible for writing code, and testers are responsible for testing that code, reporting bugs, and testing again. This approach works fine, but it divides developers and testers whilst still expecting them to build a good product together.
The regression testing approach simply doesn’t work when there are hundreds of micro-services with independent releases every day. So instead of having divided teams testing the product, the monitoring testing approach was adopted (meaning no testers would be present in any of the product teams). In this case, the responsibility of the Product Quality team isn’t to test the whole product before each release, but rather to make sure that each product team has the knowledge and opportunities they need. This means tools, guidelines, and relevant data to test their context on a certain level. The obvious advantage here is that there is no need to have a huge testing team, and developers can focus on the quality of the code since they are now solely responsible for that.
Maintenance and alerting challenges
More tests = more maintenance work, we all know that. But the idea of the monitoring approach is to only run relevant and valuable tests. Since each product team is responsible for maintaining and testing their own context they’re also responsible to only keep relevant tests, as they’ll get alerts for tests that fail. The challenge here for the Product Quality team is to help the product teams organize their test collections in a way where they only get an alert for their context and when action from them is required (i.e. update the test or fix the bug).
This leads us to another challenge: each team should only test within their context and the integrations connected to their context.
How do we tackle the challenge of alerts:
- Reusable subtests and snippets to notify and not alert* a team when a test is failing due to another team context.
- Alert is triggered if a test fails at least 2 times in a row: we are using the “run again on failure” feature.
- Alerts/notifications pipeline provided by our awesome SRE teams triggers alerts based on the severity of the tests.
- We encourage our teams to delete tests that are no longer useful and don’t bring value.
How do we tackle the challenge of maintenance:
- Tests overview project (we’ll have a separate article covering this topic in the near future).
- Code review for all tests that are being created.
- Quarterly organized Product Quality Hour when we share best practices and encourage teams to share their challenges and best practices between each other.
What helps us organize the monitoring testing process in Trustpilot
Guidelines with live samples
We need to constantly keep the monitoring testing process up-to-date. We’ve set this up this with some tests that can easily be copy-pasted. We’ve worked to make the updating of the process as useful, helpful, and short as possible.
Tests review
At some point, we realised that even though the product teams are responsible for testing their context, the quality of the tests will never be their priority. But it should be ours. So once per week, we review all new tests that have been created. This process helps developers to not leave the tests unfinished, and it helps us to be aware of their challenges while developing test automation. It also helps us share best practices with other teams.
Tests overview
This was a separate project that started as a 20-percent** project (separate article covering this topic is to be published in the near future), but now we find this tool very helpful. We can see what tests are using which API, subtests (reusable test steps, reusable API requests), as well as how many tests and tests execution each team has. It also helps us check and search for the existing tests between different contexts.
Naming convention
This came out when we started our tests overview project. We decided to use the name of the tests to keep the most relevant information about the test, which is the severity and type of the test. It helps to maintain tests and provides more information to the failure alert.
Personal touch
We’re trying to talk to our teams, to listen to the specifics of their context, as well as the challenges they have in order to best help them. Our best practices are very flexible. We consider our developers our customers and end-users, and we’re trying to help them, not to make decisions for them.
Our goal is to provide our product teams with all the relevant information they need and to help them organize testing processes by themselves.
*At Trustpilot each product team has three alerting channels (P1, P2 and P3) and one notification channel. Alert means that an action is required, while notification simply makes the team aware of something that has happened.
**In Trustpilot, 20 percent of the working time is allotted for us to work on projects of personal interest (new approach/tool)
