Catch them if you can
How we tricked robots to maintain intuitive UX
The story was written together with the mighty Alberto Trindade Tavares.
At Trustpilot, we send a few (million) review invitations per day on behalf of our business users. Each invitation includes an “unsubscribe” link at the bottom of the email. “But why would anyone want to unsubscribe?”, you might ask yourself. Trustpilot has used Sendgrid’s unsubscribe system for years. But it was time for us to build our own system: to allow for more flexibility and granularity, thereby enhancing user experience.
So we got to work building our own unsubscribe system, with a strong focus on the user. In our first iteration, a user would click the “unsubscribe” link at the bottom of the email if they wanted to unsubscribe. From there, they’d be redirected to a web page where we’d unsubscribe them from the backend controller, and render a nice “Thank you” page. We released the product and started measuring its success. But after 3 weeks, we observed the data and noted a few concerns (Figure 1). After release, we observed that there were approximately 6% more “unsubscribes” than “unsubscribe page loads.”
We asked ourselves: “How could a user enter the unsubscribe controller, where the unsubscribe call is invoked, but not load the success page? After consulting with our colleagues and stack overflow, we realized we were wrong to assume our users were the culprits. It was robots!
Often, some email clients use robots to click all the links within an email. In our case, this resulted in unintentional unsubscribes, preventing some users from receiving any emails from Trustpilot ever again. We calculated that there were about 400 automatic, robot-generated unsubscribes a day! That made us eager to solve the problem.
For our second iteration, we introduced another step in the process. Instead of only clicking the “unsubscribe” link directly at the bottom of the email, a user would be redirected to a web page where we added another button to “confirm the unsubscription”. Now, we’d unsubscribe them at the second stage only(Figure 2).
After release, we observed (Line 2 in Figure 1) that we’d always had more “unsubscribe page loads” than “unsubscribes,” and the problem was solved.
While our second iteration seemed more promising, we noticed an overarching problem. We love the great user experience at Trustpilot. By introducing another step to the process, we had jeopardized our great UX of one-click action. So we ran another test to find out if there was something we could do to accommodate both our need for intuitive UX and to block unwanted robot unsubscribes.
Hypothesis:
The fact that the robots didn’t trigger the page-load tracking events was interesting, and we wanted to take advantage of that.
Based on these observations, we formulated the following hypothesis:
Robots do not trigger a JavaScript event for page load (e.g., DOMContentLoaded) when ‘clicking’ the links from emails
If our hypothesis is true, it means robots can trigger server-side actions when making a request to links that appear in an email (this is what happened in the initial unsubscribe scenario described above), but can not trigger client-side ones.
Before discussing the potential consequences of this hypothesis further, we needed to first verify if it is in fact true.
Test conducted:
First, we had to find a way to send numerous emails that would hopefully be picked up by the robots, and then include bate in the form of a link for them to click. Lucky for us, we send more than a million email invitations a day on behalf of our business users. We decided to piggyback on part of those invitations with our test. We added an invisible link (Snippet 1) redirecting to a test page we built (Snippet 2).
The links that are placed in an email invitation’s HTML are hidden for humans (by applying some CSS magic). We hoped that only the robots would notice the existence of the links! We believe the robots simply parse the HTML of an email and fetch all the links from its content, and then “click” all of them. It, therefore, makes sense to assume that all clicks on an invisible link would come from the robots, and we’d be able to tell if they’re indeed triggering a page loaded event or not.
The last question we faced was tracking. How would we know how many robot clicks we have and how many page loads there were? As employees of a company that bases its business decisions on data, we track everything via Segment. It was a bit of a hurdle to connect this test to our tracking system, so we came up with a much simpler (and genius, if I may be humble) approach: use bit.ly as our tracker! bit.ly, a URL shortener, as a simple way to track clicks/requests. The invisible link we included in the emails used a shortened URL (https://bit.ly/xxxxxxx) that lead to a test page that listens to the DOMContentLoaded event, and then makes an asynchronous call to a second shortened URL (https://bit.ly/yyyyyyy). By checking the bit.ly dashboard, we were able to tell how many times the link was clicked and how many of those clicks triggered a page loaded event.
Results:
We kept the invisible link in the invitation emails for two days over a weekend, but it was enough for us to deliver 47696 emails with the link (Figure 3). Out of those, 17502 were opened. We identified 222 clicks on the link, which presumably came from the robots. This number is an overall indication that only a few email clients use robots that click links, representing only 0.5% of the total emails delivered. But when we take unsubscription into consideration, the numbers are significant and we want to avoid 0.5% of users being unsubscribed.
What about page loads?
The most successful result we could expect would be zero page loads. That would prove, without any doubt, that the robots don’t trigger this event. We observed a number quite close to zero, but not zero. We saw 24-page loads, from 222 clicks, which is only ~11%. Most (if not all?) of the clicks triggered by the robots don’t actually load the page in a browser. Perhaps they just make a request to “GET” the HTML content and analyze it, without running the code.
But why 24 and not zero? A possible explanation could be that the link might not always be invisible to humans. The link might eventually show for users who are using uncommon clients that don’t render HTML properly. We have seen some email clients ignoring CSS styling before. In this case, the user sees only a “.” (That’s why we decided to use a dot in the link content: to make the link less noticeable for humans in case it becomes visible). So, perhaps, these 24 clicks came from curious humans who happened to see this exotic link at the bottom of the email.
The possibility also exists that these page loads came from robots after all, but 24 is such a small number, that it’s just not significant enough. Considering the number of emails delivered, it’s only around 0.05% of them that turned out to load the test page.
Our results are not entirely scientific, but we dare to claim that email robots, in general, do not trigger client-side page loaded events.
Conclusions:
The first result from this test is that we proved that robots click links within emails. So, if you’re relying on one-click actions from emails, you should be aware of this. As we initially mentioned, introducing an extra step to confirm action is a solution to avoid robots but adds friction to the user experience. Conversion is certainly affected, as some users may leave the page before completing their action, However, when looking at other unsubscribe pages, in order to find an “industry standard”, we found that having this extra step seems to be the norm.
Then comes the second interesting result from this test. Knowing that robots don’t trigger client-side events, you can implement a safe one-click action that dismisses robots. Instead of directly dispatching actions from the server-side when the request is made, (from a controller, for example) you can only dispatch it after the page loads. From the client-side, you can make a call to handle whatever you need from the server-side. Now, the difference is that only humans could make it this far!
Allowing robots to inadvertently perform actions is not a good solution, and neither is jeopardizing intuitive user experience. We, therefore, chose to use the conclusion above to implement a one-click action that’s called from the client-side after the page loads, to preserve the ease of our user experience.
Now, it is time to share our findings with the world so more users can have an easy experience, without compromising the right functionality.
Special thanks to Mads Godvin Jensen, Cvetanov Goce, @michelle Mangione, and others who proofread and helped with writing this article.
