Automated Analytics Testing

Automated Analytics Testing

Published: November 25, 2013 by Brian Kitchener

Most modern web sites and mobile applications use an analytics service to keep track of a user’s actions. While there are a variety of different providers, Google Analytics and Omniture are probably the best known. Every time the user does something important (like logging in, or adding an item to their cart) an HTTP call is sent to a server with information about the user. They do this for a variety of reasons. The product owner may want to know how many people end up purchasing an item after browsing. The developers may want to figure out what percentage of logins are unsuccessful. There are many reasons that keeping tracking of each user is important. In fact, some web sites generate revenue on these calls. For example, every time an advertisement is displayed to a user an analytics call is made to track it and a client is billed based upon it. So validating that these calls are happening correctly is vitally important, but it actually presents a variety of difficulties in trying to test it.

Testing Analytics

Testing analytics manually is a straightforward, yet tedious, process. Since the client sends HTTP requests to a third party server, nothing will appear on the web page, and nothing will show up in our application’s server logs. To validate the traffic was sent, the tester must proxy their web browser through some sort of HTTP proxy like Charles or Fiddler. These tools will record and track all HTTP calls sent from the client. The tester will then perform a specific action (like logging in), and verify that the appropriate HTTP calls were sent, and that they contained the correct information. Each call may have upwards of ten to twenty parameters that need to be validated and a single scenario can have multiple Business Intelligence (BI) calls sent. Since there can be dozens, if not hundreds of these tests necessary to validate, testing this manually can take days or weeks.

Using Automation

Automating this scenario is challenging, but not nearly as tedious as validating it manually. Just like the manual steps, this will involve an HTTP Proxy called BrowserMobProxy (http://bmp.lightbody.net/). BrowserMobProxy operates just like any other proxy, except that it has a REST API that can be queried to start and stop recording, and to fetch the list of HTTP calls. So testing these calls in an automated fashion is straightforward in any automation tool that supports REST. First we install BrowserMobProxy and run it. Second, we proxy our mobile device or web browser through the proxy, causing it to record all the traffic. Third, we execute our test, and once it is complete we make a GET request against BrowserMobProxy’s REST API to get the list of HTTP calls that it recorded. Validating that a specific request was sent becomes as easy as verifying a string is in the HTTP response body. Not every automated testing tool will work. We typically use a code-based tool like Selenium-WebDriver, or a GUI-based tool that supports both UI tests and HTTP requests such as SOASTA’s TouchTest. Keep in mind that if you are using TouchTest the mobile device, proxy, and SOASTA server all must be on an externally-visible network.

As you can see, automated testing of analytics calls is fairly simple, once we know what the process is, and what tools to use. We kick off a test and once it is done, query our proxy to verify the correct requests were sent. And while it may take several days or even weeks to build the automated tests and set up the proxy, once everything is working it will drastically reduce the amount of manual work necessary to validate a release.