Analytics manager using data to make digital content more effective.
Google Analytics misreported a sizable percentage of Android devices within its mobile device reports. This issue is traced as far back as mid-November 2011 and as recently as mid-October 2012. Between that period, our reports list many other Android devices as Sony Xperia Arc or “(Not Set)”. That issue was resolved between October 16 and 17 of this year across all our accounts, Web properties and profiles. It’s not widely reported though a few members in the Google product forum confirm that this bug has affected their Google Analytics data. I go in depth on the impact of this bug on Android device metrics.
The Story
Last week, our mobile lead tasked me to report data on the top Android mobile devices used within our environment. During the discussions about the project this report was to be used for, the lead brought up preliminary data on the most used Android devices he found. That device was the Sony Xperia Arc. Curious, I thought, that a device I knew nothing about was topping our charts (I follow Android news closely). It was a side note to our meeting and I didn’t think much of it.
The most sought after Android devices on the market are subject to change. To get an accurate representation of our Android toting visitors, we need to look at recent data. The trick is to examine usage from enough visitors while keeping the numbers fresh. Our top Android devices over a one year window will look different from those of the past month. The US market differs from European and Asian markets in that consumers upgrade every two years through carrier subsidized incentives. Canadian consumers increasingly upgrade every 3 years for the same reason. European consumers still buy unsubsidized devices and bring them to the carrier. In a university market, roughly a fourth of the community is new each year, and those visitors often upgrade their handsets before starting their experience.
The Contradition
I wanted to report on both a 12 month and one month timeframe to validate the hypothesis that Android device churn is high within a given 12 month period. I noticed an unexplained discrepancy between the one year report and the past 30 days report. The former listed Sony Ericsson’s Xperia Arc as the most used device by a sizable margin. The later listed Samsung Galaxy S3 as the most used device. A bit of research contradicted the Web analytics reports. The Xperia Arc was never released by North American carriers and several reports corroborate that its radio is incompatible with North American bands. It was released in Europe in April of 2011. How then did it become the Android device to beat?
The Investigation
Advanced segment are the perfect tool to track unique Xperia Arc visitors. I expected to see churn lead to a gradual decline in Xperia Arc users. Instead, I found that the Xperia Arc dropped off a cliff on October 17, moving from top dog to subterranean dweller. That sharp drop drew my attention. The device was upgraded to Android 4.0 (ICS) earlier this year. However, carrier upgrades for Android devices never roll out overnight across all users. Upgrades also shouldn’t affect device information.
Attention turned to attributing the sharp drop-off as a bug. An advanced segment for unique Galaxy S3 visitors showed a surge in visitors between October 16 and 17, the same days that the Xperia Arc declined. The Google product forum confirmed that this bug did exist. It’s the only report of this bug, and you’d have to refine your Google search to find it, along the lines of “xperia arc analytics bug”.
Unique visitors the week before and after the Xperia Arc bug fix: segments showing devices reported as Sony Ericcson Xperia; Android OS devices reported as (not set); and a range of Android devices previously reported as Xperia Arc.
The Impact
Did the bug cover non-Android devices such as iOS and desktop visitors?
It’s inconclusive, but its unlikely that non-Android devices were impacted. I built a few custom reports and looked at Android operating system and version number for Xperia Arc users. All devices reported Android as their OS, and ran versions from 2.3.3 to 4.0.3. According to these two dimensions, the visitors reported as using Xperia Arc we all running a version of the Android OS. Screen resolutions for visitors reported as using Xperia Arc were far too random. I’m convinced that there is a bug in the screen resolution reports during this period. For instance, there were 178 different variations in screen resolutions between 1002 pixels tall and 1755 pixels tall. Either the devices are misreporting their resolution or the Xperia Arc bug also affected device resolution.
What other devices did the bug impact?
It’s also hard to tell. I looked at the pivot table that I’d created for Android device uniques in the past month, and pulled a list of top 10 devices. I ran advanced segments for each of these devices and found that some devices were not affected and others saw a drastic change.
In all, 5 devices categorized by their marketing names were affected. Affected devices registered no unique visitor usage prior to the bug fix, and shot up after the fix. From the comparison of the three segments above, these 5 devices account for roughly two thirds of devices reported as Xperia Arc.
These are the major Android devices that were affected within our reports:
- Galaxy S3
- Galaxy S2
- Razr 4G
- myTouch4G
- Galaxy Note
The Galaxy S2 was an interesting study. Some Galaxy S2 devices were registering prior to October 17, but most Galaxy S2 devices were missing prior to that day. It seems that one or two models were reporting correctly.
The image above shows a combined segment with all the affected devices.
Replicating the Results
If the issue seems like it would have impacted your measurement campaigns, you can run these tests on your Google Analytics profiles. These are the advanced segments for the impacted devices you could build and experiment with.
Affected Android Devices (Galaxy S2/S3/Note, Razr 4G, myTouch 4G)
Here is the text for the regular expression field: (galaxy s( ?ii|2|3))|(razr 4g)|(mytouch)|(galaxy note)
Android + Not Set
Sony Ericsson Xperia Arc
Android device names are unnecessarily complicated. For example, the Samsung Galaxy S3 has no less than 10 device names within the US, including 2 variants for each of the 4 national carriers (AT&T, Verizon, Sprint, T-Mobile) and 2 variants for the international version.
