Lies, Damned Lies, and Statistics: A Misleading Study Compares TikTok and Instagram
Using a false comparison to scare Americans into banning TikTok.
A new study, “A Tik-Tok-ing Timebomb,” that compares the use of political hashtags on TikTok and Instagram is being widely shared by those calling for a ban or forced sale of TikTok. The report, from the Network Contagion Research Institute, shows that political posts which run counter to Chinese national interests underperform on TikTok (owned by Chinese company Bytedance) relative to Instagram (owned by US-based Meta). For the authors, this suggests that Chinese authorities are algorithmically throttling content on TikTok that harms their international interests, like posts about Tibet, the Uyghurs, and Tiananmen Square.
But the authors of the study made two remarkably basic errors that call into question the fundamental utility of the report.
First, the authors chose a flawed methodology that failed to account for how long each platform has existed. Instagram (launch 2010) is roughly twice as old as TikTok (international launch 2017). Thus, topics that were the subject of intense public discourse in the early 2010s, but which have not been heavily featured in the decade since, will underperform on TikTok versus Instagram.
Second, the authors assumed that the same people use both platforms, which led them to miss the potential for generational cohort effects. In short, the median user of Instagram is older than the median user of TikTok. Compare the largest segment of users by age on each platform: 25% of TikTok users in the US are ages 10-19, while 27.4% of Instagram users are 25-34. That roughly decade modal age gap will skew the ratio of topics covered on the two platforms. Simply put, older and younger users have different interests.
To see how these errors compounded to distort the study’s findings, let’s consider just one of the ten examples proferred by the authors. The study compares the frequency of three common hashtags related to Tibet and the Dalai Lama, finding a highly skewed ratio of 37.7 posts on Instagram for every 1 post on TikTok. That certainly appears suspicious given longstanding Chinese censorship of domestic activists advocating for Tibetan independence.
But bear in mind that the study’s authors were not assessing only those posts uploaded in a discrete period of time, like, say, in the fall of 2023. No, they were comparing the total number of posts ever posted since the platforms were created (Instagram: 2010 / TikTok: 2017).
And a quick peek at Google trends data shows that public discourse about Tibet in the US has been in a general decline throughout the 2000s and 2010s, albeit punctuated by exponential spikes (as much as 15x) in April 2008 and December 2016 which corresponded to moments of intense Tibetan activism.
Both spikes predated the international rollout of TikTok but happened at a time when Instagram already had hundreds of millions of users. It really isn’t surprising that six years worth of TikTok posts from a time of relatively low US interest in Tibet would be swamped by 13 years of Instagram posts from a time of intense US interest in Tibet.
Furthermore, even if one were hypothetically able to restrict the data on the Tibet hashtag to only posts made after the creation of TikTok, you would still expect to run into generational cohort effects. To put it simply, knowledge of and interest in a topic tends to persist. Millennials who were in college in 2008 during the Tibetan uprising are going to be more likely to be interested in — and thus post about — Tibet today than are members of Generation Z, some of whom were still in diapers in 2008.
None of what I’ve written here proves that TikTok didn’t or couldn’t manipulate its algorithm to downgrade content unfavorable to the CCP. But it does strongly suggest that this particular study is poorly designed and should not be used as evidence of algorithmic manipulation by TikTok. Frankly, I’m surprised at just how sloppy it is. A better study would choose a discrete time frame (such as when both platforms actually existed) and assess the ratio of posts in that time frame on controversial issues of importance to the Chinese government. If someone does that study, I’d like to see a copy.
Even then, however, it’s important to remember that these platforms aren’t synonymous and any outcomes might be skewed by platform-specific differences. For example, why should we expect the ratio of non-political and political content to be similar? That assumes a constant level of political interest between those who use the two platforms. But that is hardly a guarantee given generational differences between their user bases, the downstream contrasts from being photo-first versus short form video-first formats, and other platform specific divergences.
Regardless, the fact that many major news organizations missed these basic flaws in the study and then ran credulous coverage of the resulting report is an indictment of mood affiliation in journalism, especially when they are tasked with covering social media platforms with which they compete for public attention.