Soon after he took control of Twitter, the social media platform’s new owner Elon Musk instituted an unexpected management mechanism: key decisions were suddenly being presented to the site’s users for final arbitration.
Should former president Donald Trump be returned to the platform? Users said yes, so — “vox populi, vox Dei,” as Musk liked to tweet! — he was back. Should previously banned accounts be granted amnesty? You bet! The populi had spoken!
Having seen prior examples of Twitter polls being taken seriously, I wrote at the time that the unscientific results of Musk’s surveys were not necessarily useful reflections of sentiment. Just because 15 million people weigh in, as happened with the Trump survey, that doesn’t make the results statistically significant.
But this was just rhetoric, an argument about such polls in general. It’s not like an actual pollster was replicating the results and allowing us to compare scientific polling with Musk’s surveys, right.
Wrong. YouGov did exactly that.
The results are fascinating. The pollsters took nine Musk tweets that included polls and recreated them within their polling system precisely. They even surveyed a statistically significant number of Twitter users, allowing us to compare the unscientific surveying of Twitter users on Twitter with the scientific assessment of the views of Twitter users off of Twitter. YouGov even asked whether respondents followed Musk, so we can compare the views of Musk’s follower base with the actual results of the surveys.
Before we walk through the results, there are some caveats. For example, the YouGov polls necessarily followed the Musk ones, at times by several weeks. In other words, public sentiment could possibly have shifted. There are also measurable margins of sampling error in the YouGov results, including relatively high ones for the Musk-follower subset.
But, barring an agreement between YouGov and Musk to run polls simultaneously (stay tuned for part three, maybe?), this is as good as we’re going to get.
First, the overview. On average, the results from Musk’s polls on Twitter varied by 16 points from the responses of everyone in YouGov’s surveys. Just among Twitter users responding to YouGov, the average was smaller, but still 11 percentage points. Among Musk followers, the average difference was 18 points. In other words, if the results of a tweet showed a 50-50 split, a YouGov result that matched the average deviation would show something like a 66-34 margin overall, a 61-39 margin among Twitter users and a 68-32 margin among Musk’s followers.
Nor did the number of responses make a difference. That 15-million response Twitter poll (the Trump one) was only 1 point off of the YouGov overall results — but the same difference occurred in a 3-million response Twitter poll. Among Twitter users, the Trump poll results on Twitter were each 6 points off of the YouGov results. Musk’s poll had it 52-48. YouGov had Twitter users at 58-42.
Precise, Twitter polls are not.
Here are the nine Twitter polls and the YouGov results, in chronological order. On our graphs, “on Twitter” refers to the results of Musk’s own poll. The other three are categories from YouGov: all respondents (“overall”), the subset that are Twitter users and the subset of that group that follows Musk.
– Redo elections of annexed regions under UN supervision. Russia leaves if that is will of the people.
– Crimea formally part of Russia, as it has been since 1783 (until Khrushchev’s mistake).
– Water supply to Crimea assured.
– Ukraine remains neutral.
— Elon Musk (@elonmusk) October 3, 2022
Difference in margin with all YouGov respondents: 40 points. Among Twitter users in YouGov’s poll: 60 points.
Let’s try this then: the will of the people who live in the Donbas & Crimea should decide whether they’re part of Russia or Ukraine
— Elon Musk (@elonmusk) October 3, 2022
Difference in margin with all YouGov respondents: 28 points. Among Twitter users in YouGov’s poll: 38 points.
Bring back Vine?
— Elon Musk (@elonmusk) October 31, 2022
Here I’ll note that the results may have been skewed by the likelihood that many non-Twitter users may not have known what Vine is. (It is a defunct video tool.) Difference in margin with all YouGov respondents: 62 points. Among Twitter users in YouGov’s poll: 24 points.
Advertisers should support:
— Elon Musk (@elonmusk) November 2, 2022
And here I’ll note that the verbiage of this poll is deeply leading. But YouGov recreated it faithfully nonetheless. Difference in margin with all YouGov respondents: 2 points. Among Twitter users in YouGov’s poll: 4 points.
Are you seeing far fewer bots/scams/spam?
— Elon Musk (@elonmusk) November 10, 2022
There are no overall responses here since non-Twitter users wouldn’t have insight into the frequency of bots on Twitter. Difference in margin among Twitter users in YouGov’s poll: 2 points.
Reinstate former President Trump
— Elon Musk (@elonmusk) November 19, 2022
Difference in margin with all YouGov respondents: 2 points. Among Twitter users in YouGov’s poll: 16 points.
Should Twitter offer a general amnesty to suspended accounts, provided that they have not broken the law or engaged in egregious spam?
— Elon Musk (@elonmusk) November 23, 2022
Difference in margin with all YouGov respondents: 28 points. Among Twitter users in YouGov’s poll: 6 points.
Apple should publish all censorship actions it has taken that affect its customers
— Elon Musk (@elonmusk) November 28, 2022
Difference in margin with all YouGov respondents: 26 points. Among Twitter users in YouGov’s poll: 8 points.
I am not expressing an opinion, but did promise to conduct this poll.
Should Assange and Snowden be pardoned?
— Elon Musk (@elonmusk) December 4, 2022
Difference in margin with all YouGov respondents: 77 points. Among Twitter users in YouGov’s poll: 45 points.
This last one is a useful example. There were 3.3 million responses to Musk’s Twitter poll, but there’s little doubt that the result was powered to some degree by users being motivated to participate. The results mirror Musk’s follower count closely — but that’s nowhere near public opinion in general or even Twitter overall.
Yes, some results from YouGov are close to the ones Musk got. Many aren’t. Many are way off the mark — as are the averages. Twitter polling is simply not a reliable measure of opinion, even of Twitter users.
On Thursday night, Musk tried another poll, asking when banned accounts that Musk claimed had revealed his location in real-time should be allowed back on the platform. “Now” won. So Musk declared there had been too many options presented in the poll and it should be re-run — certainly his prerogative but a move that reflects the malleability of Twitter polling.
As of writing, with six hours left, “now” is winning the new poll by an even wider margin than the first time.