Labour MP’s Brexit poll is meaningless

“84% of Labour members want online ballot on new Brexit referendum, Watson poll finds”

Huffington Post, 29 May 2019

“The results of my Brexit poll are clear.  84% of Labour members and supporters who took the survey want an all-member ballot to decide our party's Brexit policy. As deputy leader I'll support them to make this happen.”

Tom Watson MP, 29 May 2019

Back in August, we wrote a blog on how to spot misleading poll figures. We ran through a range of issues which lead to bad polling, including using unreliable sources, working with unrepresentative samples, and asking leading questions.

Not for the first time since then, we find ourselves having to factcheck a poll from a Member of Parliament which deploys some or all of these bad polling techniques.

Labour Deputy Leader Tom Watson ran a recent poll on how “Labour members and supporters” think the party should set its future Brexit policy. He says that 84% of respondents want an all-member ballot to decide the party’s future policy, but the poll has flaws that make the findings effectively meaningless.

These “findings” were then reported by the Huffington Post with no level of critique.

So what went wrong?

Let’s start with the claim that these views represent “Labour members or supporters”.

All the poll (which ran on Mr. Watson’s personal website) asked for was your answer to the question “How do you think Labour should agree a new Brexit policy?”, and to provide your name, email address and say whether or not you are a member of the Labour party.

It’s not clear to us whether any further checks were done on the responses to ensure they were submitted by Labour members or supporters, or if they were filtered to include only the responses of those who said they were members. We’ve contacted Mr Watson’s office in an effort to confirm this, without success.

Even if you say that you are not a member, your answer is submitted. This means anybody could have answered the poll, not just Labour members and supporters. It appears there is no way to distinguish between respondents who were non-members but supporters, and those who were neither members nor supporters.

In addition, someone ticking a box saying they are a Labour party member is not evidence that they actually are. The survey didn’t ask for any other information, such as a membership number.

Also, there is no barrier to answering the poll multiple times with multiple email addresses. (We tested this by answering the poll multiple times – once for each option – after the results were published.)

There are more reasons why the poll isn’t representative

Even if every one of the 8,885 respondents is indeed a Labour member or supporter, the data still has a host of other problems.

Polls, by design, ask a small group of people (the “sample”) what they think, and tries to make sure that those people represent the wider population—meaning a survey is only as useful as the people it surveys.

For this survey to be meaningful, it would have to be “weighted” or have used quotas to be representative of the known characteristics of the Labour Party membership (such as age, gender, and social class).

But you can’t set quotas or do weighting unless you ask questions about a respondent’s age, gender and so forth.

Given the poll didn’t ask questions about this, we assume the results haven’t been adjusted to represent all Labour party members or supporters. In theory the email addresses could have been matched to a Labour party member database, linked to demographic information and then weighted accordingly. But that still doesn’t account for the views of “supporters”.

The poll is framed in a leading way

Then there are issues with the way the question is phrased. The three options offered to the question are that Labour’s new Brexit policy should be set by:

• - an all member poll,
• - a special conference,
• - or at the general Labour conference in September.

But the text on the website next to the poll tells the reader that the general Labour conference may come too late. It adds “we need a process to give members a say sooner” and describes the other two options as “good”.

Generally if you’re trying to find out the honest opinions of people through a survey, you don’t try and influence their opinion before asking them.

Additionally, the question is based on the premise that Labour party members or supporters want the party’s Brexit policy to change. The page is titled “Give Labour members a say on Brexit”, and Mr. Watson promoted the survey through social media by saying “Labour urgently needs to re-think its Brexit position and realign with members and voters”.

There is no option to choose if you think the current policy is the correct one, or if you think it should be decided in some way other than a poll or a conference. People who hold these views will have been unable to answer this poll.

Media reporting adds to the inaccuracies

Finally, the Huffington Post’s headline says the results show 84% of Labour members and supporters want an online ballot on a new Brexit referendum. The poll did not mention anything about a second referendum. The poll only asked how a new policy should be developed—without specifying what that policy could be.