Master of all you survey

Torture numbers and they'll confess to anything
Greg Easterbrook

Since Domesday Book was compiled in 1086, large-scale data collection by survey has been part of government in this country. Can surveys based on self-reported information ever be reliable, whether it's an 11th century peasant reporting how many ducks she has to a Norman or a 21st century woman giving personal information over the phone or online? Are such surveys inherently any more reliable and honest than the 'personality' quizzes in teenage magazines? Not least among the factors influencing responses are the way questions are framed and how the respondent thinks the information will be used.

A recent survey for the Office for National Statistics (ONS) based on interviews with 450,000 people for an Integrated Household Survey (IHS) has been reported as finding that 71% of people in the UK are Christian and 20% have no religion, 1.5% are gay or bi and nearly 80% perceive themselves to be in good health. (All IHS statistics are considered experimental until assessed by the UK Statistics Authority).

The IHS asked respondents 'What is your religion, even if you are not currently practising?' with the intention of discovering 'religious affiliation - that is identification with a religion irrespective of actual practice or belief'. Not surprisingly, this has been reported as 71% are Christian.

Another survey by the ONS in 2008 found that only 22% described themselves as Christian and 45% said they had no religion.

The 2011 census will include the question 'What is your religion?'. This question was asked in the 2001 census and before that only in 1851. There has been controversy about the result of the 2001 census where 71% of people self-reported as Christian. Many of these, it is believed, identify as culturally rather than religiously Christian. Most of them rarely go anywhere near a church. According to a survey done by the Church of England, only 5% go at Christmas and 2.8% at Easter - the most important Christian festival.

Less seriously, 0.8% of people put their religion as Jedi in 2001 which offically makes them a bigger group than Sikhs, Jews or Buddhists in the UK.

The ONS deputy director said that the religion question in the next census would be 'a fabulous insight into societal changes to see how people register their religion'. Registering it is of course not the same as actively practicing it. And is 'fabulous' the best word to use for such a serious project?

Both religious and non-religious groups use these surveys in their campaigns to demand or challenge legal, financial and educational privileges, and governments use the findings to decide on funding, among other things, so they are more important than being of passing cultural interest. The widely differing findings are hardly a solid basis for policy or anything other than reflecting how variable survey findings can be.

People can and do change their minds about what they believe (although probably not so many of them in such a short space of time) but sexuality is a little less mutable.

The gay/bisexual statistic in the survey has led to headlines like 'Only one in 100 Britons is gay despite long-held myth' in the Mail.

It's inevitable that some groups will use stats to serve their own agenda and affected people will challenge them, especially if they have fought hard for equality.

One response to these findings on Facebook was '‎2,185,072 gay men and lesbians are currently registered on Gaydar in the UK - equating to 6.7 per cent of the UK population'.

More officially, in 2005, HM Treasury and the Department of Trade and Industry did a survey to help the Government analyse the financial implications of the Civil Partnerships Act (pensions, inheritance, tax benefits). They found that there were 3.6 m gay people in the UK – around 6% of the population. This figure was greeted by some gay rights activists as realistic.

If there are now only 1.5%, where have the other 4.5% gone since 2005? It should be noted that 3% of IHS respondents either said 'Don't know' or refused to answer. Again, the differences in survey results make basing any action on them a leap in the dark.

As to the 80% who 'perceived themselves to be in good health', what does this prove? Feeling well and being well are not the same thing at all for a start.

Even when stats are not used to tax the hell out of conquered Anglo Saxon peasants with very good reasons to be creative in their self-reporting, surveys are not like scientific tests. They are not reproducible in lab conditions, the methodology can be peer-reviewed but there can't be placebo questions, double blinding or a control group. At best, they can provide useful demographics, at worst they tell us nothing and can be used for propaganda. If you don't like the findings of the current survey, just hold on and there'll be another one along shortly.