Are we really getting happier?

If you're happy and you know it, click your mouse.

"The world is a happier place today and we can actually measure it because we have been tracking it," Reuters quoted John Wright, senior vice president of Ipsos, as saying. Like many other news sources, Reuters also repeated the suggestion from Ipsos that people are happier "despite economic woes, wars, conflicts and natural disasters".

But what do the numbers really say?

Every month from April 2007 to November 2011, Ipsos asked 18,687 people in 24 countries to say how happy they are. This may seem like a big number, but keep in mind that it was split across many countries, which gives us an average of 779 people per country. The Ipsos press release clarifies that some countries had over a thousand and others (like South Africa) over 500.

Now, just from these few details the dramatic headlines already start crumbling. There are a plethora of unanswered questions about Ipsos's methodology.

Whereas 24 countries were included in the survey – 172 countries were not. This makes any generalisations about the ‘world’ being happier unjustified.

While the Ipsos press release is headlined "Despite Woes, Conflicts, World a Happier Place than in 2007", the survey did not cover any conflict-ridden countries. There is no Iraq, no Afganistan, no Egypt, Tunisia, Libya, Pakistan or Iran. Why Ipsos chooses to frame their findings in terms of conflict is therefore puzzling. (The countries included in the survey were: Argentina, Australia, Belgium, Brazil, Canada, China, France, Germany, Great Britain, Hungary, India, Indonesia, Italy, Japan, Mexico, Poland, Russia, Saudi Arabia, South Africa, South Korea, Spain, Sweden, Turkey and the United States.)

Now, had we had credible evidence that people are happier despite being exposed to conflict, that would be interesting and something worth exploring. However, by suggesting that without any credible evidence, Ipsos is trivialising the impact of conflict in a way that is both unscientific and ethically distasteful.

The reported increase in happiness is merely 2 percentage points. The study reports no confidence intervals and so we cannot tell if the change in happiness is statistically significant. Ipsos only tells us that the estimated margin of error for a study of this size, if there was a 100% response rate, would be 3.1 to 4.5 percentage points per country. We are also not told what the response rate was. It is rare for a study of over 18,000 people over four years to have anything close to a 100% response rate. A graph showing the month by month fluctuations in reported happiness shows no trend toward increased happiness. On the contrary, other than what appears to be random variation, the graph is flat.

Sampling problems

The report on the survey lacks detail on many things. For example, it seems --though it is unclear-- that the same people were interviewed across the duration of the survey. If this was the case, how were people who dropped out of the survey or did not respond counted? To see why this is critically important, let's assume someone committed suicide mid-way through the survey period, presumably because she was unhappy. If she was included in the survey's final results then clearly the survey would be biased towards increased happiness, because she would not have been around to indicate her happiness in recent months. If on the other hand she was excluded from the survey, then the survey was no longer representative of unhappy people, severely biasing it.

Another obvious problem with this survey that more journalists should have picked up is the extremely small sample size per country. There are over 1.3 billion people in China. This study surveyed approximately 1,000 people. There are 50 odd million people in South Africa and less than 1,000 were surveyed. As an interesting aside, polls on local news websites often get a few thousand respondents. This survey is clearly closer to those than to proper research.

We also have little idea of who was surveyed. Ipsos tells us that the sample was weighted to represent each country's most recent census. No details are provided on how this was done or how the Ipsos Online Panel --the mechanism used to obtain answers-- actually works. What kind of reading and computer literacy skills were needed to participate in the study? What kind of access to computers and electricity would participants have needed? Are unemployed people with only primary school education who live in shacks in high crime areas really as well represented as middle-class people living in leafy suburbs? Of course it is possible that poor people might be getting happier and rich people less so or vice versa, but the point is that the survey's representativeness should be questioned.

If you take a close look at the month by month change by country, there is enormous variation, so that comparisons between countries are impossible and so is a broad conclusion about overall happiness. "Total happiness" (i.e. the total of "very happy" and "rather happy" people) in Argentina dropped from 88 to 67%, although people who said they were "very happy" rose from 19 to 23%. "Total happiness" increased in Australia, China, India, Indonesia, South Korea, Turkey and Saudi Arabia, but dropped everywhere else. Or did it? In fact the graphs of changing happiness in most of the surveyed countries --including happy India-- appear to fluctuate randomly with no discernible trend.

The China results however are interesting, not because China is interesting (of course it is, but that's not my point), but because it shows another flaw in the study. In April 2007, the first month of the survey, China's "total happiness" was 59%. The next month it rose to 92%. Then it drops back to mid-50s for two months and then for the middle of the survey period it's in the 80s and then drops to the 70s, and is at 78% in November 2011. Something is wrong here. If happiness fluctuates so wildly from month to month, what is being measured is clearly not very clear. It also shows how comparisons between countries cannot really be made.

Interestingly the press statement only reports the very happy findings. It does not report total happiness.

Happiness, or momentary perception of happiness?

The happiness question asked in this survey was: "Taking all things together, would you say you are very happy, rather happy, not very happy or not happy at all?"

In other words, the study is measuring people’s perception of their own happiness at a specific moment in time. This might be a very transient measurement. People may feel a certain way while they are busy filling in forms online, but very different a few hours later. And then there is also the problem of defining happiness. Did all the people in this study mean roughly the same thing with the term?

In fact, the problem of measuring and defining happiness is well-recognised and psychologists have made much more serious attempts at cracking it. I’m sure most serious psychologists in this field will be horrified at the way the findings from this survey are being presented.

There seems to be an underlying acceptance that market research doesn't require the same rigorous standards as proper scientific research that will be subjected to peer-review. It may well be that those who commission such studies are happy with wider confidence intervals and hyperbolic conclusions. After all, gambling with a company's money is not the same as gambling with people’s health.

However, the rules of statistics don't change. More pointedly, the relationship between the numbers and what those numbers can reasonably be interpreted to mean, doesn’t change.

Of course, when you make unwarranted claims in a scientific journal, your peers will embarrass you and show you why you are wrong. If you make far-fetched claims as a market research company, scores of newspapers and websites will report them uncritically. It is not only Ipsos that should be embarrassed, but the many unquestioning journalists and their editors who reported the survey without scepticism.