YoungAfricaLive's Flawed Youth Survey

Often bad surveys get publicity despite having results that are almost meaningless.

Surveys, done properly, can tell us important things about people's beliefs, attitudes and behaviour. In many countries opinion polls usually correctly predict election outcomes. Surveys in Iraq gave us a good idea of how many additional people died in violence as a consequence of the US 2003 invasion and surveys in the Congo alerted the world to the extent of the humanitarian catastrophe due to the war there. In South Africa, regular surveys carried out by the Human Sciences Research Council and Statistics South Africa give us very useful information on the socioeconomic situation in the country. Conducting surveys properly is a science requiring extensive knowledge of statistics and the field being surveyed. But even well-conducted carefully thought through surveys almost always have shortcomings and are subjected to intense criticism.

When surveys are done badly they can perpetuate confusion and myths. Often they receive publicity despite giving results that are almost meaningless. One such example of how not to conduct a survey was released --with much fanfare including several favourable media reports-- a couple of weeks ago by the Praekelt Foundation. I don't know much about the foundation, but they have the noble aim of building "open source, scalable mobile technologies and solutions to improve the health and wellbeing of people living in poverty". So my criticism here is not meant to cast aspersions on the organisation, only on their survey.

The main findings of what the foundation describes as a "groundbreaking" survey are described on its website:

Based on over 50 revealing questions and over 138 954 frank responses, the YoungAfricaLive Youth Sex Survey reveals that whilst South Africa's youth are unquestionably sexually active (44%), they have strong views on the role of HIV/AIDS in sexual encounters, with a massive 81% equating 'not telling a sexual partner that you carry the virus' to outright murder.

In other findings, the survey showed that 66% of the responders identified themselves as less likely to have sex after they had been drinking with 33% admitting that they would be more likely to. The good news for national government's drive to use circumcision as part of an overall HIV prevention programme, a huge number of females - 78% - stated very clearly that they prefer their partner circumcised. [Also see this video which presents some of the results.]

Health24 and Eye Witness News summarised the survey's press statement without a hint of scepticism. But the survey, frankly, is bad science. It is not groundbreaking, at least not in a good way, and no generalisations about the attitudes of South African youth to sex and AIDS can be made from it.

The population from which the polls were drawn is the approximately 360,000 people who use YoungAfricaLive, a mobile phone platform run by the foundation. This is the first problem. While it might not be an exclusive service, it is unlikely to be a representative group of South Africans. At best the polls can only tell us about the attitudes of YoungAfricaLive users and not South Africans generally.

From the data in the Praekelt Foundation statement, we can calculate that there were on average less than 2,800 answers to each survey question. How were these samples drawn? Were people randomly asked to answer? Or did they opt in? It looks like the latter, in which case one of the most fundamental rules of conducting scientifically valid surveys was ignored: ensuring that the sample is representative by randomly selecting participants.

These two problems alone render the survey meaningless.

But the questions themselves, at least the ones presented in the foundation's video and in their press statement, are poorly designed. Many ask for value judgments, for an expression of prejudices and lead responders to answers, or provide vague choices that are not mutually exclusive. In essence they invite answers from people with particularly strong opinions, further biasing the survey. Here are some examples:

• True or False: HIV is a life sentence not a death sentence.
• Would you have protected sex with a hot girl or guy if they are HIV+?
• What do you think of gays in South Africa? (a) People born gay. (b) Committing a sin against God, (c) Their parents raised them gay.
• Girls prefer cut or uncut.
• 1 in 4 SA men have raped because (a) Men drink too much alcohol, (b) Women wear clothes that are too revealing, (c) men prove their manhood by physically dominating others.

In the last question, the statistic that 1 in 4 men have raped is an over-extrapolation of a statistic from another survey. One almost (but not quite) useful question is "Are you sexually active?" Leaving aside that "sexually active" is open to several interpretations, many people lie when asked in surveys if they have had sex.

Here's a particularly bad leading question, loaded with prejudice: "Not telling your partner that you're HIV+ is as bad as murder."

The foundation, as well as the news articles parroting their press release, state with satisfaction that 81% of respondents said that not telling is as bad as murder. Let's presume momentarily that this statistic accurately reflect the views of South African youth. Is this the message youth are getting from the hundreds of millions of rands spent annually on behavioural change programming? I hope not. I hope the question attracted only respondents with strong opinions, because it's opposite to the message that should be getting through: i.e. that protecting yourself from HIV is about taking responsibility not to harm yourself (except of course in the case of sexual assault but that's not hinted at in the question). It's about presuming that your partner is HIV-positive and taking the appropriate precautions, such as using condoms. If the message getting to youth is that not disclosing HIV status to a sexual partner is murder, it seems to me that is a deterrent to getting tested and finding out your status. It also ignores the extremely hard situation thousands of people, especially women, find themselves in from time to time with their partners, especially after just finding out their HIV status.

Part of the problem with bad surveys is that they have become very easy to do. Putting a poll onto a webpage is technically trivial. Unfortunately website polls are often gimmicks aimed at getting users to stay on the site. They definitely cannot be used to reach conclusions about lofty topics such as youth attitudes to sex and AIDS. It was irresponsible of the Praekelt Foundation to have claimed to have done so.