YoungAfricaLive's Flawed Youth Survey

Surveys, done properly, can tell us important things about people's beliefs, attitudes and behaviour. But when they are done badly, as they often are, they perpetuate confusion and myths. One recent survey on youth attitudes to sex and HIV, announced with much fanfare, was so poor that its results are meaningless. And several journalists reported the survey without a hint of criticism.

Mortality survey in the DRC
Well conducted surveys showed the extent of the humanitarian catastrophe caused by the war in the Democratic Republic of Congo
Often bad surveys get publicity despite having results that are almost meaningless.

Surveys, done properly, can tell us important things about people's beliefs, attitudes and behaviour. In many countries opinion polls usually correctly predict election outcomes. Surveys in Iraq gave us a good idea of how many additional people died in violence as a consequence of the US 2003 invasion and surveys in the Congo alerted the world to the extent of the humanitarian catastrophe due to the war there. In South Africa, regular surveys carried out by the Human Sciences Research Council and Statistics South Africa give us very useful information on the socioeconomic situation in the country. Conducting surveys properly is a science requiring extensive knowledge of statistics and the field being surveyed. But even well-conducted carefully thought through surveys almost always have shortcomings and are subjected to intense criticism.

When surveys are done badly they can perpetuate confusion and myths. Often they receive publicity despite giving results that are almost meaningless. One such example of how not to conduct a survey was released --with much fanfare including several favourable media reports-- a couple of weeks ago by the Praekelt Foundation. I don't know much about the foundation, but they have the noble aim of building "open source, scalable mobile technologies and solutions to improve the health and wellbeing of people living in poverty". So my criticism here is not meant to cast aspersions on the organisation, only on their survey.

The main findings of what the foundation describes as a "groundbreaking" survey are described on its website:

Based on over 50 revealing questions and over 138 954 frank responses, the YoungAfricaLive Youth Sex Survey reveals that whilst South Africa's youth are unquestionably sexually active (44%), they have strong views on the role of HIV/AIDS in sexual encounters, with a massive 81% equating 'not telling a sexual partner that you carry the virus' to outright murder.

In other findings, the survey showed that 66% of the responders identified themselves as less likely to have sex after they had been drinking with 33% admitting that they would be more likely to. The good news for national government's drive to use circumcision as part of an overall HIV prevention programme, a huge number of females - 78% - stated very clearly that they prefer their partner circumcised. [Also see this video which presents some of the results.]

Health24 and Eye Witness News summarised the survey's press statement without a hint of scepticism. But the survey, frankly, is bad science. It is not groundbreaking, at least not in a good way, and no generalisations about the attitudes of South African youth to sex and AIDS can be made from it.

The population from which the polls were drawn is the approximately 360,000 people who use YoungAfricaLive, a mobile phone platform run by the foundation. This is the first problem. While it might not be an exclusive service, it is unlikely to be a representative group of South Africans. At best the polls can only tell us about the attitudes of YoungAfricaLive users and not South Africans generally.

From the data in the Praekelt Foundation statement, we can calculate that there were on average less than 2,800 answers to each survey question. How were these samples drawn? Were people randomly asked to answer? Or did they opt in? It looks like the latter, in which case one of the most fundamental rules of conducting scientifically valid surveys was ignored: ensuring that the sample is representative by randomly selecting participants.

These two problems alone render the survey meaningless.

But the questions themselves, at least the ones presented in the foundation's video and in their press statement, are poorly designed. Many ask for value judgments, for an expression of prejudices and lead responders to answers, or provide vague choices that are not mutually exclusive. In essence they invite answers from people with particularly strong opinions, further biasing the survey. Here are some examples:

  • True or False: HIV is a life sentence not a death sentence.
  • Would you have protected sex with a hot girl or guy if they are HIV+?
  • What do you think of gays in South Africa? (a) People born gay. (b) Committing a sin against God, (c) Their parents raised them gay.
  • Girls prefer cut or uncut.
  • 1 in 4 SA men have raped because (a) Men drink too much alcohol, (b) Women wear clothes that are too revealing, (c) men prove their manhood by physically dominating others.

In the last question, the statistic that 1 in 4 men have raped is an over-extrapolation of a statistic from another survey. One almost (but not quite) useful question is "Are you sexually active?" Leaving aside that "sexually active" is open to several interpretations, many people lie when asked in surveys if they have had sex.

Here's a particularly bad leading question, loaded with prejudice: "Not telling your partner that you're HIV+ is as bad as murder."

The foundation, as well as the news articles parroting their press release, state with satisfaction that 81% of respondents said that not telling is as bad as murder. Let's presume momentarily that this statistic accurately reflect the views of South African youth. Is this the message youth are getting from the hundreds of millions of rands spent annually on behavioural change programming? I hope not. I hope the question attracted only respondents with strong opinions, because it's opposite to the message that should be getting through: i.e. that protecting yourself from HIV is about taking responsibility not to harm yourself (except of course in the case of sexual assault but that's not hinted at in the question). It's about presuming that your partner is HIV-positive and taking the appropriate precautions, such as using condoms. If the message getting to youth is that not disclosing HIV status to a sexual partner is murder, it seems to me that is a deterrent to getting tested and finding out your status. It also ignores the extremely hard situation thousands of people, especially women, find themselves in from time to time with their partners, especially after just finding out their HIV status.

Part of the problem with bad surveys is that they have become very easy to do. Putting a poll onto a webpage is technically trivial. Unfortunately website polls are often gimmicks aimed at getting users to stay on the site. They definitely cannot be used to reach conclusions about lofty topics such as youth attitudes to sex and AIDS. It was irresponsible of the Praekelt Foundation to have claimed to have done so.

Comments in chronological order (2 comments)

Sam Manclark wrote on 22 June 2011 at 1:42 p.m.:

Praekelt Foundation’s recently launched Young Africa Live Sex Survey stands by its intention, which is to give “fascinating insight into the sexual behaviour and beliefs of South Africa’s youth”.

In releasing the results of a YoungAfricaLive-specific survey, Praekelt Foundation stated that the results were based on polls that form a key part of YAL’s offerings to users. In this, the specific parameters of the survey were clearly spelt out in a way that never claimed the YoungAfricaLive Sex Survey was conducted within traditional research paradigms.

In fact, Praekelt Foundation, through its extensive and intense work in mobile platforms in Africa, is adamant that new ways of gaining insight (not scientifically-based knowledge) through elements like polls cannot be dismissed out of hand because they are non-traditional.

Indeed for many, results that are drawn from a highly interactive user base on a free (to Vodacom subscribers) mobile platform that encourages open discussion on sometimes difficult topics, cannot be described as “almost meaningless”. The poll results, whilst sometimes carrying a value judgement, are in fact a starting point for discussion amongst many different communities including HIV/AIDS educators, government agencies, families, international funders, peer, educators and many more. In addition, YoungAfricaLive’s polls are valuable in that they allow for a quick response to topical issues of the day. In this, the results benefit from reflecting what young people think at the precise time a topical issue arises as opposed to assessing their opinions after a significant time lapse.

We believe that the value of this can never be underestimated and Praekelt Foundation will continue to release the results of polls as a way of taking the temperature of a young society that’s in flux and in constant transition – and facing some of the critical challenges of our time.

  • For the record, the polls responders are all YoungAfricaLive users (currently, YoungAfricaLive can only be access on the Vodacom platform, Vodafone Live"). In a recent poll, that went up on YoungAfricaLive Friday morning (17 June), and came down Monday morning (20 June) a total of 8726 respondents answered the question: Is YoungAfricaLive available to all South Africans? Help us help you.... tell us where you live”. The results are below:

Eastern Cape 620 votes (7 %) Free State 625 votes (7 %) Gauteng 2102 votes (24 %) KwaZulu-Natal 1283 votes (14 %) Limpopo 1440 votes (16 %) Mpumalanga 1321 votes (15 %) North West 720 votes (8 %) Northern Cape 207 votes (2 %) Western Cape 408 votes (4 %)

  • ends -

admin wrote on 22 June 2011 at 2:39 p.m.:

Hi Sam

First, I want to re-emphasise again that my criticisms are confined to the foundation's survey. It seems to me the mobile phone technology work you're doing is valuable.

But I don't believe your response above addresses the problems with the survey or my criticism that it was misleading.

Your press statement says, for example, "YoungAfricaLive Youth Sex Survey reveals that whilst South Africa's youth are unquestionably sexually active (44%), they have strong views on the role of HIV/AIDS in sexual encounters".

This and other lines in the press statement, subsequently copied in media reports, all wrongly convey the impression that this survey was representative of South African youth, when it's not even an unbiased representative sample of YoungAfricaLive users.

I'm not sure what is meant by a traditional versus non-traditional research paradigm, nor is there any mention of this in the foundation's statement.

If you're using polls to generate discussion that's fine, but then they should not be released as research findings.

Regards Nathan