Confidentiality and bad science
The case of the Solal survey
Published: May 10, 2012, 4:06 p.m., Last updated: May 10, 2012, 7:01 p.m.
When evaluating survey data the devil is always in the detail. In a recent case the Advertising Standards Authority (ASA) allowed Solal Technologies to keep crucial survey details secret. This decision deprived the complainant, Marcus Low, from an opportunity to interrogate spurious survey findings.
Confidentiality in this case is ill-advised.
At the outset it is worth emphasising that the ASA predominantly does good and important work. They are under-resourced, swamped by complaints, under consistent public scrutiny, and are being taken to court by a supplement company. Yet, they mostly still deliver sensible rulings. This article is about the exception to the rule.
Confidentiality
Matters before the Advertising Standards Authority (ASA) often hinge on whether or not companies can provide evidence to support their advertising claims. If you claim that your patented concoction can reduce consumers' risk of heart disease, you need to present substantiation (evidence) proving that is indeed the case.
In rare cases though, it can be argued that the substantiation contains sensitive or commercial information that should be treated as confidential. In such cases, the ASA have to decide whether the substantiation warrants confidentiality. In other words, they have to weigh up the complainant's right to respond to the substance of the substantiation against the respondent’s reasons for not letting the complainant see the substantiation.
The ASA rather sensibly rejects most requests for confidentiality. After all, you cannot have a fair hearing if the complainant does not have an opportunity to respond to the substance of the matter. However, in one recent case, involving me as the complainant and Solal Technologies as the respondent, the ASA rather puzzlingly did grant confidentiality.
A puzzling decision
On June 29 last year the ASA ruled, in response to my complaint, that Solal's advertising pay-off line "prescribed by doctors, recommended by pharmacists" was unsubstantiated. At the time Solal simply did not present the ASA with any evidence in support of this claim.
In cases where claims are found to be unsubstantiated, the ASA allows companies to present new substantiation when it becomes available. Accordingly, Solal commissioned a market research company called Catalyst to do a survey and presented the new substantiation from this survey to the ASA.
Crucially though, Solal requested confidentiality for the new survey details – which the ASA granted. This means that I never had the opportunity to respond to the new survey by Catalyst, even though I was the complainant. All I have seen of the survey is a very short executive summery. In my response to the ASA I argued that I could not respond to the new substantiation without actually being given the details of the survey.
The ASA ignored this line of argument and ruled on the matter on 3 April without letting me see or respond to the survey details. Based on the same survey they found the claim "prescribed by doctors, recommended by pharmacists" to be substantiated.
I wrote to the ASA again asking them to reconsider the decision to provide confidentiality. They responded that granting confidentiality is a matter of discretion (as is indeed stated in the advertising code), an administrative function and therefore cannot be appealed. In other words, there is no way for a complainant to appeal against the wrongful granting of confidentiality – which indicates either a shortcoming in the advertising code or in the ASA directorate’s interpretation of the code.
Why the decision on confidentiality was wrong
The ASA did send me a brief summary of the survey that was devoid of all the crucial survey details. In fact, the few scraps of information that were included in the summary raised red flags about the survey and its methodology which I then raised with the ASA.
The ASA did not respond to these concerns. They seem to have ignored the details of the survey and to have taken the survey company’s word that the claim "prescribed by doctors, recommended by pharmacists" was substantiated. The fact that the head of the survey company is SAMRA (South African Market Research Association) accredited, seems to have satisfied the ASA that they did not need to engage with the details of the survey.
In any case, here are some key reasons why confidentiality in this case is ill-advised:
-
The claim that doctors in any significant numbers specifically prescribe Solal’s products is far-fetched. This doesn’t mean that the survey should be rejected, but rather that the methodology of the survey should be carefully studied and openly scrutinised to ensure that the conclusion is valid.
-
There are reasons to doubt the independence of the survey. The survey summary contained the phrase "collaborate the in-house database". It is of course impossible to know what this phrase means without having access to the full survey, but it suggests that Solal’s in-house database was used in the survey. If Solal’s in-house database was used, the survey is not independent as required by the advertising code. More importantly, it would mean that a biased sample was used and that the results of the survey are meaningless. I raised this concern with the ASA, but I was not given any clarification.
It is also interesting that if the survey was truly independent, the case for confidentiality on commercial grounds would be very weak. If the survey convincingly showed that a significant number of randomly selected doctors responding to a high quality questionnaire agreed that they prescribed Solal’s products, then making the survey data available could only benefit Solal, unless of course there is something wrong with the survey.
Finally, some of the language used in the survey summary is worrying, especially given that the summary emanates from a SAMRA accredited entity.
For example, the summary states that the claim in question is "unequivocally true". That is very strange language for this kind of survey. Surveys have margins of error but the market research company did not include these figures in the summary.
Of course, none of the above means that the survey findings are necessarily nonsense. However, it raises such serious concerns about the survey that the granting of confidentiality doesn’t seem to make any sense. I and other members of the public should have the opportunity to scrutinise the survey.
Remarkably, even as the complainant in this matter I was never given answers to simple questions like:
-
How were doctors sampled?
-
How many doctors were surveyed?
-
What were the questions in the questionnaire?
-
What were the actual findings?
If you have any ideas on why details such as these warrant confidentiality (allowing of course for anonymity), I’d be interested to read your views in the comments section of this article.
Good article - thanks Marcus.
It is sad that Solal are hiding behind 'confidentiality' here. It does nothing to enhance the belief that this is an open and honest evaluation of the claim. It merely suggests they have something to hide.
Once of the hardest, but most virtuous, things in the scientific method is to be scrupulously open and honest about methodologies. That's why, despite the initial hype, we no longer have claims of FTL neutrinos or cold-fusion.
One of the most common behaviours of pseudoscience is to hide from the light. Shame on Solal.