Various Sikh organisations in the UK have conducted research to better inform policy and decision making. The most recent is the UK Sikh Survey conducted by The Sikh Network. Complex issues of race hate crime, employment discrimination and sexual grooming were addressed, and some interesting, but also debatable conclusions arrived at.  This leads to questions needing to be asked about methodology, sample size and selection, and responder characteristics (age, gender make-up etc.) so that the results can be interpreted more correctly. The Sikh Network, Jagbir Jhutti-Johal argues, should be more cautious about the conclusions it makes, although the data collected is still valuable.

UK Sikh Survey 2016

The Casey Review into community cohesion in Britain has raised some concern amongst the Dharmic faiths, who argue that research and government reports that have been commissioned to study migrant communities have generally focused on the Abrahamic faiths and that the less common faiths, in context of the UK, particularly Dharmic faiths have remained invisible to government.

Various Sikh organisations have conducted research to ‘better inform policy and decision making by government departments, other public bodies and political parties to properly address the needs and issues that matter to the British Sikh community’.

The most recent and highly publicised report The UK Sikh Survey, commissioned by The Sikh Network provides valuable information on the successes and contributions made by the British Sikh community, but also highlights some issues such as discrimination in employment and race hate crimes.

While the survey is interesting and the endeavour of measuring such views is commendable, there are problems and shortcomings in that its conclusions are provoked by the sample selected, which may not be representative of the estimated 430,000 Sikhs in the UK, and extrapolation of results by government bodies and MPs, media and professionals need to bear this in mind.

Here are four examples where methodological problems can have an impact on the conclusions.

1.    Sampling and representation

“The UK Sikh Survey 2016 is the largest and most comprehensive ever Sikh survey in the UK with over 4,500 respondents aged 16 and over. (p.4)”

The sample is 1% of the total Sikh population so it cannot be claimed that the survey is representative of the total Sikh population, and the total population may be 430,000 but the questionnaire only includes people 16 and over, so 20-25% missing, and so the figure is more like 1.5% of 16+.

It is a common misapprehension that the size of a survey is what gives it its representativeness, but the way the sample is selected is far more important. For robustness, the sample should be completely random or as close to random as possible. Indeed, this is what led to the errors in the prediction of the general election result.

However, to take a random sample a comprehensive list is needed, or sampling frame, from which to make the random selection. ‘Random’ here is not the same as haphazard or non-selected: it means that everyone has the same probability of being selected – like a lottery. When there is no list, it is impossible to know with what probability people are recruited.

There is no list of Sikhs in the UK from which to draw a sample. In the absence of a frame, researchers have to do the best they can. This survey, one would say, has done quite well:

The UK Sikh Survey 2016 was a combination of self-selection and targeting of groups i.e. the elderly or non-digital population, that are more difficult to reach through self-selection to ensure a better cross section of the community was covered. These groups were targeted through face to face interviews and hard copies of written questionnaires that were completed in public places like Gurdwaras attended by most Sikhs. (p.4)

Bias can be introduced into a sample in two ways: selection bias and nonresponse bias. The selection bias is where the people selected to be in the sample are not representative of the population, such as the hard to reach people, which in this instance demonstrated a bias to practicing and religious Sikhs.

Nonresponse bias is when people who are invited to take part do so selectively. That is, the people who choose to take part are not typical of the population – usually because they have ‘extreme’ views. So, for example, asking people if they enjoyed their meal will more likely get a response from those who are either very happy and want to express their thanks, or those who are very unhappy and want to register the fact. Those in the middle are less likely to respond. Consequently the ‘average’ may tell you nothing about how good the food is.

It is not possible to ascertain the nonresponse bias. A sample of over 4,559 is quite large, but a small percentage of the population: this does not make it unrepresentative, but it increases the possibility of bias. Under these circumstances the report should be more tentative about the precision of their conclusions – there must be a large margin of error. Take the first reported finding: ‘69% of all Sikhs are born in the UK’ (p.5). This might be too high: possibly those born outside the UK are more difficult to access (selection bias) or they may be less willing to take part (nonresponse bias).

2.    Extrapolation and generalizability of the results

The report equates the percentage of responders to the percentage of Sikhs in the UK, leading to some startling claims. For example, ‘Over 100,000 hate crimes against Sikhs aged 16 and over in the last 12 months’ (p.5).

More than 1 in 5 or 21% of those taking part in the survey have personally experienced race hate crime i.e. verbal or physical abuse in the last 12 months. (p.9)

To get from 21% of the sample to a population estimate, it appears the report has taken 21% of the population (around 430,000). But the sample does not include people under 16 (about 20-25%), so the wrong population figure is being used. Nevertheless, 21% is a very high figure, and it may be distorted by nonresponse bias (those not experiencing it may be more likely to be indifferent to taking part). But even if the estimate is twice as big as it should be, that is still a lot compared to government figures released for 2015-16.

The survey notes:

• 1 in 7 Sikhs directly experience discrimination in the workplace
• 8% (or 1 in 12) have experienced discrimination when dealing with public officials in the last 12 months (p5)

Such statements and figures ignore the complexity that surrounds discrimination. Did respondents actually understand what discrimination meant? Can we be sure that the cases mentioned are actual cases of discrimination, and not just incidents reflecting a lack of religious literacy? For example, if a security guard asks a Sikh to remove the turban or Kirpan that is not necessarily individual discrimination, but a lack of religious literacy and understanding of what the five Ks mean.

3.    Misleading conclusions

In general the report seems to stick to the data, but this is not true for sexual grooming. The report says:

More than 1 in 7 of Sikh women indicate being targeted by grooming gangs (p.5)

The actual question is ‘Have you or a friend/relative been a victim or a target for sexual grooming?’ So the question is not just ‘you’, but people you know, which makes the extrapolation to 1 in 7 of all Sikh women dubious. It should say ‘1 in 7 Sikh women have experienced or have a friend or relative who has experienced …’ The question includes no mention of ‘gangs’, so it is inappropriate to include that in the conclusion.

4.    Survey questions

Generally the questions are fair and not leading. However, this is not the case for the section on ‘Sikh Manifesto issues’ (see p.14). Here the viewpoint and bias of the Sikh Network is evident in the question design and analysis. The questions also contain assumptions. For example:

42. Do you think the independent public inquiry should also cover actions the UK Government has taken against the Sikh community at the request of India from raising its voice of concern?

This has the pre-defined premise that the UK Government has taken actions against the Sikh community at the request of India.

43. Do you believe the UK Government has been influenced by the Indian Government in its unfair treatment of Sikh political and religious activists?

This has the premise that the UK Government has been unfair in its treatment of Sikh political and religious activists, and has been influenced in this by the Indian Government.

46. Do you believe that every Sikh has the right to determine their own future as a basic human right?

Can one say no to this question?!

The design and complexity of some of the questions provide cues that might influence some of the answers, and the response options for several questions are designed such that people are forced to choose an opinion, even if they do not have one, and may lead to more positive options being selected.

There is a failure to define and explain to survey responders some of the terminology being used in questions which would have helped them give a more informed answer. Question 18 ‘Have you or a friend/relative been targeted or experienced conversion to another faith?’ is problematic because the word ‘targeted’ is very subjective.  Does targeted in this context mean being approached by a Jehovah Witness or Christian missionary in the street, which most of us have?  Or does it refer to more aggressive forms of conversion through violence and emotional blackmail?

Conclusion

Despite the reasonable sample size of the survey, it is difficult to generalize answers across the whole of the Sikh community given issues of responder numbers, selection and bias and a possible unrepresentative sample. Further, weaknesses in some of the questions which could be seen as leading through their design and complexity, and can provide cues that might influence the answers of some of the respondents, and the response options for several questions are designed such that people are forced to choose an opinion, even if they do not have one, and may lead to more positive options being selected. Although caution is suggested in taking this data at face value, it does not mean that the data is invalid, just that the data needs to be viewed with these possible issues in mind.

Given the importance of such research and the issues it raises, additional methods could have been utilised to support the online data. Qualitative methods could have been used to study in-depth key groups, separate from the hard to reach groups, such as Amritdhari (initiated) or Keshdhari (non-initiated) turban-wearing men who may have faced discrimination or race hate crime to give credibility and support to the findings. This is something that all future surveys conducted by Sikh organisations can take on board in order to ensure that a survey is fair and objective as possible.

About the author

Jagbir Jhutti-Johal is Senior Lecturer in the Department of Theology and Religion at the University of Birmingham

Correction

On Monday 23rd January the following sentence was removed:

“On the other hand, it does not include people below 16, who are more likely to be UK born, which would make it an underestimate.”

In the text above I assumed that the 69% figure for Sikhs born in the UK (as cited in pg.5 and 6 of the Report) did not include children under 16 years of age as I thought it was a figure taken directly from the responses to the Survey. As has been pointed out, this estimated figure does indeed include this population. It is my understanding that this figure has been calculated by extrapolating the survey results to the whole of the UK Sikh population and is therefore subject to the issues raised in the blog.