Analysing a binary variable
Test: Binomial test
(if you prefer to watch a video on this than read, click here)
In the example we noticed that 26% of the respondents indicated to be Female, and 74% Male. This might appear as a big difference, but is it a ‘significant’ difference, i.e. will there also be a difference in the population.
As discussed in the general section on significance (see here), the significance is the probability of a result as in the sample or even more extreme, if the assumption about the population is true.
With only two options to choose from, most often the assumption about the population is here that both groups are equal. This would mean that if we pick a random person from the population, the chance of him/her belonging to either category is 0.5 (50%).
The result in the sample was that we had 12 Female respondents. ‘More extreme’ would be less than 12. What we can ‘easily’ determine now is the probability of getting 12 or less Female out of 46, if in the population the chance of picking a female is 0.5. This can be done using a so-called binomial distribution. The chance for this is 0.0008. However, ‘more extreme’ can also mean that we have a similar over-representation of females. If the expected proportion is 0.5 (50%) then we can simply double the result, so the significance is 2 x .0008 = 0.0016.
We could report this as:
An exact binomial test indicated that the percentage of female (Nf = 12, 26%), was significantly different from the male percentage (Nm = 34, 76%), p = .002.
Click here to see how to perform a biniomial test with SPSS, R (Studio), Excel, Python or manually.
using non-parametric tests
using Legacy Dialogs
with R (Studio)
Download R script from video here.
Download Excel file from video here.
Download Jupyter Notebook from videos here.
or without using any libraries:
Manually (Formula's and example)
Given a probability of success (p), which for the binomial test is the expected proportion in the population, the number of trials (n), which for the binomial test is the total sample size, and the number of successes (k), which for the binomial test is number of occurences in one of the categories.
The formula for the cumulative binomial distribution (F(k; n,p)) is:
If p = 0.5 the formula could be simplfied into:
In the formula ⌊k⌋ is the 'floor' function. This gives the greatest integer (whole number) less than or equal to k. So for example ⌊2.8⌋ = 2, and ⌊-2.2⌋=-3.
is the binomial coefficient, this can be calculated using:
In this formula the ! indicates the factorial operation:
, and 0! is defined as 0! = 1.
The example worked out
We begin with filling out the values we know from the example in the main formula. I'll use the regular one (rather than the simplified for p = 0.5) as to illustrate how this would work. We get:
In the example the value for the floor function is already an integer, so we can rewrite:
Now let's focus on the binomial coefficient which is in general:
In the example we need to do this 13 times (for i = 0, to i = 12). The first one will give:
The second one (i = 1) gives:
The third one (i = 2) gives:
etc. all the way to the last one (i=12):
In this formula the ! indicated the factorial operation:
, and 0! is defined as 0! = 1.
In the example we then get of i = 0:
For i =1 we get:
For i = 2 we get:
For i = 12 we get:
However we are not done yet. These binomial coefficients each need to be multiplied with something, since we had:
This means we need to fill out i=0 to i=12 again. For i=0 we get:
For i = 1 we get:
For i = 2 we get:
etc. uptil and including i = 12:
Summing all of them up will yield:
To obtain the two-sided significance we multiply this by two, to finally obtain:
Note that if you have a different expectation about the population than the 0.50, we cannot simply double the result anymore. Well we could for a quick approximation, provided the sample size is large, but usually a more complex technique is then used, known as the ‘method of small p’, and in some cases another method known as ‘equal distance’. How these methods work is discussed in the end notes at the bottom of this page.
The binomial test can be computational heavy, so sometimes an approximation is used. The approximation is then either done using the Normal distribution, or a goodness-of-fit test. In both cases so-called continuity corrections can be applied and there are different variations on these corrections.
In short the binomial test has the following steps:
- The assumption about the population (the null hypothesis (H0)) is that the proportion of one of the two categories will be some amount (e.g. 0.5).
- The alternative is that it isn't (Ha) (e.g. the proportion in the population is not 0.5). This would be the so-called two-tailed test.
- Perform the binomial test and find the p-value (sig.).
- If the p-value is less than .05, the chance of a result as in the sample or even rarer if the assumption is true, is considered so low, that the assumption is probably NOT true. The proportion in the population is then probably NOT the one assumed at step 1. This is then called a significant result.
- If the p-value is .05 or more, the chance of a result as in the sample or even rarer if the assumption is true, is considered not low enough, that the assumption could be true. We don't have enough evidence to reject the assumption. This is then called a non-significant result.
The test informs us that there most likely will also be a difference in the population, it however does not say anything about if it is a big or small difference. For that we need a so-called effect size, which is the topic for the next section.
End notes (click to expand)
What if the expected proportion is not 0.50? (click to see the answer)
If in the example we’d expected 30% to be Female, our assumption about the population would change from 0.5 to 0.3. There are two methods we can then use to interpret ‘or more extreme’. The method of equal distance, and the method of small p-values.
The method of equal distance
This method looks at the number of cases. In a sample of 46 people, we’d then expect 46 x 0.3 = 13.8 Female respondents. We only had 12, so a difference of 13.8 – 12 = 1.8. The ‘equal distance method’ now means to look for the chance of having 12 Female or less, and 13.8 + 1.8 = 15.6 Female or more. Each of these two probabilities can be found using a binomial distribution. The ’12 or less’ probability is 0.3448, and the ’15.6 or more’ probability is 0.4031 (note that the 15.6 is always rounded down). Adding these two together than gives the two-sided significance of 0.7479.
The method of small p-values
This method looks at the probabilities itself. The probability of having exactly 12 Female scores out of a group of 46, if the chance of Female is 0.3, is 0.1119 (this is again a binomial distribution). The method of small p-values now considers ‘or more extreme’ any number between 0 and 49 (the sample size) that has a probability less or equal to the 0.1119. This means we need to go over each option, determine the probability and check if it is lower or equal. So, the probability of 0 Female, the probability of 1 Female, etc. In the example all counts of 12 or less, and all of 16 or more, each have a probability of 0.1119 or less. In total this results in a two-sided significance of 0.6300.
Single binary variable