Analysing two ordinal variables

Part 3: Test and effect size

On the previous page we saw that there appears to be some relation between what students think of the teacher's ability to link theory to practical situations, and his/her ability to motivate the student. However, this was just a sample, so what does that mean for the entire population. There are various tests that could be used, but I will use Goodman-Kruskal gamma (γ) (Goodman & Kruskal, 1954). This will test if a so-called monotonic relationship exists between two ordinal variables. Gamma uses so-called concordent and discordent pairs to check for this.

In the example this test will have a significance of .000, which is the chance of having a sample with a Gamma value of 0.877 or even higher, if in the population it would be 0. Since this chance is so low (below 0.050), we can conclude that most likely in the population Gamma will be significantly different from 0. Or you can say that by knowing the score on one of the variables we can predict the other with 87.7% more accuracy.

Unfortunately there is no formal way to determine if 0.877 is high or low (although almost everyone would agree this is pretty high), and the rules of thumb floating around on the internet vary quite a lot, often depending on the field (e.g. biology, medicine, business, etc.). I will use the rule of thumb from Rea and Parker (1992):

0.00 < 0.10 - Negligible
0.10 < 0.20 - Weak
0.20 < 0.40 - Moderate
0.40 < 0.60 - Relatively strong
0.60 < 0.80 - Strong
0.80 < 1.00 - Very strong

Note that Gamma can range from -1 to +1 and the table above is for the absolute values of Gamma (so ignoring the minus, i.e. a gamma of -0.15 could be interpreted as weak since 0.15 falls in the 0.10 < 0.20 category). Click here to see a table with various other rule of thumbs for the interpretation.

In the example Gamma was 0.877 which would indicate a very strong effect.

A positive Gamma (i.e. above 0) indicates a positive relation, which means that if someone scores high on one variable, s/he will most likely also score high on the other. A negative Gamma (i.e. below 0) indicates that if someone scores high on one variable, s/he will most likely score low on the other. It is important to check how each variable was coded. Most likely they were coded the same, but if they weren’t the interpretation of Gamma is in reverse. An example to illustrate this. Let’s say we have one variable that is coded as 1 = very good to 5 = very bad, and another variable is coded as 1 = very bad to 5 = very good. The Gamma was -0.73 between these two variables. Now a negative gamma, means a negative association, so scoring high on one variable, means scoring low on the other. However since in this example a low score is actually the same as a high score on the other, it is actually a positive association.

In the report we could add:

The Goodman-Kruskal gamma showed that there was a significant high positive association between how motivational the teacher was, and how s/he was able to link the theory to practice, γ =.88, p < .001.

Click here see how to perform the test with SPSS, R (studio), Excel, Python, or Manually.

with SPSS

click on the thumbnail below to see where to look in the output

with R (Studio)

note the video uses a different example than used here.

click on the thumbnail below to see where to look in the output

with Excel

note the video uses a different example than used here.

with Python

Manually (formulas and example)

Goodman-Kruskal gamma

Formulas

The formula for Goodman-Kruskal gamma is:

$\gamma=\frac{P-Q}{P+Q}$

In this formula P and Q are defined as:

$P=\sum_{i,j}P_{i,j}$

$Q=\sum_{i,j}Q_{i,j}$

With:

$P_{i,j}=O_{i,j}\times{CO_{i,j}}$

$Q_{i,j}=O_{i,j}\times{D_{i,j}}$

And:

$C_{i,j}=\sum_{h<i}\sum_{k<j}O_{h,k}+\sum_{h>i}\sum_{k>j}O_{h,k}$

$D_{i,j}=\sum_{h<i}\sum_{k>j}O_{h,k}+\sum_{h>i}\sum_{k<j}O_{h,k}$

Where $O_{h,k}$ is the number of cases that scored h for the first variable, and k for the second.

Example

Note: Different than the one used in the other sections.

The observed values from two variables are shown below:

Oij	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	1	4	9	1	3
disagree	0	1	2	2	1
neutral	2	0	4	1	1
agree	1	1	1	1	3
fully agree	1	0	3	0	2

The concordant scores (Cij) are given by the formula:

$C_{i,j}=\sum_{h<i}\sum_{k<j}O_{h,k}+\sum_{h>i}\sum_{k>j}O_{h,k}$

This means it is the sum of all values in the cells to the upper-left and the lower-right.
For the neutral-neutral this is the sum of the cells highlighted in yellow: 1+4+0+1+1+3+0+2 = 12.

The results for all cells are:

Cij	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	23	21	11	7	0
disagree	17	17	13	20	15
neutral	11	11	12	22	20
agree	5	8	10	25	27
fully agree	0	4	10	26	31

The discordant scores (Dij) are given by the formula:

$D_{i,j}=\sum_{h<i}\sum_{k>j}O_{h,k}+\sum_{h>i}\sum_{k<j}O_{h,k}$

This means it is the sum of all values in the cells to the lower-left and the upper-right. For the neutral-neutral this is the sum of the cells highlighted in green: 1+3+2+1+1+1+1+0 = 10.

The results for all cells are:

Dij	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	0	4	6	16	20
disagree	17	17	9	16	15
neutral	23	20	10	11	8
agree	29	25	10	9	4
fully agree	35	29	13	8	0

The calculation of each individual cell for Cij and Dij can be found in this pdf.

Now we can determine:

$P_{i,j}=O_{i,j}\times{CO_{i,j}}$

Pij	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	1x23 = 23	4x21 = 84	9x11 = 99	1x7 = 7	3x0 = 0
disagree	0x17 = 0	1x17 = 17	2x13 = 26	2x20 = 40	1x15 = 15
neutral	2x11 = 22	0x11 = 0	4x12 = 48	1x22 = 22	1x20 = 20
agree	1x5 = 5	1x8 = 8	1x10 = 10	1x25 = 25	3x27 = 81
fully agree	1x0 = 0	0x4 = 0	3x10 = 30	0x26 = 0	2x31 = 62

Then we need to sum all these up to get P:

$P=\sum_{i,j}P_{i,j}=23+84+99+7+0+0+17+26+40+15+22+0+48+22+20+5+8+10+25+81+0+0+30+0+62=644$

Now the same but with the discordant results:

$Q_{i,j}=O_{i,j}\times{D_{i,j}}$

Qij	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	1x0 = 0	4x4 = 16	9x6 = 54	1x16 = 16	3x20 = 60
disagree	0x17 = 0	1x17 = 17	2x9 = 18	2x16 = 32	1x15 = 15
neutral	2x23 = 46	0x20 = 0	4x10 = 40	1x11 = 11	1x8 = 8
agree	1x29 = 29	1x25 = 25	1x10 = 10	1x9 = 9	3x4 = 12
fully agree	1x35 = 35	0x29 = 0	3x13 = 39	0x8 = 0	2x0 = 0

Then we need to sum all these up to get Q:

$Q=\sum_{i,j}Q_{i,j}=0+16+54+16+60+0+17+18+32+15+46+0+40+11+8+29+25+10+9+12+35+39+0+0=492$

Finally we can then determine gamma:

$\gamma =\frac{P-Q}{P+Q} =\frac{644-492}{644+492} =\frac{152}{1136} =\frac{19\times8}{142\times8} =\frac{19}{142} \approx0.1338$

The test

Formulas

Unfortunately there seems to be a three variations in the formula for the test statistic of Goodman-Kruskal gamma. The first is:

$Z=\gamma\times\sqrt{\frac{P+Q}{n\times(1-\gamma^2)}}$

The second and third at first are the same:

$Z=\frac{\gamma}{ASE}$

But vary in how they determine the ASE. The software R uses:

$ASE=\frac{4\times\sqrt{\sum_{i,j}Q_{i,j}\times(Q\times{C_{i,j}}-P\times{D_{i,j}})^2}}{(P+Q)^2}$

While SPSS uses:

$ASE=\frac{2\times\sqrt{\sum_{i,j}((C_{i,j}-D_{i,j})^2)-\frac{(P-Q)^2}{N}}}{P+Q}$

n in these formulas is the total sample size, all other variables are the same as in the Goodman-Kruskal gamma section.

Example

Note: Continuation of the Goodman-Kruskal gamma section.

The first formula can be completed almost immediately since we already calculated all needed values, when calculating gamma itself, except for the total sample size:

$n=\sum_{i,j}O_{i,j}=1+4+9+1+3+0+1+2+2+1+2+0+4+1+1+1+1+1+1+3+1+0+3+0+2=45$

The we can determine the approximate z-value:

$Z=\gamma\times\sqrt{\frac{P+Q}{n\times(1-\gamma^2)}} =\frac{19}{142}\times\sqrt{\frac{644+492}{45\times(1-\left(\frac{19}{142}\right)^2)}}$

$=\frac{19}{142}\times\sqrt{\frac{1136}{45\times(1-\frac{19^2}{142^2})}} =\frac{19}{142}\times\sqrt{\frac{1136}{45\times(1-\frac{361}{20164})}} =\frac{19}{142}\times\sqrt{\frac{1136}{45\times(\frac{20164}{20164}-\frac{361}{20164})}}$

$=\frac{19}{142}\times\sqrt{\frac{1136}{45\times\frac{20164-361}{20164}}} =\frac{19}{142}\times\sqrt{\frac{1136}{45\times\frac{19803}{20164}}} =\frac{19}{142}\times\sqrt{\frac{1136}{\frac{891135}{20164}}}$

$=\frac{19}{142}\times\sqrt{\frac{1136\times20164}{891135}} =\frac{19}{142}\times\sqrt{\frac{22906304}{891135}}$

$=\frac{19}{142}\times\frac{1}{891135}\times\sqrt{22906304\times891135}$

$=\frac{19}{126541170}\times\sqrt{20412609215040} =\frac{19}{126541170}\times\sqrt{1704^2\times7030065}$

$=\frac{19\times1704}{126541170}\times\sqrt{7030065} =\frac{32376}{126541170}\times\sqrt{7030065}$

$=\frac{76\times426}{297045\times426}\times\sqrt{7030065} =\frac{76}{297045}\times\sqrt{7030065} \approx0.6784$

For the second formula, we need to determine for each cell:

$ASE=\frac{4\times\sqrt{\sum_{i,j}Q_{i,j}\times(Q\times{C_{i,j}}-P\times{D_{i,j}})^2}}{(P+Q)^2}$

The calculations for each cell is shown below.

	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	1x(492x23-644x0)^2	4x(492x21-644x4)^2	9x(492x11-644x6)^2	1x(492x7-644x16)^2	3x(492x0-644x20)^2
disagree	0x(492x17-644x17)^2	1x(492x17-644x17)^2	2x(492x13-644x9)^2	2x(492x20-644x16)^2	1x(492x15-644x15)^2
neutral	2x(492x11-644x23)^2	0x(492x11-644x20)^2	4x(492x12-644x10)^2	1x(492x22-644x11)^2	1x(492x20-644x8)^2
agree	1x(492x5-644x29)^2	1x(492x8-644x25)^2	1x(492x10-644x10)^2	1x(492x25-644x9)^2	3x(492x27-644x4)^2
fully agree	1x(492x0-644x35)^2	0x(492x4-644x29)^2	3x(492x10-644x13)^2	0x(492x26-644x8)^2	2x(492x31-644x0)^2

Which has the following results:

	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	128051856	240622144	21566736	47059600	497683200
disagree	0	6677056	720000	430592	5198400
neutral	176720000	0	1149184	13987600	21977344
agree	262958656	147962896	2310400	42302016	343983792
fully agree	508051600	0	35748912	0	465247008

Now we can determine:

$\sum_{i,j}Q_{i,j}\times(Q\times{C_{i,j}}-P\times{D_{i,j}})^2=128051856+240622144+21566736+47059600+497683200+0+6677056+720000+430592+5198400+176720000+0+1149184+13987600+21977344+262958656+147962896+2310400+42302016+343983792+508051600+0+35748912+0+465247008=2970408992$

And use this result to calculate:

$ASE=\frac{4\times\sqrt{\sum_{i,j}Q_{i,j}\times(Q\times{C_{i,j}}-P\times{D_{i,j}})^2}}{(P+Q)^2} =\frac{4\times\sqrt{2970408992}}{(644+492)^2}$

$=\frac{4\times\sqrt{4^2\times185650562}}{1136^2} =\frac{4\times4\sqrt{185650562}}{1290496} =\frac{16\sqrt{185650562}}{1290496}$

$=\frac{16\sqrt{185650562}}{16\times80656} =\frac{\sqrt{185650562}}{80656} \approx0.1689$

Which gives a z-statistic of:

$Z=\frac{\gamma}{ASE} =\frac{\frac{19}{142}}{\frac{\sqrt{185650562}}{80656}} =\frac{19\times80656}{142\times\sqrt{185650562}} =\frac{19\times568\times142}{142\times\sqrt{185650562}}$

$=\frac{19\times568}{\sqrt{185650562}} =\frac{19\times568}{\sqrt{185650562}}\times\frac{\sqrt{185650562}}{\sqrt{185650562}} =\frac{19\times568\times\sqrt{185650562}}{\sqrt{185650562}\times\sqrt{185650562}}$

$=\frac{19\times568\times\sqrt{185650562}}{185650562} =\frac{19\times284\times\sqrt{185650562}}{92825281} =\frac{5396\sqrt{185650562}}{92825281} \approx0.7921$

For the third formula we first need to determine for each cell:

$ASE=\frac{2\times\sqrt{\sum_{i,j}((C_{i,j}-D_{i,j})^2)-\frac{(P-Q)^2}{N}}}{P+Q}$

The calculations for each cell is shown below.

	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	1x(23-0)^2	4x(21-4)^2	9x(11-6)^2	1x(7-16)^2	3x(0-20)^2
disagree	0x(17-17)^2	1x(17-17)^2	2x(13-9)^2	2x(20-16)^2	1x(15-15)^2
neutral	2x(11-23)^2	0x(11-20)^2	4x(12-10)^2	1x(22-11)^2	1x(20-8)^2
agree	1x(5-29)^2	1x(8-25)^2	1x(10-10)^2	1x(25-9)^2	3x(27-4)^2
fully agree	1x(0-35)^2	0x(4-29)^2	3x(10-13)^2	0x(26-8)^2	2x(31-0)^2

Which has the following results:

	Variable2
Variable1	fully disagree	disagree	neutral	agree	fully agree
fully disagree	529	1156	225	81	1200
disagree	0	0	32	32	0
neutral	288	0	16	121	144
agree	576	289	0	256	1587
fully agree	1225	0	27	0	1922

Now we can determine:

$\sum_{i,j}(O_{i,j}\times(C_{i,j}-D_{i,j})^2)=529+1156+225+81+1200+0+0+32+32+0+288+0+16+121+144+576+289+0+256+1587+1225+0+27+0+1922=9706$

Using this in the formula for ASE:

$ASE=\frac{2\times\sqrt{\sum_{i,j}(O_{i,j}\times(C_{i,j}-D_{i,j})^2)-\frac{(P-Q)^2}{N}}}{P+Q} =\frac{2\times\sqrt{9706-\frac{(644-492)^2}{45}}}{644+492}$

$=\frac{2\times\sqrt{9706-\frac{(152)^2}{45}}}{1136} =\frac{2\times\sqrt{9706-\frac{23104}{45}}}{1136} =\frac{2\times\sqrt{\frac{436770}{45}-\frac{23104}{45}}}{1136}$

$=\frac{2\times\sqrt{\frac{436770-23104}{45}}}{1136} =\frac{2\times\sqrt{\frac{413666}{45}}}{1136} =\frac{2\times\frac{\sqrt{413666}}{\sqrt{45}}}{1136} =\frac{\frac{2\times\sqrt{413666}}{\sqrt{45}}}{1136}$

$=\frac{2\times\sqrt{413666}}{1136\times\sqrt{45}} =\frac{\sqrt{413666}}{568\times\sqrt{45}} =\frac{\sqrt{413666}}{568\times\sqrt{3^2\times5}} =\frac{\sqrt{413666}}{568\times3\times\sqrt{5}}$

$=\frac{\sqrt{413666}}{568\times3\times\sqrt{5}}\times\frac{\sqrt{5}}{\sqrt{5}} =\frac{\sqrt{413666}\times\sqrt{5}}{568\times3\times\sqrt{5}\times\sqrt{5}} =\frac{\sqrt{413666\times5}}{568\times3\times5}$ $=\frac{\sqrt{2068330}}{8520} \approx0.1688$

Which gives a z-statistic of:

$Z=\frac{\gamma}{ASE} =\frac{\frac{19}{142}}{\frac{\sqrt{2068330}}{8520}} =\frac{19\times8520}{142\times\sqrt{2068330}} =\frac{19\times142\times60}{142\times\sqrt{2068330}} =\frac{19\times60}{\sqrt{2068330}}$

$=\frac{19\times60}{\sqrt{2068330}}\times\frac{\sqrt{2068330}}{\sqrt{2068330}} =\frac{19\times60\times\sqrt{2068330}}{\sqrt{2068330}\times\sqrt{2068330}} =\frac{19\times60\times\sqrt{2068330}}{2068330}$

$=\frac{19\times6\times\sqrt{2068330}}{206833} =\frac{114\sqrt{2068330}}{206833} \approx0.7927$

We can complete the report now by combining all the parts, which will be shown on the next page.

FAQ:
Q: Why did you choose gamma?
A: Since this site focusses on survey data, and often the ordinal variable is based on a 5 point scale (e.g. fully agree to fully disagree) there is a risk of having many ties. As suggested on the bottom of the page here, with many ties Goodman-Kruskal gamma is prefered above

Ordinal vs. Ordinal

Introduction

Impression

Visualisation

Test, and effect size

Reporting

Google adds