How did we managed reliability?

With our first coding contract, we had to get accustomed to the categories and indicators of the cognitive presence of the model. As stated earlier, it took us three tries to finally be able to have reasonable reliability (generally between 75% and 95% agreement between both coders).

We coded, discussed our coding results and talked about how we had each applied the rubric then went back and recoded the whole discussion with our new understanding of the categories and indicators. Eventually our reliability got much higher.

We recorded all our coding result in an Excel spreadsheet, as it was easier to record our result (see example) and eventually compare them. (see example note: the red represent our differences). It was easier also to send files back and forth using different versions (e. i. v-1, v-2) ) for each 'bout' of coding.

After coding, we reported our results (both codes and reliability rates) to the investigator. Occasionally, at the request of the investigator, we reviewed our coding in order to see if reliability could be increased. In these cases we met to review those messages upon whose coding we disagreed. We then had a "debate" about the meaning of the message and the most appropriate code. In some instances, when we could come to an agreement on the most appropriate code for a given message, we were able to report agreement. This was not always the case, and when an agreement could not be reached between the coders, we allowed the disagreement to stand.

Another approach to reliability was the review of "zero codes". When one coder recorded a zero (no value) and the other recorded a value from the rubric, for a particular message, we reviewed the message. If the "zero value" coder was able to see the value attributed to the message by the other—than value was taken, thereby affecting coding reliability.

All reliabilities were reported in terms of percentage agreement between coders.