From POERUP - Policies for OER Uptake
Jump to: navigation, search

Many but by no means all benchmarking methodologies applied to e-learning accept that the level of applicability of each benchmarking criterion should be ranked on a short numeric scale. This is often 1-5 but scales of 1-6 and 1-4 are also found. (Shorter scales do not give enough discrimination, longer scales have been found to be too complex to use.)

Some methodologies try to disguise scoring a little - in particular eMM with its use of colours and phrases such as adequate. Motivated by the example of eMM, other methodologies are tending to adopt colours also - including as a presentation device.

There are a few methodologies such as some variants of MIT90s which do not accept the idea of scoring.

Even the devotees of scoring accept that a raw score is not very helpful and so all scores should be accompanied by a scoring narrative. This could be one or more paragraphs and is not necessarily just one sentence.

How many points on the scale?

The 5-point scale is in some sense the norm. Its justification seems partly to derive from MIT90s theories of transformation levels, augmented by the general approach to 5-point scales in social science research derived from the work of Likert. A particular advantage of a 5-point scale is that scale point 3 denotes a convenient mid-point where one can be a bit vague about deciding whether this is good enough or not.

However 5-point scales are not universally favoured by the scoring devotees. A number of methodologies including Pick&Mix and some favoured by the DfES for use in schools and further education use a 6-point scale - the point 6 is normally used to denote "excellence".

Some methodologies derived from concepts of quality favour a 4-point scale. These include eMM and OBHE. In such scales, there is a clear distinction between "good enough" and "not good enough" since there is no mid-point.

Some older methodologies such as BENVIC used scales with 3 points but this appears to be gradually decreasing, presumably because the 5-point scale is now so entrenched in social science research and government statistics. However, there can be an argument for using a 3-point scale for presenting benchmarking results as it maps neatly into the traffic lights approach.

One thing almost all experts agree on is that adding and/or averaging a set of scores rarely makes any sense. However, it is often done in the understructure, for example to generate process scores from practice scores in eMM. In some cases it is also favoured for presentational purposes.

>> Glossary