Two fundamental virtues to look for when using information in making decisions - they are reliability and validity. Both are closely related yet different in what they represent. Here is a simple explanation of the two using a common example
Consider a weighing machine that a doctor uses to weigh patients to decide the dosage of medicine to prescribe. The doctor needs the scale to record the exact weight every time patients stand on it. Let’s assume the scale sometimes registers weights that are less than the actual weight of patients and at other occasions more than the real weight, perhaps because of not being calibrated to start at 0 Kg each time it is used. A patient seeing the doctors at two different times would have two weights recorded on the scale. You can see how the scale can mislead the decisions the doctor makes concerning dosages for this patient. The scale is consistently failing to measure the correct weight of this patient. Once the doctor is aware of that, he or she would discard the scale, knowing it is no longer reliable. Reliability, therefore, is about consistency to record the exact measurement every time the scale is used.
Consider a weighing machine that a doctor uses to weigh patients to decide the dosage of medicine to prescribe. The doctor needs the scale to record the exact weight every time patients stand on it. Let’s assume the scale sometimes registers weights that are less than the actual weight of patients and at other occasions more than the real weight, perhaps because of not being calibrated to start at 0 Kg each time it is used. A patient seeing the doctors at two different times would have two weights recorded on the scale. You can see how the scale can mislead the decisions the doctor makes concerning dosages for this patient. The scale is consistently failing to measure the correct weight of this patient. Once the doctor is aware of that, he or she would discard the scale, knowing it is no longer reliable. Reliability, therefore, is about consistency to record the exact measurement every time the scale is used.
Validity relates, to the measurement being what it is meant to be. In the case of the example used here, validity is recording the actual weight of the patient standing on the machine. The doctor on recognising the lack of calibration in the scale ensures the scale started at 0 Kg the next time it is used. Assuming all other things are working as they should, the scale would record weight that is valid. It can happen that the scale is off by 5 kg every time it is used, perhaps because of the spring not positioned correctly. The scale consistently produced the same weight each time the patient stands on it but is off by 5 kg. You can see the scale appears to be reliably but the weight recorded is not valid. Thus having reliability does not ensure validity to be assumed, they need to be separately established.
Tool kit for examining reliability and validity
It is up to individuals to be fully satisfied with reliability and validity before relying on the information. Without being too scientific, reliability and validity can be assumed when similar results can be obtained from different sources within a given time frame. Below are few questions to ask when examining reliability and validity, particularly when the information is from a small group of individuals as in the case of surveys, website visitors or customers who fill in a satisfaction form. The providers of the information must provide answers to all the questions and individuals must be satisfied with the answers. To help know if the answers are satisfactory, the nature of the answers deemed acceptable when it comes to reliability and validity is mentions against each question.
- What is the margin of error reported? The margin of error places a range around the figure or estimate. A narrow range is indicative of precision. As the range gets wider, the precision becomes fuzzy, which should raise concerns.
- What is the sampling technique used? The sample technique needs to be probability-based to allow generalisation of the findings.
- What is the sample size? Results based on small samples (perhaps less than 500) should be treated with caution, particularly when results are reported for subgroups.
- What is the sampling frame used to draw the sample? The sampling frame must relate to the target population of interest.
- What is the methodology used? The methodology must support probability based sampling techniques.
- How representative is the sample to the population? The results must include a demographic makeup of the sample, showing comparison to the standards such as the Census data.
- What is the response rate? A low response rate is indicative of poorly conceived methodology been used.
- How credible is the source? Assess the reputation of the organisation responsible for the information.
- What is the wording of the question used to collect the information in the case of surveys? Question working must be established through scientific investigation.
No comments:
Post a Comment