Self-assessment of researcher’s objectivity in conducting empirical experiments

⇐ ПредыдущаяСтр 2 из 4Следующая ⇒

As mentioned earlier, the cause of experimental errors can be not only an chosen evaluation procedure, but also “cognitive distortions” – common, subconscious, thinking errors that we all systematically make. Such errors are inherent in most people and predictable.

The following is a method to increase the objectivity of experimental results, which helps to take into account the maximum number of known cognitive distortions that affect result of an experiment at all stages of empirical research lifecycle.

Method for increasing the objectivity of experimental results

Step 1. Design of experiment. Setting goal of the experiment. Choice of factors and response

After setting goal of the experiment, input and output parameters are identified and selected based on collection and analysis of background information. Factors can be controlled and uncontrolled (random), introducing a systematic or random error in the measurement results: instrumental errors, changes in properties of investigated object during experiment, the impact of personnel, etc.

Step 2. Application of checklist No. 1 and adjustment of the experimental plan

Table 3 – Checklist No. 1, devoted to distortions arising during experiment organization

We ask you to read 2 statements and mark on the scale which one and how much reflects your activity during planning and organization of the current experiment

Statement 1

Statement 2

Distortion example

An empirical research is planned in face of a lack of a priori data

An empirical research is planned to be carried out after a detailed research of all factors that can affect the result

Experimenter didn’t study literature enough on the problem under research, did not conduct a survey of experts and specialists related to object of research. Did not conduct preliminary univariate and screening experiments

When planning an empirical research, no analysis of noise factors was carried out and their possible impact on the accuracy of experiment was not evaluated

When planning an empirical research, noise factors were analyzed and their possible impact on the accuracy of experiment was evaluated

When studying impact on something different areas of animal brain on its behavior, the very fact of undergoing surgery to remove corresponding area can change the experimental animal behavior

Change in the characteristics of a object of research over time was not taken into account

It was taken into account that over time, significant changes can occur with a object of research

When longitudinal studies of groups of animals should take into account their life expectancy

No attention was paid to ensuring the groups homogeneity of research objects

The uniformity of the groups of objects of research was ensured

Main and control group of objects should correspond to each other with the exception of those properties that experiment aims to identify

Only direct verification methods are included in the experiment

Indirect methods were used additionally together with methods of direct verification

Congruence bias. The subject was presented with two buttons, pressing one of which (left) opens the door. A direct test is a left-click; indirect test – pressing the right button. Since the door remains closed when you press the right button, you can indirectly conclude that you need to press left button

Such experimental research methods were selected that guarantee a high probability of a favorable research outcome

To solve the problem, various methods were used, including those as a result of which probability of a research favorable outcome was unknown

The ambiguity effect suggests that people tend to choose a solution with known probability of a favorable outcome, compared with a solution where the probability of a favorable outcome is unknown

It considered a model that was popular in a specific subject field, and not in an interdisciplinary field

The model was considered on the basis of a more general, interdisciplinary point of view

Table 4 – The scale for assessing objectivity of an empirical research at planning stage

Total points	Objectivity	Corrective actions
7-18	Low	Revision of the experimental design
19-29	Medium	Selection of cognitive distortions with worst objectivity indicators and experimental design adjustment in order to improve these indicators
30-35	High	Go to the next step

Step 3. Research experiment organization (for example, choice of instruments, choice of personnel)

Clarification of the experiment conditions: instruments, timing, financial resources, executors, etc. Determination of required accuracy of measurement results (output parameters), possible variation of input parameters, types of effects clarification. Type of samples or objects to be studied is selected, taking into account the degree of their compliance with a real product in: condition, structure, shape, size and other characteristics. The accuracy of experimental data also substantially depends on the number of experiments – the more experiments (under the same conditions), the higher reliability of the results. The design of the experiment – the number and order of tests, a method of collecting, storing and documenting data.

Step 4. Conducting an experiment using checklist No.2

Table 5 – Checklist No. 2, devoted to distortions arising during experiment

We ask you to read 2 statements and mark on the scale which one and how much reflects your activity during the current experiment
Statement 1						Statement 2	Distortion example
It was allowed to transfer the characteristics of the sample to other samples or to the general population	1	2	3	4	5	Statistical methods checked presence in other samples or population selected in a particular case of characteristics	Based on a positive test result of a small sample of electronic components from a batch, a decision was made about suitability of the entire batch
It affected conditions of experiment after obtaining experimental data after the first series of the experiment	1	2	3	4	5	The impact on the experimental conditions was minimized after obtaining the experimental data after the first series of experiment	When conducting a series of experiments, having received “good” and “bad” data in the first series of the experiment, I tried to “improve” the first and “deteriorate” the second in the next series
When conducting an empirical research timing of its implementation was not respected	1	2	3	4	5	The empirical research was conducted in accordance with the originally developed schedule	"Walking through the minefield" when trying to perform an empirical research as soon as possible, however, attempts to work faster lead to frequent errors
Influence of one of experimental conditions on subsequent ones was allowed	1	2	3	4	5	It was not allowed to influence one of the experimental conditions on the following experiment	If object of research is a person or an animal, then they can react in different ways to same conditions. If during the experiment researcher changes order of its conduct, then adaptation of the subject to this experiment will take place in different ways
Value of one of parameters was underestimated in comparison with a newly discovered contrasting parameter	1	2	3	4	5	Taken into account impact of contrast effect in object parameters analysis	Underestimation one of the factors significance with statistically proven significance or even its removal from model when a factor is detected whose impact is higher
When one recorded situation guarantees the content of another, then only the last was reported in the research report	1	2	3	4	5	The results of the empirical research are fully and in detail described in the report	Substitution of one concept with another may occur. Researcher may uncritically assume that one situation recorded by him guarantees content of another and report only the last situation

Table 6 – The scale for assessing objectivity of an empirical research at the stage of its implementation

Total points	Objectivity	Corrective actions
6-15	Low	Conduct repeated experiments using other measuring instruments, experimenters, measurement methods. Increase number of considered factors. Choose measuring instruments with increased accuracy and noise immunity. Use randomization of the tests. Take additional measurements to test hypothesis about type of mathematical model. Increase sample size
16-24	Medium	Increase the number of experiments, additional measurements. Choose a more complex math model. Repeat experiment with another experimenter.
25-30	High	Go to the next step

Step 5. Statistical processing of experimental results

Statistical processing of the experimental results is processed with identification and elimination of blunder and systematic errors.

Step 6. Application of checklist No.3 and statistical processing of the survey result

Table 7 – Checklist No. 3, devoted to distortions that occur during processing of results

We ask you to read 2 statements and mark on the scale which one and how much reflects your activity while processing results of the current experiment
Statement 1						Statement 2	Distortion example
Only experimental data were used that confirm the hypothesis of an experimental research	1	2	3	4	5	To confirm the hypothesis of the experimental research, all obtained experimental data were used	When processing measurement results procedure of statistical verification of “abnormal” data was not carried out, they were discarded as a “blunder” error
Methods convenient for hypothesis confirmation were used	1	2	3	4	5	The most objective methods were used to confirm the hypothesis	Using the same data, the use of different statistical methods can lead to different results. It is important not to fall to temptation to use methods convenient for confirming a hypothesis, but to give preference to most objective methods
To find the model parameters, only similar data were used and the “outliers” were ignored	1	2	3	4	5	To find the model parameters, not only grouping data was used, but also “outliers”	You can shoot at the wall, and only then, in the place where the largest number of holes appeared, draw a target
Intuition was used to assess the significance of factors	1	2	3	4	5	The significance of factors was estimated using standard statistical techniques	The experimenter neglected the preliminary and screening experiments
When analyzing the results, more importance was attached to data obtained after first series of experiments than subsequent	1	2	3	4	5	Information obtained after the first series of experiments and subsequent information are of equal value	Primacy effect – the experimenter unconsciously pays more attention to the data obtained after the first series of experiments, focuses on them and based on them builds a certain scheme in his head in advance
The tendency to look for common features among the "survivors" and to lose sight of fact that no less important information is hidden among the "dead"	1	2	3	4	5	There is no tendency to underestimate data that are not accessible to direct observation from the group of "dead"	Survivorship bias. For example, holes on returning planes show places where they can get damaged and survive, and those who got damaged in other places could not return to base
Assessing frequency or possibility of an event by the ease with which these examples or cases are recalled	1	2	3	4	5	Frequency or likelihood of event occurrence was estimated by results of processing statistical data	Availability heuristics. For example, a person estimates the risk of heart attack in middle-aged people, recalling similar cases among his friends
When analyzing the results of an experimental research, already occurred events were perceived as obvious and predictable	1	2	3	4	5	When analyzing the results of an experimental research, possible alternatives for explaining the event were considered	Hindsight bias (“knew-it-all-along” effect). Physicians, as a rule, overestimate their ability to predict outcome of a medical case, stating that they knew the result in advance
With known frequency and specifics of some event there is a tendency to ignore the first event and focus on the second	1	2	3	4	5	With a known frequency and specifics of events, both are taken into account	John wears gothic clothing. How likely is he to be a Christian or a Satanist? If people ask this question, they will underestimate the likelihood that he is a Christian (although there are about 2 billion Christians), and overestimate the likelihood that he is a Satanist (according to statistics several thousand people are Satanists)
The procedures for constructing and verifying the model were carried out using the same experimental data	1	2	3	4	5	The procedures for constructing and verifying the model were carried out using experimental and test data	The experimenter decided to save on experiments and not split the sample into experimental and test
When processing the empirical research, the difference in the type of scales used was not taken into account, too high results of measurements were underestimated and too low ones were overstated	1	2	3	4	5	When processing the results of the empirical research, too high measurement results were not underestimated and too low measurement results were not overstated	When working with values measured on different scales, certain rules must be followed, otherwise blunders are inevitable. The use of unreasonable methods for determining average values can lead to artificial overstatement or underestimation of average value of a system quality indicator
I emphasized a number of measurements in the experimental series	1	2	3	4	5	The value of certain data obtained experimentally wasn’t artificially underestimated / overestimated
Results of the experiment were evaluated on basis of belief in truth of hypothesis	1	2	3	4	5	When evaluating the results of experiment, opinion of an expert not aware of hypothesis was additionally taken into account
The evidence supporting the hypothesis may be an established or generally accepted fact	1	2	3	4	5	Established and generally accepted facts may not constitute evidence supporting the hypothesis. Reputable scientists can also be wrong	As a sufficient or significant conclusion a statement is proposed that evidence supporting any point of view is an established or generally accepted fact. Instead of validated arguments for evaluating content of hypotheses, a reference is made to opinion of an authority.

Table 8 – Scale for assessing the objectivity of an empirical research at the stage of constructing a model of the process/phenomenon

Total points	Objectivity	Corrective actions
14-35	Low	Redesign the experiment so that the same data is not used to build model and verify its predictive properties. Pay attention to the “anomalies” in the data, try to find the reason for their appearance
36-54	Medium	Researched process/phenomenon may be more complex than you expected, and more complex models should be used to describe them
55-70	High	Your empirical research is objective enough

Conclusion

This article identifies a problem associated with the lack of a regular practice of taking into account experimenter subjectivity in conducting empirical research. A method is proposed for self-assessment of research objectivity, which includes checklists filled out by experimenter and allowing to reduce subjectivity of the research by minimizing cognitive distortions.

It can be concluded that the method proposed in this article is able in practice to facilitate work of any experimenter who has set himself task of conducting an empirical research that is as objective and free from cognitive distortions as possible. This technique allows you to take into account the most common cognitive distortions and minimize their effect on the results of the experiment.

The problem of the objectivity of empirical research still open, the version of method proposed in this article, of course, is subject to discussion.

As part of future research, further refinement of the scales is necessary, the development of standard norms for a larger number of samples. Prospective approach is the further development of diagnostic capabilities developed checklists in the method.

Bibliography

[1] Gurvil' D. Psihologiya otnosheniya k novoj produkcii // Harvard Business Review. 2006. October. p. 77-86.

[2] Rosenzweig, Phil. The halo effect: ...and the eight other business delusions that deceive managers. – 1st Free Press trade pbk. – New York, NY: Free Press, 2007.

[3] Ernst Mach. Knowledge and Error: Sketches on the Psychology of Enquiry. Springer Science & Business Media. 2012. 394p.

[4] Michael Polanyi. Personal Knowledge: Towards a Post-Critical Philosophy. University of Chicago Press. 2015. 464p.

[5] ISO 5725-2:1994. Accuracy (trueness and precision) of measurement methods and results – part 2: basic method for the determination of repeatability and reproducibility of a standart measurement method. ISO/TC 69/SC 6. 42p.

Дата добавления: 2019-11-16; просмотров: 207; Мы поможем в написании вашей работы!

Поделиться с друзьями:

⇐ Предыдущая 123 4 Следующая ⇒

Мы поможем в написании ваших работ!