Known methods for increasing the objectivity of experimental results
Method of increasing the objectivity of empirical research
D N Galiahmetova 1 , O A Feoktistova 1 , Ya A Shchenikov 1
1Saint-Petersburg State University of Aerospace Instrumentation, Bolshaya Morskaia str. 67A, Saint-Petersburg, 190000, Russian Federation
E-mail: yar2409@mail.ru
Abstract. The article describes types of cognitive distortions that can affect objectivity of empirical research and empirical data obtained. A self-assessment method for evaluating the experiment objectivity is presented, which can be used to self-assess the objectivity of empirical research. The developed method uses checklists at some stages of empirical research and allows reducing the research subjectivity by minimizing cognitive distortions that may arise due to deviations in the research or defects in the methods used.
Introduction
Companies involved in product development and production pay much attention to assessment and description of internal processes in order to improve their quality and objectivity. At the same time, one of the conditions for company successful development in the market is innovation. At the same time, minimal attention is paid to quality of experimental evaluation of innovative products, the quality of so-called “accurate experiments”.
According to a research published in the Harvard Business Review – 40-90% of new products don’t take top of the market. Consumers are usually conservative, and tend to overestimate advantages of an old product by 3 times, while manufacturers tend to overestimate the advantages of an innovative product by 3 times [1]. Innovative products can be overestimated by a manufacturer, for example, the “survivorship bias” – if product test is unsuccessful, then it will most likely not be published and will not be taken into account in a presence of other more successful tests.
In an empirical research, an experimenter can act as both an “on” and an “not on” observer. In the second case, the observer claims to be objective, especially if statistical methods were used to increase the obtained data objectivity. Any empirical research begins with the fact that researcher fixes severity of properties of interest to the object or objects of research, usually with the help of numbers. Thus, it is necessary to distinguish between objects of research (processes, objects, phenomena), their properties – what makes up the subject of research and signs that reflect severity of the properties. At the same time, it is extremely important for the experimenter to be aware that the accuracy with which the characteristic reflects the measured property depends on the measurement procedure. But is it that simple?
|
|
|
Every second human brain processes trillions of different processes. It is not surprising that brain is looking for patterns of behavior that can reduce the number of own processes, thereby saving more energy. However, in scientific activity, these acquired patterns of behavior can adversely affect the objectivity of research results.
Cognitive distortions are errors of thinking that a person makes when processing information. They occur when a person: remembers and recalls, processes a large amount of information, quickly responds to a situation, a person doesn’t have enough information (table 1).
Table 1 – Types of cognitive distortion
| Context | Cognitive distortion |
| Remember and recall | Store memories in different ways depending on the experience situation |
| Simplify events and lists to individual key points and elements | |
| We discard particulars for constructing and fixing generalizations | |
| Edit and enhance memories after events | |
| Quick response | We prefer simple-looking and unambiguous choices to be more complex and uncertain |
| They tend to maintain personal autonomy and current status in the group avoiding irreversible decisions | |
| We prefer to complete what we already invested in time and effort | |
| We prefer to focus on immediate and close results | |
| To act, one must be confident in an ability to change something and feel a importance of one’s actions | |
| We process a large amount of information | We more readily notice what we previously remembered or often met |
| Strange/funny/outwardly attractive/ anthropomorphic things attract more attention than usual / unfunny | |
| Notice when something has changed | |
| We are attracted to particulars that confirm existing beliefs | |
| It’s much easier to notice flaws in others than in ourselves | |
| We lack information | Discover stories and patterns even in poor data |
| We supplement information gaps with well-known attributes from stereotypes, generalizations and past experience | |
| We value more familiar/nice things and people | |
| Simplify numbers and probabilities to make them easier to think about | |
| We think that we know what others think | |
| Projecting our current mindset into past and future |
|
|
|
The subjectivity of empirical data acquisition tools was developed by such scientists as: Richard Gregory (the brain automatically completes the image according to his data and does the same with conclusions from incomplete information), R. Rosenthal (an experimenter, who believes in the hypothesis, unconsciously acts in such a way as to obtain results confirming the hypothesis) [2]. The topic was also developed by: Gerald Holton (a scientist was initially set up for what he wants to discover), Michael Polani (the concept of personal implicit knowledge), Mendeleev (measurement accuracy and the importance of metrology), Ernst Mach (data obtained by a person through experience – this is the only reality with which the person deals with, an analogy with the bus schedule), Moritz Schlick (principles of verification, tolerance and physicalism), Edward Thorndike and Phil Rosenzweig (Halo effect), as well as many contemporary authors: S.N. Tits, A.I. Khudyakov, A.D. Nasledov, L.V. Kopets, V.N. Druzhinin, P. Kline [3,4].
An empirical research is designed and implemented by a person – an experimenter, who, of course, can be affected by certain cognitive distortions. The effects encountered in the literature that affect the subjectivity of an empirical research are shown in table 2.
Table 2 – Effects affecting the subjectivity of an empirical research
| Effect | Demonstration |
| The Rosenthal effect is a subconscious attempt to adjust their research results to the desired theoretical scheme. It can appear in any science at any stage of empirical research. | Deformation of scale; the Texas sharpshooter fallacy; tendency to evaluate logical strength of an argument depending on belief in truth or falsity of conclusion; reassessment of value of a certain parameter in the experimental series; biased selection of data that contradicts the hypothesis, so as not to appear biased; sequence effect |
| Halo effect – the result of exposure to general impressions of something on perception of its particular features | Distortion in connection with the wording of a scientific law; framing effect – using too narrow approach to describe a situation or question; missing links between explanation and generalization; wrongful appeals to authority or to generally accepted fact; wrongful main cause allocation – a particular reason is taken as main |
| Excessive concern for success | Amplification – investing in achieving a goal more effort than necessary (attempt to kill a fly with a sledgehammer); bias towards information search; acceleration; advance |
| Reassessment of particular cases significance | Focusing effect; reassessment of a particular variable impact on overall result; deviation towards the result |
| Errors of experiment design | Lack of accounting for changes in an research object over time; associative confounding; selection (objects group nonequivalence in composition, which causes a systematic error); factors affecting sample size of objects; wrongful allocation of main cause – a particular reason is taken as main; statistical regression – a group of objects was selected on “extreme” indicators basis; zero risk preference; planning fallacy – the tendency to underestimate resources that need to be spent on tasks |
| Exaggeration of special cases probability | Contrast effect; the Baader-Meinhof phenomenon or the illusion of frequency; generalization of special cases |
| Reassessment of one's own opinion/position/choice/ opportunity significance | Congruence bias; irrational escalation; the phenomenon of expression of sympathy for the object of research only on the basis of existing acquaintance with it; professional deformation – look too narrow, discarding a more general point of view; retrospective tendency to ascribe positive qualities to an object or action that a person has chosen |
Known methods for increasing the objectivity of experimental results
|
|
|
Even in our time, the well-known methods for increasing the objectivity of research are often used only by psychologists, while representatives of other branches of science not only don’t use these methods, but may not be familiar at all with such a formulation of question as the significance of cognitive distortions impact.
|
|
|
Engineers typically consider the following errors: instrument errors, measurements, and processing. As a rule, their geometric mean is taken and they are considered statistically independent. The instrument error is determined by manufacturer and technical condition of the instrument. The measurement error is determined by observer eye. The processing error depends on how complex and multi-stage processing must be done with the data before the result is obtained. If the law is known or assumed, then it can be found from the formula. Thus, the more accurate measuring device experimenter has, the more accurate the results of an empirical research.
In 1994, standard ISO 5725-2:1994 “Accuracy (correctness and precision) of measurement methods and results” [5] was adopted. The standard is about methods for improving accuracy of measurements, about methods for statistical analysis of experimental data to assess precision, balancing results at uniform levels, but does not take into account the presence of cognitive distortions that can lead to experimental errors.
Science community recognizes two criteria on the basis of which it distinguishes scientific knowledge from pseudoscientific:
1. The principle of verification proposed by Bertrand Russell – only that knowledge is scientific, which can be confirmed.
2. The principle of falsification, proposed by Karl Popper – only that knowledge is scientific, which can be refuted.
To counter experimental errors, you can also use a number of methodological techniques:
3. The "placebo" method, which allows you to overcome the observer-expectancy effect.
4. The field experiment is an experiment that is conducted under usual conditions of research object existence with minimal intervention by an experimenter in these conditions. The disadvantage of this method is uncontrolled factors that influence the result.
5. The use of statistical methods for replicate observations.
Дата добавления: 2019-11-16; просмотров: 185; Мы поможем в написании вашей работы! |
Мы поможем в написании ваших работ!