Mitigating Bias in Scientific Evaluation and Decision-Making Processes

Related guidelines:

The heads of HEIs and non-HEI research institutions create the basic framework for research. They are responsible for ensuring adherence to and the promotion of good practice, and for appropriate career support for all researchers. The heads of research institutions guarantee the necessary conditions to enable researchers to comply with legal and ethical standards. The basic framework includes clear written policies and procedures for staff selection and development as well as for the support of early career researchers and equal opportunities.

The head of each HEI and non-HEI research institution is responsible for ensuring that an appropriate organisational structure is in place at the in- stitution. He or she makes certain that the tasks of leadership, supervision, quality assurance and conflict management are clearly allocated in accordance with the size of individual research work units and suitably communicated to members and employees.

With regard to staff selection and development, due consideration is given to gender equality and diversity. The relevant processes are transparent and avoid implicit bias as much as possible. Suitable supervisory structures and policies are established for early career researchers. Honest career advice, training opportunities and mentoring are offered to researchers and research support staff.

The head of a research work unit is responsible for the entire unit. Collabo- ration within the unit is designed such that the group as a whole can perform its tasks, the necessary cooperation and coordination can be achieved, and all members understand their roles, rights and duties. The leadership role includes ensuring adequate individual supervision of early career researchers, integrated in the overall institutional policy, as well as career development for researchers and research support staff. Suitable organisational measures are in place at the level of the individual unit and of the leadership of the institution to prevent the abuse of power and exploitation of dependent relationships.

The size and the organisation of the unit are designed to allow leadership tasks, particularly skills training, research support and supervisory duties, to be performed appropriately. The performance of leadership tasks is associated with a corresponding responsibility. Researchers and research support staff benefit from a balance of support and personal responsibility appropriate to their career level. They are given adequate status with corresponding rights of participation. Through gradually increasing autonomy, they are empowered to shape their career.

To assess the performance of researchers, a multidimensional approach is called for; in addition to academic and scientific achievements, other aspects may be taken into consideration. Performance is assessed primarily on the basis of qualitative measures, while quantitative indicators may be incorporated into the overall assessment only with appropriate differentiation and reflection. Where provided voluntarily, individual circumstances stated in curricula vitae – as well as the categories specified in the German General Equal Treatment Act (Allgemeines Gleichbehandlungsgesetz) – are taken into account when forming a judgement.

High-quality research is oriented towards criteria specific to individual disciplines. In addition to the generation of and critical reflection on findings, other aspects of performance are taken into consideration in the evaluation process. Examples include involvement in teaching, academic self-governance, public relations, and knowledge and technology transfer; contributions to the general good of society may also be recognised. An individual’s approach to research, such as an openness to new findings and a willingness to take risks, is also considered. Appropriate allowance is made for periods of absence due to personal, family or health reasons or for prolonged training or qualification phases resulting from such periods, and for alternative career paths or similar circumstances.

Fair behaviour is the basis for the legitimacy of any judgement-forming process. Researchers who evaluate submitted manuscripts, funding proposals or personal qualifications are obliged to maintain strict confidentiality with regard to this process. They disclose all facts that could give rise to the appearance of a conflict of interest. The duty of confidentiality and disclosure of facts that could give rise to the appearance of a conflict of interest also applies to members of research advisory and decision-making bodies.

The confidentiality of third-party material to which a reviewer or committee member gains access precludes sharing the material with third parties or making personal use of it. Researchers immediately disclose to the responsible body any potential or apparent conflicts of interest, bias or favouritism relating to the research project being reviewed or the person or matter being discussed.

In decision-making processes, it is important to identify and mitigate both conscious and unconscious (or “implicit”) bias. The term “bias” denotes cognitive distortions, pre‑formed patterns of thought or predispositions that are automatically present to varying degrees in every individual and facilitate rapid, automatic processes of judgement. As research shows, however, they can distort evaluation and decision‑making processes both positively and negatively, and may therefore lead to discrimination – for example on the grounds of gender, origin, disability or chronic illness. Even in science and the humanities, which are by definition committed to facts and evidence, it is impossible to rule out bias. Decision‑making processes affected in this way are not science‑driven, however.

Guideline 3(interner Link) of the Code of Conduct builds on this point and calls for transparent procedures in staff selection and development, avoiding implicit bias as much as possible.

In research design, too, this can be an issue – according to Guideline 9(interner Link), applying methods to mitigate bias can form part of good research practice.

Guideline 16(interner Link) identifies neutrality of review processes and discussions as an important basis for credibility, fairness and quality in scientific decision‑making processes.

Examples of bias

One example of “status bias” or the “halo effect” is provided by Huber et al. (2022). The authors demonstrate that the reputation of researchers can influence assessment in the peer review process. They submitted an article jointly authored by a renowned Nobel laureate in economics and a less well‑known researcher to a behavioural economics journal. The manuscript’s success with reviewers varied depending on the information provided about the authors: when the Nobel laureate was named as the author, the identical manuscript received 77 percent approval; when the lesser‑known researcher was named, approval dropped to 35 percent; without any author names, approval was 52 percent.

In another study, Murray et al. (2019) found that in invitations to submit full articles to a biology journal, manuscripts by male authors were preferred over those by female authors for publication in the journal in question. The difference was even greater when the review panel consisted exclusively of men.

Mitigating bias

In order to counteract distortions in judgement caused by non‑scientific factors, it is important to be aware of these processes and to engage in continuous critical self‑reflection in selection and decision‑making contexts.

The DFG provides a range of resources that are aimed at raising awareness of bias in scientific evaluation and decision-making processes:

These resources are primarily aimed at DFG reviewers and committee members, but their content can also be applied to other scientific evaluation and decision-making processes such as peer review procedures, appointment procedures and recruitment processes. For this reason, use of these resources is open to all who wish to work towards avoiding bias.