The San Francisco Declaration on Research Assessment (DORA) is a worldwide initiative aimed at improving the ways in which outputs of scholarly research are evaluated. DORA is a declaration which seeks to gain commitment among funding agencies, higher education institutions and journal organisations to promote and adopt the responsible use of metrics when measuring the evaluating research. In particular, the declaration seeks to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations. There is, instead, an encouragement to assess research on its own merits rather than on the basis of the journal in which it is published.

DORA includes a series of recommendations for researchers and for funding agencies, institutions, publishers, organisations that supply and use metrics.

Recommendations for research institutions are to:

1. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published. 
2. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative

Recommendations for researchers are to:

1. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics
2. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due
3. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs
4. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs

The recommendations included in DORA are also complemented by other initiatives such as the Leiden Manifesto (2015) and the establishment of the Forum for Responsible Research Metrics in the UK. Both of these initiatives advocate for a qualitative expert assessment and greater transparency in the use of metrics and introduce the notion of responsible metrics as a way of framing appropriate uses of quantitative indicators in the governance, management and assessment of research.

Taking these developments forward, Queen’s has become signatory to DORA and has developed a series of institutional principles on the use of research metrics. These principals will ensure that our approach to the use of research metrics is aligned fully to the Declaration.

1. Quantitative evaluation should support qualitative, expert assessment

The expert judgement and narrative context provided by peer review is a well-embedded part of the research process. Quantitative indicators, however, can be useful to challenge preconceptions and inform overall decision-making. While recognising that the balance between both approaches will vary by discipline, any adoption of quantitative indicators should be conducted in tandem with qualitative approaches in order to provide supporting information and help build a strong evidence base when assessing research quality.

2. A combination of indicators should be used

When adopting quantitative assessment, it is important that any approach seeks a variety of perspectives by adopting a suite or basket of indicators.  Research quality is multifaceted and cannot be captured by a single indicator used in isolation.

3. Research evaluation should have clear and strategic objectives

There should always be clear reasons for the incorporation of quantitative indicators in any research assessment and this approach should align with relevant School, Faculty and University strategies. The alignment between the evaluation approach and any relevant strategy must be clearly outlined in any analysis.

4. Differences between research disciplines should be acknowledged

Research practices across disciplines can vary widely with some indicators serving some disciplines better than others. Citations, for example, are largely relevant to journal and conference outputs rather than monographs or other forms of output, while international co-authorship can be less relevant to disciplines in which academics tend to publish alone. Any disciplinary biases in indicators used must be explicitly acknowledged and addressed. In line with best practice, indicators should be normalised whenever appropriate, and based on percentiles rather than averages where a single outlier can skew the numbers. The degree of availability of bibliometric data should not drive decision-making about research activities and priorities, either individually or collectively.

5. Data sources should be robust, accurate and open for verification

Source data should be made available where possible. Those subject to evaluation should be given details of how the information was sourced and analysed, as well as being offered guidance how to request corrections (where necessary) and how to access and verify the data. Quantitative indicators used should be easily reproducible.  Any limitations inherent in data sources or any potential factors that could bias interpretation of data must be explicitly acknowledged.

6. Research indicators and data sources should be regularly reviewed and updated

The systems of evaluation should be sensitive to the evolving needs of the institution and responsive to the changing nature of the research landscape. As the institutional understanding of quantitative indicators increases, the institution should seek to explore, review and enhance the measures used, provided the sources are robust, reliable, accurate and transparent.

7. A rounded and comprehensive assessment of research should be used

Performance against some indicators can be heavily influenced by career stage, gender and discipline, and any research assessment exercise should take these factors into account. Queen’s University recognises that academic staff undertake a wide range of research communication activities, not all of which can be easily measured or benchmarked.  When assessing research performance, it is important to provide as rounded and wide-ranging a picture as possible by capturing a comprehensive view of expertise, experience, outputs, activities, impact and influence.

For further information, please contact the Research Policy Office rpo@qub.ac.uk