You are in : Home » About the REF » Guidance and criteria » FAQs » Research outputs (REF2)

Research outputs (REF2)

The information on this page was published whilst the exercise was being conducted; this material is provided for background information only and will not be updated.

Latest FAQs (October 2013)

I want to submit a journal article but it does not have an ISSN. How should I submit this research output?

The journal article should still be submitted as output type 'D' = journal article. The ISSN field should state that this information is not available.

All research outputs should be submitted against the output type which an institution considers to best represent the research being submitted for assessment. The choice of output type should not be dictated by the metadata available for that output. If mandatory metadata does not exist for a particular output you should enter 'n/a' in the required submission system field. Wherever the DOI is not available for journal articles, this field should be left blank. 

Can I provide a URL as the research output when submitting output type 'H' = website content?
When submitting web content as a research output, URLs alone should not be submitted and you should provide a portfolio with material sufficiently substantial to constitute evidence which allow sub-panel members to assess the research dimensions of the work. This may mean a combination of media are used which could include, for example, screen shots from the website, images and photographs, a written description of the structure of the section and how it interacts with other sections, descriptive or contextual information to evidence the research process. You will need to decide whether it is necessary to provide a copy of the whole website or a representative selection. You should also provide evidence that this was indeed first publically available within the publication period.

Previous FAQs

Part2B of the 'panel criteria' (paragraph 56) requires the name and contact details of a senior industrialist in the output additional information field, where claims are made relating to the industrial significance of an output. Will this information be published?
All additional information for REF2 will be published as part of the submission information. It is not possible for HEIs to redact elements of the additional information field. Therefore, if it is not appropriate for the name and contact details of the individual to be published, you should instead state the individuals' position and company. If audited, the institution will be expected to provide contact details for the individual.
If two different chapters within an edited book are being submitted in the same UOA and we plan to submit a print copy of the book, do you require two copies?
If you are submitting two different chapters within a book as two separate outputs, you should provide two copies of the book.
Where a number of items are required to represent an output (i.e. a portfolio), can I upload a PDF to the submission system as well as providing additional evidence by deposit to the warehouse?
No. Each output must be provided either as a PDF upload or deposited as a physical output; not as a mixture of the two. A physical output may include multiple items to represent one output. It is not possible to provide some of the output as a PDF upload and the send another part of the same output as a 'physical' output.
Is it possible to rank reserve outputs if a staff member is submitting two outputs for which double-weighting is requested?
The submission system requires that a reserve output is associated with a specific output for which double-weighting has been requested. If the request is declined, the panel will assess the associated reserve output. However, where two double-weighting requests are made for an individual and the institution wishes to rank the two reserve outputs, it may use the additional information field in REF2 to indicate which of the two reserves should be assessed if only one of the double-weighting requests is accepted.
Will HEIs be reimbursed for any loss of research outputs?

Unless specifically requested by the REF team, artefacts, devices, products and items of great rarity or value should not be deposited. For these types of outputs, a representation of the output and sufficient information about the research it embodies should be provided to enable the panel to assess it, without providing the item itself.

The REF team will reimburse HEIs for outputs that are lost during the assessment phase. Arrangements will be made directly with the institution concerned where this is required.

What is the significance of the numbering of the outputs for a member of staff on the submission system? Is '1' the output of the greatest quality?
The numbering of research outputs has no significance and will play no role in the assessment of output quality. It is simply a means to identify the outputs and any associated information within the REF submission system.
Main Panel A requires information about an author’s contribution to the output where there are 6 or more authors, and the submitting author is not identified as either lead or corresponding author. How will the panel determine who the lead or corresponding author on a paper is?
In the disciplines covered by Main Panel A, many (but not all) journals have well established conventions for the order in which the lead and/or corresponding author are listed - typically as first or last author. Corresponding authors are also often indicated by a typographic mark and/or a footnote. Where these conventions are well established, the sub-panel will accept them as identifying the lead or corresponding author. If in doubt about the journal’s conventions, then it would be safest to use the 'additional information' field to state what the author’s role was.
How should e-books be returned to the REF?

Outputs of type A (Authored book) and type B (Edited book) cannot be uploaded to the submission system as a PDF and must be provided in physical form. For e-books, institutions may choose to submit the output on a CD, DVD or USB, or provide a printed hardcopy. For output type C (Chapter in book), a PDF of the book chapter (e-book or otherwise) may be uploaded to the REF submission system. In these instances the PDF should include a copy of the page(s) of the book that bear the title, publisher, editor and publication date.

The REF team will not source e-books submitted to the REF. They must be provided by the HEI as described above and it is not sufficient to provide only the URL for an e-book within REF2.

For the UOAs in Main Panel A, what information should be provided to establish the author's contribution to a multi-authored research output?
Main Panel A have requested standard statements about the individual's contribution to a co-authored output, where required (see paragraphs 34–37 of Part2A of the 'Panel criteria'). In addition, to assure themselves about an author's substantial contribution, the sub-panel may request further details of the specific contribution of the author, through an audit query. Main Panel A have indicated that such audit queries are especially likely for outputs with 15 or more authors (and where the individual is not the lead or corresponding author).
In the case of an edited work, where the editor has made several contributions (for example, the introduction, a substantive chapter and the conclusion), can these be submitted as a single output? Or could they, if preferred, be submitted as separate outputs?
The edited work could be submitted as a single output or alternatively, could be submitted separately if they are coherent as separate outputs. If returning them as separate outputs, you should consider whether they have significant material in common. (Further guidance on how outputs with significant material in common will be assessed is available in Part2 of the 'panel criteria'.)
When matching research outputs to Scopus within the REF submission system, what should I do if the electronic and hard copy versions of the same paper return different citation counts?
Where Scopus returns multiple matches for different formats of the same output, you should not confirm the match and should contact Scopus user support (via the REF submission system). Scopus user support will be able to create a single record for the output which consolidates citations for both formats. You will then be able to match this new record against the output within the submission system.
If an individual is eligible for inclusion by multiple HEIs, can each HEI submit that individual with the same research outputs?
Yes. As stated at paragraph 79f of the 'guidance on submissions', an individual may be returned as Category A by more than one HEI if they have a contract with and receive a salary from more than one HEI. The same research outputs may, but need not be, listed in each submission.
How should eligible 'online first' journal articles be returned to the REF?

Online first journal articles should be returned as output type 'D - Journal article'. The REF team recognises that some of the meta-data may not be available for an online first publication, but sufficient information must be provided to enable the REF team to identify the output.

There may be some fields for which data is not available, that are mandatory for type D outputs (e.g. first page, volume number). These fields should be completed with ‘n/a’ to allow the submission to be made. You will not be able to submit without completing these mandatory fields.

For a research output that is a product, what is considered the date it first became publicly available?
If submitting a product (output type = ‘P’) the date that the product was produced would be considered the date it became publicly available in the form in which it is submitted (see the ‘Output information requirements’ spreadsheet, available for download on the 'Submission system data requirements' page ).
In what circumstances can a related series of items be submitted as a single research output?

Separately published papers are discrete outputs and cannot be grouped together as a single output. Substantial dictionary or encyclopaedia entries and groups of short items including groups of entries may be submitted as a single output (see paragraph 51, Part2D of the ‘panel criteria’).

The following additional guidance was added in November 2013:

Where research has been split by a publisher and would only be considered a single coherent work when viewed together, these should be returned as one research output. 

What output type should a 'portfolio' be submitted as?
A portfolio is not an output type in itself but is the format in which the evidence of certain outputs can be submitted (see paragraph 59 of Part2C and paragraph 71d, Part2D of the ‘panel criteria’.) For example, an output that is a performance or exhibition may be submitted in the form of a portfolio of evidence. Where submitting a portfolio, the most applicable output type should be used, and the ‘media’ field should be used to state that a portfolio will be submitted (see the ‘Output information requirements’ spreadsheet, available for download on the 'Submission system data requirements' page).
How will a research output submitted multiple times, to the same or different Units of Assessment (UOAs), be assessed?

Where a research output is submitted multiple times to the same UOA, it will normally be allocated to the same individuals on the sub-panel and reviewed only once (although the additional information may differ). Please note, an output may only be submitted a maximum of 2 times within one submission.

If a research output is submitted to multiple UOAs, it will be assessed by each of the sub-panels according to their respective criteria, including in terms of its significance to the disciplines concerned. The output will not, therefore, necessarily receive the same quality score.

What is the outcome if a sub-panel considers an individual has not made a substantial contribution to a co-authored research output returned against them?
If the sub-panel considers that the individual may not have made a substantial research contribution to the output and there is not sufficient evidence in the submission to make an informed judgement about this, the sub-panel will raise an audit query. The HEI will then be asked to explain the individual’s research contribution, and provide any appropriate evidence. The sub-panel will consider all the information provided before deciding. (Note that in situations where this information has already been requested as part of the submission, for example for sub-panel 9 where there are more than 10 co-authors, the sub-panel will normally decide on the basis of the submitted information without any further audit query.) If the sub-panel does not accept that the individual made a substantial research contribution, the output will be graded as ‘unclassified’ (see paragraph 126e, Part 1 of the ‘panel criteria’.)
If an output pre-published in 2007 was submitted to the RAE 2008 by a different institution or in a different UOA, can it be submitted in final form to the 2014 REF?
See paragraph 43 of the ‘panel criteria’. The final proviso should be interpreted to mean that such an output is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE by the same institution, or was not listed against the same member of staff (if they were in a different institution at the time).
Are 'online first' journal articles that are not published in hard copy by 31 December 2013 eligible for inclusion in the REF?
Yes, 'online first' and other pre-published outputs that are in the public domain during the publication period (1 January 2008 – 31 December 2013) are eligible for submission.
What output information is required for patents and patent applications?
Both a granted patent and a published patent application are eligible as research outputs. If you are submitting a granted patent you should include the patent number and date of the granted patent; if you are submitting a published patent application you should include the publication number and the date that this was published. Where a patent application was submitted to the 2008 RAE, the granted patent is not eligible for return to the REF.
How will journal impact factors, rankings or lists, or the perceived standing of publishers be used to inform the assessment of research outputs?
No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs. An underpinning principle of the REF is that all types of research and all forms of research outputs across all disciplines shall be assessed on a fair and equal basis.
What kind of outputs can be categorised as a 'working paper' (Output type = 'U')?
A working paper is often written in the style of a journal article, is usually made freely available on line, may not have been peer reviewed, and has not yet been formally published (typically, in a journal). Working papers will not have a Digital Object Identifier. The use of working papers is more prevalent within certain research communities, for example ArXiv is often used in particular disciplines. The output must still meet the eligibility definitions described in 'guidance on submissions'.
Can an output authored by a member of staff before they became eligible for submission to the REF (or before they met the definition of an early career researcher) be listed against that member of staff?