You are in : Home » FAQs » ALL FAQs about the Research Excellence Framework
All FAQs about the Research Excellence Framework (REF)
Latest FAQs (April 2013)
I have tried to retrieve a citation count for a research output and get the result 'journal not indexed'. Can Scopus add this journal to their coverage so that I can retrieve a citation count?
Scopus has a Content Selection and Advisory Board which advises Elsevier on new journals which should be added to the database. Further details on the criteria which the Board uses to consider new material can be found on the Scopus website (http://www.info.sciverse.com/scopus/scopus-in-detail/facts). It is unlikely that the request to add new journals to the Scopus database will be resolved within the REF submission phase and therefore a citation count will not be returned. This does not preclude an institution from including the research output in their REF submission.
I have run the report 'Journal articles REF have been unable to retrieve' but it is only showing a small number of articles. Does this mean that REF have obtained almost all of the journal articles within a submission or is the report malfunctioning?
To ensure that this report is generated correctly and that all outputs of type D and E which need their journal articles supplied are listed, no outputs within that submission should be removed and re-added or have their DOI changed for a period of three weeks before the report is run. If some outputs have been altered in this way the three week count held against them will reset and they will appear on the report later than the outputs which have not been modified.
I have waited three weeks after adding my journal articles’ DOIs to the submission system and there is still no data within the report 'Journal articles REF have been unable to retrieve'. Why is this?
In this instance there are two reasons why the report 'Journal articles REF have been unable to retrieve' may not contain any data. If within the three week period, outputs have been removed and re-added (for instance by clearing a submission before importing updated data) the submission system will reset the three week count. To enable this report to generate correctly, outputs should not be removed (cleared) and re-added to the submission system three weeks before the report is run.
It is also possible that REF has managed to locate all of the outputs' journal articles within a submission, meaning that none need to be provided. If the submission's outputs and their DOIs have been present within the system for three weeks (i.e. not removed and re-added) this report will show correctly (which may be blank).
How should institutions calculate the additional environment data on postgraduate research students, requested by sub-panel 8 and sub-panel 26?
Both these sub-panels require data for students registered for a research doctoral degree and who are actively pursuing their research programme. Institutions should supply the total FTE of students active in the academic year by summing the HESA Student Instance FTE (STULOAD) field for students that have a HESA General qualification aim of course (COURSEAIM) of ‘D00: Doctorate degree that meets the criteria for a research-based higher degree’ and ‘D01: New Route PhD that meets the criteria for a research-based higher degree’ and that are in the HESA session population, for each of the academic years 2008-09 to 2012-13.
Where students have changed their mode to writing-up within the same academic year, institutions should include their writing-up activity for that academic year in the FTE figure reported for that academic year (there should be a single total FTE of students actively pursuing their research programme, to include students who have changed their mode to writing up during the year). Where a student has been writing-up for the whole of the academic year, institutions should exclude the student from the FTE figure reported for that academic year.
Where you are including the writing-up activity, please use the HESA guidance to calculate this:
The FTE allocated to writing-up students should be 10% for both full and part-time students. The 10% applies where the student has been writing-up for a full year. Where a student has only been writing up for part of the year the FTE should be adjusted accordingly. So, where a student does 6 months full-time study and then 6 months writing up within an academic year, they would be recorded as 0.55 FTE.
Can an institution change its code of practice after the document has been approved by the funding body?
If an HEI identifies an exceptional need to make significant changes to the content of its code of practice (COP) after it has been approved by the funding body, the HEI must provide a revised COP to the REF team, along with a covering letter from the head of institution. This letter should:
Outline how the COP has been amended and the reason for the amendment.
Confirm that the changes have not reduced the extent to which the COP adheres to the published guidance.
Provide details of how the institution will communicate the changes to all relevant staff with ample time before final selection decisions are made.
The letter and revised COP should be uploaded to the REF extranet and an e-mail should be sent to email@example.com to notify the REF team that this has been done.
For minor changes to a COP, for example a change in an individual's membership of a committee, we will not require the HEI to follow the above procedure. Any changes made to selection procedures, the terms of reference of a committee, timetables and so on, will be considered significant changes to the content of a COP and will require the above procedure to be followed. Institutions should contact firstname.lastname@example.org if they require any clarification on this point.
Final versions of COPs will be collected from all submitting institutions in early 2014, for publication along with the submissions in 2015.
Previous FAQs (March 2013)
Is it possible to list the same individual twice within a joint submission, if that person has an eligible contract with two of the HEIs making the joint submission?
Yes, all HEIs involved in a joint submission are eligible to return the same individual within their part of the joint submission, provided the individual fulfils the eligibility criteria with each HEI.
Each HEI returning this individual in their part of the submission should return them with the same research outputs. The information will be collated and presented to the sub-panel as a single submission. Each distinct output listed against the individual will be assessed only once and count once towards the outputs sub-profile.
In the case of an edited work, where the editor has made several contributions (for example, the introduction, a substantive chapter and the conclusion), can these be submitted as a single output? Or could they, if preferred, be submitted as separate outputs?
The edited work could be submitted as a single output or alternatively, could be submitted separately if they are coherent as separate outputs. If returning them as separate outputs, you should consider whether they have significant material in common. (Further guidance on how outputs with significant material in common will be assessed is available in Part2 of the 'panel criteria'.)
For the UOAs in Main Panel A, what information should be provided to establish the author's contribution to a multi-authored research output?
Main Panel A have requested standard statements about the individual's contribution to a co-authored output, where required (see paragraphs 34–37 of Part2A of the 'Panel criteria'). In addition, to assure themselves about an author's substantial contribution, the sub-panel may request further details of the specific contribution of the author, through an audit query. Main Panel A have indicated that such audit queries are especially likely for outputs with 15 or more authors (and where the individual is not the lead or corresponding author).
When matching research outputs to Scopus within the REF submission system, what should I do if the electronic and hard copy versions of the same paper return different citation counts?
Where Scopus returns multiple matches for different formats of the same output, you should not confirm the match and should contact Scopus user support (via the REF submission system). Scopus user support will be able to create a single record for the output which consolidates citations for both formats. You will then be able to match this new record against the output within the submission system.
Where an individual moved to the HEI after 1 January 2008, and their research had a continuing impact that started before they joined the HEI, which part of the impact can the HEI claim?
The HEI can claim the impact from the point at which research carried out by staff while working in the HEI made a distinct and material contribution to it. Any impact prior to that point can be described in the case study as context only (to help explain the background the impact), but cannot be claimed as part of the impact underpinned by the submitting HEI's research.
Can the number of outputs be reduced for time spent working on a teaching-only (or other non-research) contract within the higher education sector?
Time spent working within the higher education sector is not accepted as an 'absence' for the purposes of allowing a reduction in outputs. The only exception to this is where an individual had a change in contraced duties (for example, to a teaching-only contract) due to a complex circumstance or protected characteristic. Please see examples 26 and 27 of the ECU’s complex circumstances.
Are individuals based outside the UK eligible to be returned to the REF and if so, does the HEI have to demonstrate their research is connected with the submitting unit?
Staff who are employed by the HEI to work in an overseas department or campus of the submitting HEI are eligible to be returned as Categroy A staff if:
the HEI demonstrates that the primary focus of their research activity on the census date is directly connected to the submitting unit based in the UK (see paragraph 79d of the 'guidance on submissions'). In the submission, the HEI should indicate that such staff are 'Non-UK based' and provide the 'Non-UK based explanatory text'.
Staff who work at another HEI outside of the UK who also have a fractional contract with a UK HEI, may be returned as Category A staff (with a fractional FTE) if they satisfy the general eligibility criteria at paragraph 78 of the 'guidance on submissions'. For these staff, as with any other member of eligible staff on a fractional contract, the HEI is not required to demonstrate their research connection with the submitting unit.
Paragraph 79i of the 'Guidance on submissions' refers to staff who hold more than one contract for different functions within a submitting HEI. Who might this apply to? And how many outputs should be returned for such staff?
HEI staff in managerial positions, for example, might not hold a contract with a primary employment function of 'research' or 'teaching and research'. In such cases, in order to be eligible for the REF, the individual would be required to hold a separate fractional contract which does satisfy the eligibility criteria (at paragraph 78 of 'guidance on submissions'). They would not be entitled to a reduction in the number of outputs on the grounds that they have a fractional contract for research. Any output reduction should be made in accordance with the guidance provided in paragraphs 64-91 of the 'panel criteria'.
If an individual is eligible for inclusion by multiple HEIs, can each HEI submit that individual with the same research outputs?
Yes. As stated at paragraph 79f of the 'guidance on submissions', an individual may be returned as Category A by more than one HEI if they have a contract with and receive a salary from more than one HEI. The same research outputs may, but need not be, listed in each submission.
If an individual works part-time for a number of HEIs, how many outputs should they be submitted with?
Where an individual works or worked part-time for a number of HEIs, then each of the HEIs submitting the individual will need to calculate if they are entitled to a reduction in outputs. A reduction in outputs may be made for the amount of time (calculated in months) in which the individual was not working in the HE sector, over the whole of the REF period. A reduction may not be made on the basis that the individual worked part-time at other HEIs.
Each HEI returning the individual (with a fractional FTE) will need to submit the number of outputs required of that individual. The same outputs may, but need not be, listed in each HEI's submission.
If individual circumstances apply to the individual, the number of outputs may be reduced in all submissions in which the individual is returned. If research outputs are 'missing' without a qualifying individual circumstance, the 'missing' outputs will be recorded as 'unclassified'.
An individual has worked 0.5FTE at HEI A, and 0.3FTE at HEI B, for the whole of the REF period (1 January 2008, up to the census date 31 Octo0ber 2013). The individual did not work at any other HEI for the remaining 0.2FTE.
The number of outputs may be reduced as follows:
70 months working in HE at a total of 0.8FTE
The equivalent number of months 'absent' from HE = 70 x 0.2FTE = 14 months 'absent'
This entitles the individual to a reduction of 1 output.
HEI A may return the individual at 0.5FTE, in which case 3 outputs are required.
HEI B may return the individual at 0.3FTE, in which case 3 outputs are required.
HEI A and HEI B may, but need not, list the same outputs.
If a member of staff has a contract with clearly split functions and part of their duties are non-academic, can the number of outputs be reduced for the time spent on their non-academic duties?
In order to be returned to the REF as Category A staff an individual must have a contract of employment with a primary employment function to undertake 'research only' or 'teaching and research'. They should be returned with an FTE relating to the entirety of that contract. The number of outputs may not be reduced on the basis that the individual also has other (non-academic) duties.
We have a Category C member of NHS staff undertaking research at the University. If the member of staff has a full-time contract with the NHS would they be eligible for a reduction in outputs?
Yes, as stated at paragraph 86 of Part 1 of the ‘panel criteria’, in UOAs 1-6 the number of outputs may be reduced by up to two, without penalty in the assessment, for Category C staff who are employed primarily as clinical, health or veterinary professionals (for example by the NHS), and whose research is primarily focused in the submitting unit.
If an individual first started their career as an independent researcher prior to 1 August 2009, can this time be included in an output reduction calculation?
For the purpose of identifying an individual as an early career researcher (ECR) in form REF1a: Staff details, the individual must first satisfy the definition of an ECR on or after 1 August 2009 (see paragraphs 85 – 87 of 'guidance on submissions').
For the purpose of calculating output reductions in form REF1b, you should consider the date that the individual started their career as an independent researcher, according to the criteria at paragraph 85a-b of the 'guidance on submissions'. This may be at any point within the REF period, and should be considered the date at which they first met the definition of an ECR.
If an individual first met the definition of an ECR before 1 August 2009 and this is the only circumstance for the individual, Table 1, Part 1 of the 'panel criteria' applies, and the individual is not entitled to a reduction in outputs.
If an individual met the definition of an ECR before 1 August 2009 and other circumstances also apply, the number of months from 1 January 2008 up to the point the individual became an ECR can be combined with the other circumstances. In this case, Table 2, Part 1 of the 'panel criteria' applies to the total number of months 'absent'.
An individual first started their career as an independent researcher on 1 September 2008 and then worked part-time at 0.5 FTE from 1 July 2010 – 30 June 2011. A reduction of 1 output is allowed:
Why does the submission system not offer the functionality to include formatting such as bold, underline and bullet points within the larger text fields (such as the explanation field within REF1b)?
This functionality was deliberately not included within the REF2014 submission system due to a number of problems which this functionality caused within the 2008 RAE system. It is possible to create a bulleted list in another application such as Word and then paste this in to the submission system. However this is not recommended as the formatting may be altered / lost when the data is extracted from the submission system for assessment. Instead you should make use of standard keyboard characters to create formatting. For example, asterisks(*) can be used to create a bulleted list, and the return key can be used to separate paragraphs.
Previous FAQs (January 2013)
I am unable to log in to the submission system and have followed the 'forgotten password' link. Why have I not received an email containing a new password?
The submission system will only send a new password to email addresses that are registered within the system. Please check that you have entered your email address in exactly the same way as the administrator who created your account. This situation most frequently occurs when institutions have email accounts that accept email from multiple domains. For example, even if an email account receives mail from both email@example.com and firstname.lastname@example.org. You will only be able to log in to the submission system with the exact email address that your institution's administrator registered to your account during its creation. If your administrator has created your account with an incorrect email address, you should ask them to change it for you.
I am unable to log in to the submission system and have followed the 'forgotten password' link. I have received a new password via email, however I am still unable to log in.
If you are unable to log in to the submission system using the password which has been sent to you via email, it is likely that your account is locked. This occurs when an incorrect password has been entered three or more times. In this instance you should contact your institution's administrator to unlock your account. Once your account has been unlocked you may then log in to the submission system using the last password which you requested from the system.
Alternatively your account may have expired. If your administrator reports that your account is not locked, ask them to check the expiry date which they set when your account was created.
My account is locked. How can I access the submission system?
If your account has become locked, you must contact one of your institution's administrators and ask them to unlock it for you. If you are an administrator you must contact another administrator to get them to unlock your account. The REF team strongly recommend that each institution has more than one administrator account present for this reason.
I do not have an account to access the submission system. Can you create an account for me?
REF support and the REF team are unable to create user accounts. To obtain an account you should contact your system administrator within your institution. If you do not know who this is you should contact your institution's designated REF contact.
Where can I find a list of all validation rules present within the submission system?
A list of all validation rules which the submission system contains is available.
Why has the submission system not imported any data from my Excel import file despite the import job completing successfully with no errors?
The most common reason for this occurring is due to the sheets within the Excel import being named incorrectly. If the import file has been created without using our template (available under 'submission system data requirements') you must name the file's sheets to match those in our template. The default sheet names (sheet1, sheet2, sheet3) prevent the submission system from locating any data. Therefore, although the file is 'successfully' processed, no records are imported.
Why has the submission system only imported a single output record within REF2 when my import file contains many output records?
This occurs when all output records have been given the same output identifier within the import file. In this case, instead of creating a separate output record for each output, the submission system will create the first output record, and then subsequently overwrite that record repeatedly until the import process has completed. To avoid this, each output should be given a unique output identifier.
Why is the submission system creating new outputs each time my import file is modified and re-imported?
This is occurring because the output records within your import file do not contain output identifiers. When the import file is modified and re-imported, the submission system uses the output's identifier to detect that it is already present. When it detects an output is already present, it overwrites it rather than creating a new one. To resolve this, outputs should be given output identifiers.
After running an import job, the status is shown as completed with validation errors; however none of my data is present within the submission system.
Because of the validation errors found, the submission system has not added the import data to your submissions - the data has been rolled back instead. In order to add data that contains validation errors you should select 'Allow data to be imported even if there are validation errors' when setting up the import job.
During import, why do I repeatedly receive the error message 'The institution code is invalid'?
You have entered an incorrect institution code. The 'institution' field must contain your institution's UKPRN (UK Provider Reference Number). This is provided by the UKRLP (UK Register of Learning Providers). Your institution's UKPRN can be obtained by visiting: http://www.ukrlp.co.uk and searching for your institution.
I have entered a staff member with less than four outputs and have completed REF1b: Individual staff circumstances. Why is the submission system still warning me that the staff member has an unexpected number of outputs?
The submission system will always display this warning when a staff member is linked to fewer than four outputs. The warning is intended to prompt users to check that fewer than four outputs may be submitted without penalty. Provided that the REF1b section has been filled out correctly, this warning can be ignored.
I have already imported my institution's part of a joint submission within a regular submission and am now trying to convert this to a joint submission by accepting an invitation from another institution. Why am I receiving an error message stating: 'You have attempted to accept the invitation to join a joint submission but, in addition to the joint submission created by the other institution who invited you to join, you have created your own submission to the same UOA'?
It is only possible to convert a regular submission to a joint submission if all common forms (RG, REF3a, REF3b and REF5) are empty. After removing any data present within these forms you should be able to accept the invitation. This will convert the regular submission into a joint submission
When creating an output record, why is the submission system informing me that the ISSN I have provided is not in the standard ISSN format even though I am sure it is correct?
There are some ISSNs that do not conform to the standard. Due to this the submission system will allow you to submit with an ISSN that appears to be incorrect. Providing you are certain that the ISSN entered is correct, you can ignore this warning.
I have submitted a request for a bulk citation retrieval / bulk CrossRef lookup and the job has been pending for a long period of time. Why is this happening?
Both the bulk citation retrieval and bulk CrossRef service rely on external systems provided by other companies (Elsevier and CrossRef) to function. During busy times these external systems may have large queues to process, consisting of requests from both the submission system and other systems. Once your request reaches the top of the queue, it will be processed. Should you feel for any reason that your request has 'stalled' and has not been processed after a reasonable amount of time, please contact REF user support.
I would like to clear each submission within our institution before re-importing my institution's updated data. As each output record's PDF needs to be added manually, will I have to manually re-add these each time after the submissions are cleared?
No. When a PDF is uploaded for an output record, the submission system links it to that output record's output identifier. This means that even if an entire submission is cleared and then re-imported, the uploaded PDFs will remain present within each output record. Note that the output records must be re-imported with the same output identifiers. Also, if a submission is deleted rather than cleared, all output PDFs in the submission will also be deleted. Note that REF3a, REF3b and REF5 PDFs will be cleared when a submission is cleared.
Previous FAQs (October 2012)
Will it be possible to identify double weighted outputs in the published REF submissions?
The list of research outputs included in the published REF submissions will indicate the research outputs that the HEI requested be double weighted. The list will also indicate the research outputs that HEIs marked as 'reserve' outputs. No indication will be given of which requests for double weighting were accepted by the sub-panels, or which reserve outputs were assessed. The published submissions reflect the submissions made by HEIs, rather than the decisions made by sub-panels.
What level of information will be required in REF1b for complex staff circumstances?
There are worked examples of complex circumstances published on the ECU web-site. These worked examples indicate the kind of information that should be provided and how it should be presented.
As far as practicable, an estimate should be provided of the equivalent number of months absent from work. Sufficient information should be provided to show how this was calculated, by stating the nature of the circumstance(s), the period(s) of time affected and the equivalent number of months absent for each period affected.
Any further constraints on the individual’s research work in addition to the equivalent months absent should also be explained (for example, constraints on research while at work). Sufficient detail should be provided to understand the overall effect on the individual's capacity to carry out research, by outlining the nature of the constraint(s), the proportion of their research capacity that was reduced, and the length of time affected.
An individual had time off work due to ill health and no further constraints after returning to work. REF1b need only state what the illness was, when the individual was off work, and the equivalent number of months absent.
An individual had time off work due to ill health, and after returning to work had limited capacity to pursue their research due to ongoing effects of the illness. REF1b should include the information as above, and in addition state what the ongoing effect of the illness was on their capacity to carry out research, and provide an estimate of the proportional reduction in their research capacity and the length of time to which this applied.
REF1b should also show how the reduction in outputs has been calculated, according to Table 2 of the 'panel criteria' in relation to estimated months absent from work, with further constraints taken into account as appropriate.
All information should be anonymous so that EDAP can review complex circumstances anonymously. Where the circumstance has involved other individuals (for example, where the member of staff cared for a relative), those individuals should not be named and it is not necessary to specify their exact relationship to the member of staff.
Is it necessary for the HEI to keep records of what evidence it consulted regarding individual staff circumstances, for the purposes of a potential audit by the REF team?
HEIs need to assure themselves that the circumstances cited in their REF submissions took place. In the event that the REF team audits REF1b, the HEI will need to state how it assured itself that the cited circumstance(s) took place, and what evidence it had consulted. Given the potential sensitivity involved in keeping records about such information, the timescale for these audits will provide HEIs with an opportunity to check the evidence before responding to the REF team, if a record of this information is not already held. Further details about the timetable for audit will be published in 2013.
For maternity, paternity and adoption leave, what does 'substantially during the period 1 January 08 to 31 October 2013' mean?
'Substantially' means that the majority or a substantial portion of the maternity leave was taken in this period. For example, an individual who went on maternity leave in June 2007 and returned in January 2008, or an individual who goes on maternity leave in October 2013 due to return during 2014, would not be considered to have taken maternity leave substantially within the REF period. Given that periods of such leave vary, a more precise definition cannot be given, and institutions will need to exercise their judgement.
The following additional guidance was added in October 2012:
In reaching these judgements, institutions should also consider how to treat staff whose maternity leave was taken partially within the REF period fairly, in comparison to staff whose maternity leave was taken entirely within the REF period. For the latter group of staff, there is no minimum qualifying length for maternity leave, and each period of such leave qualifies for the reduction of one output. Therefore, we consider it would be fair to allow the reduction of an output for a period of maternity leave that started prior to 1 January 2008 and continued for any length of time beyond that date, where the member of staff had been submitted to the 2008 RAE without any reduction in output due to that maternity leave.
How should eligible 'online first' journal articles be returned to the REF?
Online first journal articles should be returned as output type 'D - Journal article'. The REF team recognises that some of the meta-data may not be available for an online first publication, but sufficient information must be provided to enable the REF team to identify the output.
There may be some fields for which data is not available, that are mandatory for type D outputs (e.g. first page, volume number). These fields should be completed with ‘n/a’ to allow the submission to be made. You will not be able to submit without completing these mandatory fields.
What FTE should be returned as 'FTE on the census date' where an individual holds one contract with multiple functions?
The individual should be returned with the FTE of the contract that makes them eligible for submission to the REF; not the FTE specifically related to their research duties within that contract.
Previous FAQs (July 2012)
Will staff with individual circumstances be identified in the published submissions after the completion of the REF?
The published submissions will include a single list of all the outputs, and a separate list of all the staff, in each submission. The list of outputs will include standard bibliographic data (including the author name) for each output, but will not be listed by author name.
Following the review of the Higher Education Statistics Agency (HESA) staff record, what is the revised guidance on the coding of eligible staff as per footnote 2 of the 'guidance on submissions'?
The HESA guidance has been simplified to the following:
Individuals whose primary employment function is to undertake either 'research only' or 'teaching and research' are staff returned to the HESA Staff Collection with an academic employment function of either 'Academic contract that is research only' or 'Academic contract that is both teaching and research' (identified as codes'‘2' or '3' in the ACEMPFUN field).
This guidance should be read in conjunction with staff eligibility guidance at paragraphs 77 to 81 of 'guidance on submissions'.
'Guidance on submissions' paragraph 79h states: 'Staff whose salary is calculated on an hourly or daily basis are eligible only if they meet the definition at paragraph 78 and on the census date have a contract of employment of at least 0.2 FTE per year over the length of their contract.' How should institutions calculate the full-time equivalence (FTE) of staff contracted on an hourly or daily basis?
Institutions should calculate the mean FTE of these staff using the number of hours or days worked from the start of their current employment contract up to the census date (31 October 2013), based on the standard hours or days of a full-time employee at that institution.
Are higher education institution (HEI) staff in managerial positions eligible for return to the REF?
In paragraph 78 of the ‘guidance on submissions’, Category A staff are defined as ‘academic staff with a contract of employment of 0.2 FTE or greater and on the payroll of the submitting HEI on the census date (31 October 2013), and whose primary employment function is to undertake either ‘research only’ or ‘teaching and research’.’
Individuals who do not meet these criteria are not eligible to be returned to the REF as Category A staff. Therefore, staff employed on full-time managerial contracts are not eligible for the REF. Staff who hold academic contracts of 0.2 FTE or greater with a primary employment function to undertake either ‘research only’ or ‘teaching and research’ are eligible for return to the REF, regardless of whether they also undertake managerial duties.
Any member of staff employed by the HEI, including heads of HEIs, is not eligible for return to the REF as Category C staff.
What information/explanatory text will be required for clearly defined individual staff circumstances in form REF1b?
For clearly defined circumstances, the explanation field of REF1b should be used to state what each of the circumstances was and when it occurred, to show how the reduction in outputs was calculated, and to provide enough information about the circumstances to enable the sub-panels to confirm that the criteria for output reductions have been met. The following specific details should be provided for each type of circumstance:
For early career researchers, provide brief details of the individual’s research career history, specifically identifying the point at which they first met the REF definition of an early career researcher. It is important to state why the individual did not, prior to that point, meet the definition of an early career researcher. This could be either by stating the nature of their contract (which did not meet the definition at paragraph 85a of ‘guidance on submissions'), or by outlining how their work did not include independent research.
For part-time working, provide the start and end dates and the FTE for each period in which the individual worked part-time, and state the equivalent ‘months absent’ for each period. These must be ‘absences’ in which the individual was neither working in the higher education sector, nor undertaking academic research at some other organisation (see this FAQ).
For secondments and career breaks, provide the start and end dates, and state the equivalent ‘months absent’ for each period. Provide sufficient information about the career break or secondment (for example, the name of the organisation seconded to and the job role) to confirm that the individual was neither working in the higher education sector nor undertaking academic research.
For maternity, paternity or adoption leave, state the start and end dates of each period of such leave.
Where multiple circumstances have been combined to calculate a reduction in outputs, the explanation should clearly show that where circumstances have overlapped, only one circumstance has been taken into account for that period.
Should HEIs provide information in REF1b where individual circumstances exist, but where the number of outputs for that individual is not reduced?
No. Information provided in REF1b is used only to determine reductions in the number of outputs, and will not be taken account of for any other purpose.
Will the Equality and Diversity Advisory Panel's (EDAP's) decisions on complex circumstances be final, and what information will be available to HEIs on these decisions?
EDAP’s role is to make recommendations on complex circumstances to the chairs of the main panels, who are responsible for the decisions. Details of these decisions or any consequent reduction in outputs will not be provided to the submitting HEI, as no scores for individual outputs – missing or otherwise – are identified in the quality profiles award to submitted units, and HEIs do not receive feedback on the individual parts of submissions. In order for HEIs to understand how EDAP will form its recommendations, the Equality Challenge Unit (ECU) has published a series of worked examples of complex staff circumstances. These examples include EDAP’s rationale: http://www.ecu.ac.uk/documents/ref-materials/complex-circumstances-examples.
Can HEIs submit a reserve output, to be assessed if the reduction in outputs due to individual staff circumstances is not accepted?
No, HEIs may not submit a reserve output for individuals submitted with a reduction in outputs due to individual circumstances. Allowing a reserve output would diminish the case for an individual’s circumstances significantly constraining their ability to produce four outputs or to work productively throughout the assessment period. Additionally, allowing a reserve output would potentially result in unequal treatment of individuals who do not have an additional output compared to those who do.
If an individual is employed by multiple HEIs on part-time contracts, or has multiple part-time contracts with one HEI for different roles, are they eligible for a reduction in outputs due to part-time working?
The reduction for part-time working applies only to time ‘absent’ in which the individual was neither working in the higher education sector, nor undertaking academic research at some other organisation. This is consistent with the allowances for secondments and career breaks.
Time spent working part-time in higher education, whether on an academic or non-academic contract, and whether in a UK or overseas HEI, does not qualify as an ‘absence’ when calculating reductions due to part-time working.
If an individual has had an eligible contract for many years but focused on teaching, and only undertook independent research for the first time after 1 August 2009, could they be defined as an early career researcher(ECR)?
Yes, for the purposes of the REF an individual is deemed to have started their research career from the point at which they held a contract of employment of 0.2 FTE or greater, which included a primary employment function of undertaking ‘research’ or ‘teaching and research’, with any HE or other organisation, whether in the UK or overseas, and they undertook independent research, leading or acting as principal investigator or equivalent on a research grant or significant piece of research work. Individuals qualify as early career researchers if both of these criteria were satisfied for the first time on or after 1 August 2009, even if one of these criteria had been satisfied prior to 1 August 2009.
For a research output that is a product, what is considered the date it first became publicly available?
If submitting a product (output type = ‘P’) the date that the product was produced would be considered the date it became publicly available in the form in which it is submitted (see the ‘Output information requirements’ spreadsheet, available for download on the 'Submission system data requirements' page ).
In what circumstances can a related series of items be submitted as a single research output?
Separately published papers are discrete outputs and cannot be grouped together as a single output. Substantial dictionary or encyclopaedia entries and groups of short items including groups of entries may be submitted as a single output (see paragraph 51, Part2D of the ‘panel criteria’).
What output type should a 'portfolio' be submitted as?
A portfolio is not an output type in itself but is the format in which the evidence of certain outputs can be submitted (see paragraph 59 of Part2C and paragraph 71d, Part2D of the ‘panel criteria’.) For example, an output that is a performance or exhibition may be submitted in the form of a portfolio of evidence. Where submitting a portfolio, the most applicable output type should be used, and the ‘media’ field should be used to state that a portfolio will be submitted (see the ‘Output information requirements’ spreadsheet, available for download on the 'Submission system data requirements' page).
How will a research output submitted multiple times, to the same or different Units of Assessment (UOAs), be assessed?
Where a research output is submitted multiple times to the same UOA, it will normally be allocated to the same individuals on the sub-panel and reviewed only once (although the additional information may differ). Please note, an output may only be submitted a maximum of 2 times within one submission.
If a research output is submitted to multiple UOAs, it will be assessed by each of the sub-panels according to their respective criteria, including in terms of its significance to the disciplines concerned. The output will not, therefore, necessarily receive the same quality score.
What is the outcome if a sub-panel considers an individual has not made a substantial contribution to a co-authored research output returned against them?
If the sub-panel considers that the individual may not have made a substantial research contribution to the output and there is not sufficient evidence in the submission to make an informed judgement about this, the sub-panel will raise an audit query. The HEI will then be asked to explain the individual’s research contribution, and provide any appropriate evidence. The sub-panel will consider all the information provided before deciding. (Note that in situations where this information has already been requested as part of the submission, for example for sub-panel 9 where there are more than 10 co-authors, the sub-panel will normally decide on the basis of the submitted information without any further audit query.) If the sub-panel does not accept that the individual made a substantial research contribution, the output will be graded as ‘unclassified’ (see paragraph 126e, Part 1 of the ‘panel criteria’.)
What action should an HEI take if the source to corroborate an impact case study is in a language other than English?
If the corroborating evidence is a pre-existing document not available in English, the HEI should reference the document and state what language it is in. The REF team will use the expertise of specialist advisers with the relevant language skills, if corroboration through these sources is required.
Any corroborating statements provided by users and held on file by the HEI will need to be provided to the REF team in English if requested through an audit. Where necessary the HEI will need to arrange for their translation.
Corroborating contacts should be given only for people who the REF team can communicate with in English.
Do all the outputs referenced in an impact case study need to be of at least two-star quality?
A case study should include references to up to six research outputs that represent the body of research or a research project that was carried out at the submitting institution. These should be key outputs that underpinned the impact, and that best demonstrate the quality of the body of work or project. The sub-panels will not expect each individual output to meet the quality threshold, but will wish to be satisfied that the listed work was predominantly of at least two-star quality.
Can the same impact case study be submitted by more than one submitting unit?
Where more than one submitting unit made a distinct and material research contribution to an impact, each of those submitting units may submit a case study of the impact. However, the case studies should not be identical, because each submitting unit will need to show that its research made a distinct and material contribution to the impact. This applies whether an HEI wishes to submit the same impact in different submissions, or different HEIs wish to submit the same impact.
Can an HEI submit an impact case study in a UOA, even if the individual who conducted the research is returned in a different UOA?
Yes, we recognise that individual researchers may undertake research across multiple disciplines over time and that UOA boundaries are not rigid. Provided the underpinning research is within the scope of the UOA in which it is submitted, a case study may be submitted in a different UOA from the individual.
Is a PhD by publication eligible for inclusion in REF4a?
The only qualifications that may be included in REF4a are those returned to the HESA record as ‘D00: Doctorate degree that meets the criteria for a research-based higher degree’ and ‘D01: New Route PhD that meets the criteria for a research-based higher degree’. Data returned to the HESA record as E00: ‘Doctorate degree not obtained primarily through advanced supervised research written up as a thesis/dissertation’ are excluded from REF4a. PhD by publication, PhD by portfolio and professional doctorates recorded as E00 are not eligible for inclusion in REF4a.
How will additional assessors and user members of the REF panels be integrated into the assessment procedures?
The REF team will hold briefing events for additional assessors and users in advance of the assessment phase. In addition, all sub-panels (and main panels) will undertake calibration exercises of outputs and impacts towards the beginning of the assessment phase, fully involving the assessors as well as the panel members. Assessors will attend the meetings at which the material they have been involved in assessing is to be discussed.
Previous FAQs (June 2012)
Can more than one individual from an institution access the REF extranet?
The extranet login information that has been provided to the institution’s main REF contact can be used to create logins for multiple staff. It is up to the institution to decide who should have access to the REF extranet and to manage who submits the relevant documents through the extranet. Institutions’ HESA data has been provided by the REF team through the REF extranet and will include sufficient data to identify individual research students. This should be considered when disseminating extranet login information.
If an individual takes a 'career break' from research, but remains in the higher education sector to undertake a management, administrative or teaching only role, would they be eligible for a reduction in outputs?
No. (The number of outputs may be reduced without penalty if the career break involved not working in the higher education sector and not undertaking academic research.)
What is the definition of 'independent research' for an early career researcher?
The definition of independent research is at paragraph 85b of 'guidance on submissions'. In leading a significant piece of research work, the individual may, but need not, have acted as principle investigator or equivalent on a research grant. An individual who meets the definition of a research assistant at paragraph 80 of 'guidance on submissions' would not be considered to be conducting independent research (unless, exceptionally, they are eligible for selection according to paragraph 81).
The definition of early career researchers for the REF is not based on a particular job role or job title as these may vary from institution to institution. The completion of a PhD is not always a useful indicator of the first point at which an individual undertook independent research.
What initials and surname should be provided for REF1a where the individual uses a different name (e.g. pen name or maiden name) for publication of their research outputs?
Staff who use a different name on published research outputs should be returned to the REF with this name on REF1a. If staff data verification is required from the institution, we will require additional evidence to verify the individual’s identity.
If an individual is employed by the institution but have their salary paid directly from an external funder, are they eligible for inclusion in the REF?
An individual who is not on the payroll of the submitting institution on the census date is not eligible for inclusion as Category A staff. Such an individual may be eligible for inclusion as Category C: given that their salary is directly paid by a body external to the institution, this qualifies as ‘employed by an organisation other than an HEI’. (See paragraph 82 of ‘guidance of submissions’.)
If a higher education institution is given approval to make multiple submissions in a UOA is there an obligation to make multiple submissions?
No, requests to make multiple submissions are not binding. An institution may decide to make only one submission in a UOA where they have been given approval for multiple submissions.
To what extent will the information provided in REF1b (individual staff circumstances) be treated confidentially? Will EDAP review them anonymously?
Information on clearly defined circumstances will be available to the panel secretariat and panel members.
Information on complex circumstances will be provided only to the Equality and Diversity Advisory Panel (EDAP), the REF team and main panel chairs. Sub-panels and the secretariat will be informed that there are complex circumstances and of the decision on the number of outputs to assess, but they will not have access to any information about the nature of the circumstances.
To further maintain confidentiality, EDAP will review complex circumstances in anonymous form, where they are submitted by institutions in this form. Institutions are advised to complete the REF1b form for complex circumstances, without stating the individual's name or other personal identifiers.
Where an individual is submitted with fewer than four outputs, without penalty in the assessment, how will the individual's research outputs contribute to the quality profile?
Each single output in a submission contributes an equal proportion to the outputs sub-profile for that submission, not as a proportion of the maximum of 4 outputs per member of staff. See paragraph 126, Part 1 of the ‘panel criteria’.
How should the total reduction in outputs be calculated where more than one clearly defined circumstance applies?
The calculations should be made according to the guidance at paragraphs 72-75, 80 and 82-84 of the ‘panel criteria’. The final sentence of paragraph 82 should be applied as follows: For all clearly defined circumstances to which Table 2 applies, the total ‘months absent’ should be added together, and the reduction for these circumstances calculated by applying Table 2 to the combined total. Any further reductions should then be applied due to qualifying periods of maternity, adoption or additional paternity leave.
The REF submissions system will provide a calculator function to determine the maximum reduction in outputs for combinations of clearly defined circumstances, based on the data entered about these circumstances. Some worked examples are provided below.
An individual working part-time at 0.8 FTE from 1 Jan 08 – 30 April 09 and part-time at 0.6 FTE from 1 May 09 – 30 April 11 would be eligible for a reduction of 1 output:
0.2 x 16 months
3.2 months absence
0.4 x 24 months
9.6 months absence
12.8 months absent (reduction of 1 output)
An individual took a career break outside of the HE sector from 1 Sep 09 – 31 Aug 10, then returned to research part-time at 0.5 FTE from 1 Sep 10 to 31 Aug 12. They would be eligible for a reduction of 1 output:
12 months career break
0.5 x 24 months
12 months absence for working part-time
24 months absent (reduction of 1 output)
An individual takes a 6 month period of maternity leave and then returns to works part-time at 0.4 FTE from 1 Mar 12 until the census date. The individual would be eligible for a reduction of 2 outputs:
12 months absent (reduction of 1 output)
+ 1 period of maternity leave (reduction of 1 further output)
(Reduction of 2 outputs)
An individual qualified as an ECR on 1 Mar 2010, and worked part-time at 0.6 FTE from 1 May 2012 until the census date. They would be eligible for a reduction of 2 outputs:
26 months ‘absence’ before qualifying as an ECR
0.4 x 18 months
7.2 months absence for working part-time
33.2 months absent (reduction of 2 outputs)
If an output pre-published in 2007 was submitted to the RAE 2008 by a different institution or in a different UOA, can it be submitted in final form to the 2014 REF?
See paragraph 43 of the ‘panel criteria’. The final proviso should be interpreted to mean that such an output is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE by the same institution, or was not listed against the same member of staff (if they were in a different institution at the time).
Are 'online first' journal articles that are not published in hard copy by 31 December 2013 eligible for inclusion in the REF?
Yes, 'online first' and other pre-published outputs that are in the public domain during the publication period (1 January 2008 – 31 December 2013) are eligible for submission.
What output information is required for patents and patent applications?
Both a granted patent and a published patent application are eligible as research outputs. If you are submitting a granted patent you should include the patent number and date of the granted patent; if you are submitting a published patent application you should include the publication number and the date that this was published. Where a patent application was submitted to the 2008 RAE, the granted patent is not eligible for return to the REF.
How will journal impact factors, rankings or lists, or the perceived standing of publishers be used to inform the assessment of research outputs?
No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs. An underpinning principle of the REF is that all types of research and all forms of research outputs across all disciplines shall be assessed on a fair and equal basis.
What kind of outputs can be categorised as a 'working paper' (Output type = 'U')?
A working paper is often written in the style of a journal article, is usually made freely available on line, may not have been peer reviewed, and has not yet been formally published (typically, in a journal). Working papers will not have a Digital Object Identifier. The use of working papers is more prevalent within certain research communities, for example ArXiv is often used in particular disciplines. The output must still meet the eligibility definitions described in 'guidance on submissions'.
Does the indicative maximum of 6 references to research in an impact case study refer to a combined total of research references and grant information details?
No. In addition to the maximum of 6 research references, grant information details may also be provided, where appropriate.
Can an institution return research doctoral degrees data (REF4a) and research income data (REF4b) where the supervisor or grant holder is not returned to the REF?
Yes. PGR degrees awarded to all students supervised within the submitting unit should be included. The research income of the submitted unit as a whole should be included.
Previous FAQs (November 2011)
Can a joint submission be made by higher education institutions (HEIs) from different countries within the UK?
Yes. In such cases the funding will be allocated to each HEI according to the funding method of their respective funding body, using the number of Category A staff (FTE) submitted by each institution as the volume measure.
Can requests for multiple submissions be made earlier than the published timetable (October to December 2012)?
Yes. Following feedback from the sector, HEI's will be invited in early 2012 to make their requests for multiple submissions at any time from March 2012 until December 2012. We will consider all requests received at regular cut-off points during this period, and will provide a response within six weeks of each of those cut-off points. Each HEI may only make one set of requests for multiple submissions during this period.
What will be the process for redacting parts of submissions for publication purposes?
When making submissions institutions will be able to identify individual staff, listed outputs or impact case studies that should not be published as part of the submission; and to identify other textual parts of submissions (REF3a, REF3b and REF5) that require some content to be redacted prior to publication. Following the submission deadline, institutions will have the opportunity to return redacted versions of these parts of submissions for publication. Further guidance will be provided in due course.
On what basis and at what level of detail will the 'proportion of staff submitted' be published?
If an academic had been employed solely on a 'teaching only' contract prior to 1 Aug 2009 and on a 'research' or 'teaching and research' contract after that date, could they qualify as an early career researcher?
Will the REF system limit access within the HEI to the personal information in REF1b?
Each institution will be able to determine which users of the submission system within the HEI have permission to read and/or edit each of the REF 'forms', including REF1b, for each UOA they are submitting in.
Can HEIs submit cases of complex individual staff circumstances for the Equality and Diversity Advisory Panel (EDAP) to review and provide feedback prior to the submission deadline?
Following publication of the final panel criteria and working methods, the Equality Challenge Unit (ECU) will publish worked examples of complex staff circumstances, including EDAP's recommendations on these hypothetical cases. This is intended to provide clear indications to HEIs about the operation of the arrangements for complex circumstances. Following publication of these examples, HEI will be able to suggest further hypothetical examples for publication by ECU, to add to the range of published examples.
Examples of complex staff circumstances are available on the ECU web-site.
Can an output authored by a member of staff before they became eligible for submission to the REF (or before they met the definition of an early career researcher) be listed against that member of staff?
Can an impact that had taken place before the output(s) from the underpinning research were published be submitted in a case study?
Yes. It is recognised that research may have had impact prior to the publication of the output(s). The end of the publication period for the underpinning research (31 December 2013) extends beyond the end of the period for the impact (31 July 2013).
Can 'interim' impacts be submitted in case studies, and how will they be assessed?
Submitted impacts may be at any stage of development or maturity so long as some effect, change or benefit meeting the REF definition of impact (at Annex C of 'Assessment framework and guidance on submissions') took place during the assessment period (1 January 2008 to 31 July 2013). Regardless of the stage of maturity, case studies will be assessed in terms of the 'reach and significance' of the impacts that had occurred during this period.
Can an institution submit an impact underpinned by research carried out while the staff were working at the submitting institution, but the output was published after they had left the institution?
Yes. The case study should state what research was carried out at the submitting institution, by whom, and when. As with all information provided in submissions, such information should be capable of verification by the institution.
Can an institution submit an impact that is underpinned by research carried out by staff who were returned (or were eligible) as 'Category C' staff in a former RAE?
Yes, provided that they carried out the underpinning research while working in the submitting HEI.
Can an impact case study be submitted in a different REF UOA to the UOA that the underpinning research had been submitted in a former RAE?
If a case study is graded as Unclassified, will the HEI be informed whether this was due to the reach and significance of the impact, or due to some other reason (such as ineligibility or insufficient quality of the underpinning research)?
No. The panels may include general comments on this issue in their overview reports, if applicable.
Can the institution only submit impacts underpinned by research in areas in which it still remains active?
A submitted impact must be underpinned by research that was carried out by staff while working in the submitting HEI. The HEI need not remain active in the area of research.
Note that in reaching their overall judgements about the impact element of submissions, sub-panels will consider the submitted unit's recent activity and strategy for enabling impact from its current research.
If the submitting HEI has merged with another HEI or has taken over another research unit, can it submit impacts from research that was undertaken by the other HEI or research unit before becoming part of the submitting HEI?
Where a submitting HEI is the result of a merger between former HEIs, the submitting HEI can submit impacts from the research undertaken by the former, now merged, HEIs.
Where a submitting HEI has taken over a research unit – whether from another HEI or from elsewhere – the submitting HEI can submit impacts from research that was undertaken by the absorbed unit before it became part of the submitting HEI, with prior agreement from the relevant UK funding body.
Prior agreement must be sought by providing details of the nature of the research unit and of when and how it became part of the submitting HEI, to email@example.com, no later than 30 June 2013. The REF team will liaise with the relevant funding body and communicate the decision to the HEI.
In each case the funding bodies will take into consideration whether a distinct unit was absorbed by the submitting HEI in its entirety, and the extent to which there has been genuine structural change.
When will the REF team provide HEIs with HESA data on research income and research degrees awarded to be used in preparing REF submissions?
Data on research income and research doctoral degrees awarded for academic year (AY) 2008-09 to AY 2010-11 will be provided by May 2012; and data for AY 2008-09 to AY 2011-12 by April 2013. HESA will make AY 2012-2013 data on research doctoral degrees awarded available in a suitable form through the checkdoc facility, from August 2013. Further details, including arrangements for research income-in-kind, will be provided in due course.
This data has been provided through the REF extranet, for further details please contact the REF main contact for your institution.
Why is data on research income and research degrees awarded not being collected for 2007-08?
We are aligning data on research income and research degrees awarded as far as possible with existing HESA data, and the decision was taken not include 2007-08. This is to balance data quality considerations (2007-08 data was returned to HESA before its potential use in the REF was known), with panels' requirements for a sufficient time-series of such data for the purpose of informing their assessment of the research environment. Five years of such data (2008-09 to 2012-13) is considered sufficient for this purpose.
Can HESA data on research degrees awarded be provided to HEIs at the level of the student, to enable HEIs to map the data to REF UOAs?
The REF team are looking into the feasibility of this, and will provide further information in due course.
This has been provided to HEIs in the data workbook available on the REF extranet.
How should submissions deal with changes to the income categories in HESA data during the REF period?
The income sources for the 'Research grants and contracts' table (currently Table 5b) in the HESA Finance Statistics Return have not changed during the period 2008-09 to 2010-11 inclusive, and we do not expect them to change for the remainder of the REF period.
Where research degree students were supervised by one HEI (without research degree awarding powers) and the degrees were accredited/awarded by another HEI, which of the HEIs should include the degrees awarded in their REF submission?
Research degrees awarded are returned to HESA by the HEI that registered the students. This HEI should include the research degrees awarded in their REF submission.
Can the institution publish its code of practice on selection of staff before it has been submitted to the REF team or examined by the Equality and Diversity Advisory Panel (EDAP)?
Institutions can disseminate their code of practice when finalised, in advance of submitting this to the REF team. If any changes to the code are subsequently required by the relevant funding body following examination by EDAP, the institution will need to notify staff of the changes.