Research assessment exercise 2008 - Survey of clinical academic and research staff
Report
September 2005
Summary
- The characteristics of the survey respondents is broadly representative of the UK clinical academic population as a whole.
- Almost three-quarters of respondents have been returned in previous RAEs and most have been returned in the previous 2 RAEs (1996 and 2001).
- Respondents were asked to rank a range of publications to be included in the RAE assessment. Research papers were ranked as first priority by most respondents, followed by systematic reviews. Reviews and authored books were also ranked highly.
- Whilst a quarter of respondents would like to see the number of publications included in the RAE assessment remain at the current 4, a further fifth would like to see this increased to 5 publications. The mean number of publications suggested for inclusion in the RAE assessment is 6.6.
- Four out of five respondents agree that resulting changes in NHS practice should be considered important in the assessment of publications in the RAE. The impact factor of publications and citation index are also considered to be important by most respondents.
- Most respondents agree that PhD/MD students, government grants and grants from charities or the NHS should be considered in the RAE as key elements of a clinical academic’s research capacity. Whilst many suggested that it was not possible to select a single element and that it was the entire portfolio of work which should be considered, the majority of respondents suggested that the number and type of grants was the most important element of a clinical academic’s research capacity.
- More than half of respondents agree that the best measure of collaborative research with colleagues is the number of contributors to a paper. Clinical trainees (NHS not University employees) undertaking research projects is also considered a good measure of collaborative research with colleagues.
- Less than two-thirds of respondents know in which category they will be returned in the forthcoming RAE.
- Half of respondents report that they are aware of some form of University reorganisation that has taken place to optimise anticipated RAE scores.
- Whilst a fifth of respondents regarded the RAE as having had a positive impact on their career, 40% regarded the RAE as having had a negative impact on their career.
The RAE 2008 process is moving on to review academic medicine retrospectively as part of university research assessments to prospectively determine funding of higher education institutions. There are 2 main panels set up to cover medical topics. Each panel has to determine the criteria for assessment of quality and quantity of research output. In the past the RAE has concentrated on specific research assessments that have been applied to medical academics as four publications (in the period in question) in high impact factor journals (e.g. If 5 or more), number of PhD students and research grant spending. These limited measures of assessment have been criticised for their narrow focus, a tendency to reward laboratory projects such as animal experimentation instead of human studies and hence an overall failure to adequately measure the contribution of clinical academics to medical research. In addition, there are also time factors and different commitments e.g. NHS activities that do not enter into these assessments. The BMA Medical Academic Staff Committee (MASC) consider that this is an opportune time, when the assessment criteria for the next RAE are being considered, to ascertain what medical academics consider important and relevant measures of clinical research quality and excellence.
A questionnaire was sent to clinical academics and research staff across the United Kingdom asking for views on reasonable assessments within the context of their work. A total of 756 completed responses were received. This report presents the key findings of the survey.
Results
Characteristics of respondents
Around half of respondents are currently employed at professorial level and a further quarter are senior lecturers (table 1). The distribution of the survey respondents according to grade is slightly over-representative in the more senior grade of professor and slightly under-representative of the reader/senior lecturer and lecturer grades, compared with the UK clinical academic population as a whole [go to note 1] (figure1). Most survey respondents are male (78%-22% female) and the age of respondents ranges from 28 to 80 years (mean age=48yrs). The demographic characteristics of the survey respondents is representative of the UK clinical academic population as a whole. Across the academic grades, only 20% of academics are female. More than 50% of clinical academics are above the age of 45 and only 10% are under the age of 35. According to the Council Heads of Medical Schools (CHMS), medicine is an ageing profession and academic medicine is no different. Evidence suggests a decline in the number of young doctors holding clinical lecturers in the traditional seedcorn of academic medicine and this supports the hypothesis that young doctors do not perceive clinical academia as an attractive career [go to note 2].
Table 1: Current grade of survey respondents
| Frequency | Per cent | |
| Professor | 343 | 45.5 |
| Reader | 69 | 9.2 |
| Senior Lecturer | 205 | 27.2 |
| Lecturer | 26 | 3.4 |
| Research fellow | 58 | 7.7 |
| Other | 53 | 7.1 |
| Total | 754 | 100.0 |
| No reply | 2 | - |
Figure 1: Clinical academics by grade (%)
In the case of four out of five respondents, the University holds the substantive contract of employment (table 2), but most respondents (90%) also hold an honorary NHS contract. Around half of respondents (55%) hold a clinical excellence award. Almost three-quarters (70%) of respondents have been returned in previous RAEs. In two-thirds of cases, respondents have been returned in the previous 2 RAEs (1996 and 2001) (table 3).
Table 2: Organisation which holds substantive contract
| Frequency | Per cent | |
| University | 626 | 83.6 |
| NHS | 63 | 8.4 |
| MRC | 16 | 2.1 |
| Joint appointment | 22 | 2.9 |
| Other | 22 | 2.9 |
| Total | 749 | 100.0 |
| No reply | 7 | - |
Table 3: Returned in previous RAEs
| Frequency | Per cent | |
| 1996 |
19 |
3.7 |
|
2001 |
161 |
31.4 |
|
Both 1996 and 2001 |
333 |
64.9 |
|
Total |
513 |
100.0 |
|
No reply |
3 |
- |
Publications
Respondents were asked to rank a range of publications to be included in the RAE assessment. Research papers were ranked as first priority by most respondents, followed by systematic reviews (table 4). Reviews and authored books were also ranked highly by around half of respondents (figure 2). Other suggested publications for inclusion in the RAE assessment include international workshop reports, professional commissioned reports (ie, Royal Colleges), patents, teaching outputs, websites, scientific commentary and invited lectures (especially named). The majority of respondents agree that all publications included in the RAE assessment should be peer reviewed. This is particular the case with regard to research papers, systematic reviews, case reports, reviews, invited articles and editorials (table 4). The preferred weighting for publications included in the RAE assessment is highest for research papers and systematic reviews (table 4).
Table 4: Type of publications included in RAE assessment
| % ranked as 1st |
% ranked as 2nd |
% ranked as 3rd |
Total % ranked 1-3 |
% should be peer reviewed |
Preferred weighting (mean) |
|
| Research papers | 91.4 | 2.8 | 1.1 | 95.3 | 99.9 | 6.0 |
| Systematic reviews | 8.6 | 60.3 | 12.3 | 81.2 | 96.8 | 5.7 |
| Reviews | 1.3 | 16.6 | 34.8 | 52.7 | 91.9 | 2.8 |
| Authored books | 4.0 | 13.0 | 21.3 | 38.3 | 55.3 | 3.0 |
| Editorials | 1.3 | 9.2 | 16.6 | 27.1 | 76.7 | 2.6 |
| Invited articles | 0.8 | 5.6 | 12.4 | 18.8 | 77.9 | 2.4 |
| Book chapters | 0.5 | 6.5 | 10.6 | 17.6 | 57.8 | 2.6 |
| Government publications | 1.7 | 5.4 | 10.2 | 17.3 | 67.8 | 2.2 |
| Conference abstracts | 1.3 | 4.0 | 6.6 | 11.9 | 67.3 | 1.6 |
| Case reports | 1.0 | 1.6 | 5.0 | 7.6 | 79.1 | 1.6 |
| Letters | 0.5 | 2.2 | 4.1 | 6.8 | 55.9 | 1.6 |
| Pharmaceutical papers | 0.6 | 1.5 | 3.7 | 5.8 | 72.5 | 1.7 |
Figure 2: Ranking of type of publication to be included in the RAE assessment
Whilst a quarter of respondents would like to see the number of publications included in the RAE assessment remain at the current 4, a further fifth would like to see this increased to 5 publications (table 5 and figure 3). The mean number of publications suggested for inclusion in the RAE assessment is 6.6, the median is 5 and the mode number of publications is 4. Whilst the survey evidence suggests that most respondents would prefer the number of publications included in the RAE assessment to remain at 4 or increase slightly to 5 or 6, Figure 3 shows that more than a quarter of respondents would like to see 10 or more publications assessed as part of the RAE.
Table 5: Preferred number of publications to be assessed as part of the RAE
| Number of publications | Frequency | Per cent |
| 1 |
17 |
2.5 |
| 2 | 19 | 2.8 |
| 3 | 31 | 4.6 |
|
4 |
167 |
24.7 |
|
5 |
152 |
22.5 |
|
6 |
64 |
9.5 |
| 7 | 5 | 0.7 |
| 8 | 24 | 3.5 |
| 9 | 2 | 0.3 |
| 10 | 113 | 16.7 |
| 12 | 67 | 9.9 |
| 13+ | 16 | 2.4 |
| Total | 677 | 100.0 |
|
No reply |
80 |
- |
Figure 3: Preferred number of publications to be assessed as part of the RAE (%)
Four out of five respondents agree that resulting changes in NHS practice should be considered important in the assessment of publications in the RAE. The impact factor of publications and citation index are also considered to be important by most respondents (table 6). The difficulty in separating these results from audit improvements was not investigated in this survey, however the place of audit in clinical academic research should not be underestimated. Respondents suggested other aspects of publications which could be considered important in assessment and these include scientific complexity, relevance to current practice, changes to clinical practice, originality, improvement of understanding and impact on policy.
Table 6: Aspects of publications which should be considered important in assessment
| Frequency | % of total respondents |
|
| Changes in NHS practice | 617 | 82.8 |
| Impact factor | 542 | 72.8 |
| Citation index | 498 | 66.8 |
| Acceptance by NICE | 281 | 37.7 |
| Patient group interest | 180 | 24.2 |
| Generated lectures | 132 | 17.7 |
| Number of letters generated in response | 113 | 15.2 |
| Source of audit projects | 84 | 11.3 |
| Media interest | 78 | 10.5 |
Research capacity
Most respondents agree that PhD/MD students, government grants and grants from charities or the NHS should be considered in the RAE as key elements of a clinical academic’s research capacity (table 7 and figure 4). The number of grants obtained, MSc students, editorials, the financial value of grants obtained and sitting on trial steering committees are also seen to be key elements of an academic’s research capacity by at least half of respondents. Respondents were asked to indicate the most important element of a clinical academic’s research capacity from the list given. Whilst many suggested that it was not possible to select a single element and that it was the entire portfolio of work which should be considered, the majority of respondents suggested that the number and type of grants was the most important element of a clinical academic’s research capacity.
Table 7: Elements of a clinical academic’s research capacity which should be considered in the RAE
| Frequency | % of total respondents |
|
| Editorials |
454 | 62.1 |
| Trial steering committees | 426 | 58.3 |
| RSM council | 132 | 18.1 |
| Professional council | 333 | 45.6 |
| Number of grants | 561 | 76.7 |
| Type of grants: Government |
650 |
88.9 |
| Charities | 645 | 88.2 |
| NHS | 619 | 84.7 |
| Local, pharmaceutical | 371 | 50.8 |
| Other | 165 | 22.6 |
| Professional opinion paper | 324 | 44.3 |
| Data monitoring committee | 274 | 37.5 |
| Research Society Council | 363 | 49.7 |
| Financial value of grants | 451 | 61.7 |
| Research students: PhD/MD student |
673 |
92.1 |
| MSc student | 502 | 68.7 |
| UG student | 255 | 34.9 |
| BSc student | 326 | 44.6 |
| Other research workloads | 283 | 38.7 |
| Public involvement in research activities |
129 |
17.6 |
Figure 4: Elements of a clinical academic’s research capacity which should be considered in the RAE (%)
More than half of respondents agree that the best measure of collaborative research with colleagues is the number of contributors to a paper. More than a third of respondents agree that clinical trainees (NHS not University employees) undertaking research projects is a good measure of collaborative research with colleagues and a fifth agree that requests for information on collaborative projects is a good measure (table 8). Other suggested measures of collaborative research include the number of applicants on research grants, joint grants/applications, the number of other specialties/disciplines involved in the research, a statement of contributions in peer reviewed papers, evidence of both NHS and University colleagues on papers and the type of collaboration.
Table 8: Measures of collaborative research with colleagues
| Frequency | % of total respondents |
|
| Number of contributors to paper |
371 | 55.5 |
| Request for information on collaborative projects |
150 | 22.5 |
| Clinical trainees (NHS not University employees) undertaking
research projects |
254 | 38.0 |
| Other | 212 | 31.7 |
Impact of the RAE
Less than two-thirds of respondents (59%) know in which category they will be returned in the forthcoming RAE. Half of these respondents expect to be returned in one of three categories: psychiatry, neuroscience and clinical psychology, other hospital based clinical subjects or cancer studies (table 9).
Table 9: RAE category
| Frequency | Percent | |
| Cardiovascular medicine | 42 | 9.8 |
| Cancer studies | 57 | 13.3 |
| Infection & Immunology | 45 | 10.5 |
| Other Hospital Based Clinical Subjects | 75 | 17.5 |
| Other Laboratory Based Clinical Subjects | 8 | 1.9 |
| Epidemiology & Public Health | 33 | 7.7 |
| Health Services Research | 23 | 5.4 |
| Primary Care & Other Community Based Clinical Subjects
|
24 |
5.6 |
| Psychiatry, Neuroscience & Clinical Psychology | 82 | 19.2 |
| Dentistry | 2 | 0.5 |
| Pre-clinical & Human Biological Sciences | 5 | 1.2 |
| Across panel/multi-disciplinary | 12 | 2.8 |
| Other | 20 | 4.7 |
| Total | 428 | 100.0 |
| No reply | 3 | - |
Half of respondents (51%) report that they are aware of some form of University reorganisation that has taken place to optimise anticipated RAE scores. These include:
- Creation of ‘schools’/research themes/research groups to reflect main RAE categories
- Appointment of high RAE rating staff or staff with a good ‘track record’
- Dismissal or forced resignation of staff, particularly ‘non-productive’ colleagues
- Bullying and harassment of staff who aren’t seen to be ‘performing’ to RAE standards
- Dedicated local RAE committee with the mandate of optimising scores
- Mock RAE exercise/process
- Re-branding of job titles e.g. from lecturer to research fellow
- Closure of laboratories
- Mergers with other universities/departments/schools
- Pre-RAE consultation with all academic staff to maximise paper output in high impact journals.
‘The RAE has had an immensely positive impact by encouraging research collaboration and quality and reducing apathy. It has also directed resource to those with potential to use it or proven ability to use it.’
‘I work in a high-rated unit, which has benefited from its scores; this has enabled infrastructure development (or, more accurately, avoided disastrous infrastructural deterioration). Personally, I was job-hunting just prior to the last RAE and benefited from the ‘transfer market’ that exists in the run-up to the census date.’
‘My publication list led to me being headhunted and now is securing me a senior lecturer post, despite average clinical ability.’
‘The RAE has made me focus on research/other academic activities which I’m good at and spend less time on clinical and management activities, which others can do as well or better.’
Figure 4: How has the RAE impacted on your career? (%)
Two out of five (40%) respondents regarded the RAE as having had a negative impact on their career. Concerns centre on the lack of importance placed on teaching, administration and clinical commitment. Many suggest that the RAE focuses on only one aspect of an academic’s job and that promotion and merit awards are accordingly biased. Others suggest that the RAE is detrimental to collaborative research, as ‘who get the brownie points becomes more important than the actual research’. Some respondents argue that the journals in their sub-specialty have a low impact factor and hence they find it difficult to compete against the more ‘high impact’ sub-specialties. Several respondents suggest that the pressure placed on them to ‘perform’ for the RAE is extremely stressful, to the point where many are considering resigning their posts. The fear of failure is very real for some respondents. The following verbatim comments illustrate these concerns:
‘…as this (the RAE) is the only ‘quality’ measure linked to funding, then career progression depends entirely on research output, rather than a balanced portfolio of research, teaching and scholarship and clinical activity.’
‘It has demoralised staff, discouraged newly qualified pathologists from applying for senior lecturer posts, skewed writing activities away from very valuable reviews, chapters and clinical research publications towards pure laboratory research publications (that tend to be more readily accepted by high impact factor journals), reduced time for training and teaching, and been divisive for co-authors of papers who believe that only 1st and last authors get any credit.’
‘The RAE is a paper pushing exercise – it does not guarantee quality science, simply the ability to jump through hoops. It rewards established departments not new departments, hence inhibits growth.’
‘RAE has severely damaged clinical academic medicine in favour of basic laboratory sciences. Uniquely in Europe we are regarded as ‘second class citizens’, despite making major contributions to improving health care. Many of those left in clinical academic posts no longer practice hands on medicine and the bridge between lab and patient has become longer and longer.’
References