Evaluation of the Drinking Water Quality and Health Research Programme (1996-2004)
DWI0848
July 2006


Executive Summary

Defra supports a research programme on drinking water quality and health, which is managed by the Drinking Water Inspectorate (DWI). This programme has a number of important requirements for research, including:
The policy and strategy issues behind the programme are set out in the ROAME statement on the water distribution, conservation and quality research programme.

In order to ensure that the objectives and policy requirements of the programme were being met and that the programme delivers the necessary quality of research, it is necessary to carry out a periodic evaluation of the programme.

The reviewers carried out an evaluation of the 71 projects completed in the period 1996 to 2004. This included a detailed investigation of the project records and the development of a database that could be used for cross-referencing the projects and project files. Electronic versions of the final reports from the projects were made available and of these 63 were evaluated with respect to ten key aspects: Scientific Quality of Output; Performance of Contractor; Performance of Defra Project Officer; Consistency with Project Objectives; Consistency with Programme Objectives; Collaboration with Other Groups; Effectiveness of Dissemination; Perceived Value for Money; Implications for Future Policy Decisions, and Implications for Future Technical Decisions. Each project was scored for each aspect category on a scale of 1 (poor) to 5 (excellent) by at least two reviewers independently and the scores combined. The reviewers also scored whether the project should be considered for in-depth review on a scale of 1 (definite) to 4 (definitely not) in order to select 10 projects for further review in the second phase. There was remarkable consistency between the reviewers, in spite of the fact that the reviews were done blind so that there was no cross influence. The projects selected for detailed review were those rated 1 or 2 by the reviewers. Consideration was also taken of the project size and/or importance, and performance of contractors. Those selected included examples of projects that gave DWI problems and projects of very high quality. Consideration was also given to the relevance of the research. The final list of 10 projects reflected a range of research areas, research topics and contractors, including one large collaborative project.

As well as the ten projects selected for in-depth review, a list was prepared of key stakeholders, e.g., Defra staff, advisors, government customers, external customers and contractors, who were to be interviewed. Of the list of 20 individuals chosen, 18 were prepared to be available for interview. These interviews provided the basis for a number of the comments and recommendations, and in most cases reinforced the opinions of the reviewers gained from the in-depth review of the research programme.

Based on the overall evaluation it was clear that the majority of the research programme had been carried out to a very high standard, and had not only contributed to the technical knowledge required by DWI, informing both policy and technical decisions, but in a number of cases had also had a significant impact on the wider user community beyond the UK. In addition, as part of the programme Defra had contributed to and taken part in the management of a number of significant collaborative projects with other research funders in the UK and the USA.

While the research programme is seen to be generally of a high standard there are, inevitably, several key areas identified as having scope for improvement. A number of recommendations have been made for consideration by the department in the light of current changes in the Departmental management structure.
It is recognised, from discussions with DWI officers during the course of this project, that a number of the recommendations made below are already in the process of implementation by DWI. Notwithstanding that, they are important findings from this review of the Defra drinking water and health research programme from 1996 to 2004 using project records and reports and interviews with both contractors and advisors and customers, and it is important that they are recorded.

Dissemination
Consultation
Collaboration
Project Management

Copies of this report may be available as an Acrobat pdf download under the 'Post 2000 Reports' heading on the DWI website.