Evaluation of the Drinking Water
Quality and Health Research Programme (1996-2004)
DWI0848
July 2006
Executive Summary
Defra supports a research programme on drinking water quality and
health, which is managed by the Drinking Water Inspectorate (DWI). This
programme has a number of important requirements for research,
including:
- Providing a sound science base for informing policy and
influencing the formulation of policy, for contributing to the
activities of international bodies that are influential in the
formulation of policy and standards;
- Providing good technical information that will contribute
to understanding the impact of drinking water contaminants on public
health and consumer acceptability;
- Obtaining technical information to assist the Inspectorate
in an efficient and proportionate manner;
- Providing a basis for judging other research that might
impact regulatory activity and for assessing the social and regulatory
impact of regulatory decisions.
The policy and strategy issues behind the programme are set out in the
ROAME statement on the water distribution, conservation and quality
research programme.
In order to ensure that the objectives and policy requirements of the
programme were being met and that the programme delivers the necessary
quality of research, it is necessary to carry out a periodic evaluation
of the programme.
The reviewers carried out an evaluation of the 71 projects completed in
the period 1996 to 2004. This included a detailed investigation of the
project records and the development of a database that could be used
for cross-referencing the projects and project files. Electronic
versions of the final reports from the projects were made available and
of these 63 were evaluated with respect to ten key aspects: Scientific
Quality of Output; Performance of Contractor; Performance of Defra
Project Officer; Consistency with Project Objectives; Consistency with
Programme Objectives; Collaboration with Other Groups; Effectiveness of
Dissemination; Perceived Value for Money; Implications for Future
Policy Decisions, and Implications for Future Technical Decisions. Each
project was scored for each aspect category on a scale of 1 (poor) to 5
(excellent) by at least two reviewers independently and the scores
combined. The reviewers also scored whether the project should be
considered for in-depth review on a scale of 1 (definite) to 4
(definitely not) in order to select 10 projects for further review in
the second phase. There was remarkable consistency between the
reviewers, in spite of the fact that the reviews were done blind so
that there was no cross influence. The projects selected for detailed
review were those rated 1 or 2 by the reviewers. Consideration was also
taken of the project size and/or importance, and performance of
contractors. Those selected included examples of projects that gave DWI
problems and projects of very high quality. Consideration was also
given to the relevance of the research. The final list of 10 projects
reflected a range of research areas, research topics and contractors,
including one large collaborative project.
As well as the ten projects selected for in-depth review, a list was
prepared of key stakeholders, e.g., Defra staff, advisors, government
customers, external customers and contractors, who were to be
interviewed. Of the list of 20 individuals chosen, 18 were prepared to
be available for interview. These interviews provided the basis for a
number of the comments and recommendations, and in most cases
reinforced the opinions of the reviewers gained from the in-depth
review of the research programme.
Based on the overall evaluation it was clear that the majority of the
research programme had been carried out to a very high standard, and
had not only contributed to the technical knowledge required by DWI,
informing both policy and technical decisions, but in a number of cases
had also had a significant impact on the wider user community beyond
the UK. In addition, as part of the programme Defra had contributed to
and taken part in the management of a number of significant
collaborative projects with other research funders in the UK and the
USA.
While the research programme is seen to be generally of a high standard
there are, inevitably, several key areas identified as having scope for
improvement. A number of recommendations have been made for
consideration by the department in the light of current changes in the
Departmental management structure.
It is recognised, from discussions with DWI officers during the course
of this project, that a number of the recommendations made below are
already in the process of implementation by DWI. Notwithstanding that,
they are important findings from this review of the Defra drinking
water and health research programme from 1996 to 2004 using project
records and reports and interviews with both contractors and advisors
and customers, and it is important that they are recorded.
Dissemination
- Consideration should be given to making Defra funded
research outputs freely available (in PDF or MS WORD format).
- Consideration should be given to the incorporation of a
dissemination plan into the research planning process.
- Consideration should be given to preparing an annual
summary of research conducted under the DWQH with details of individual
projects.
- Consideration should be given to holding regular research
workshops; these could be held annually, early in the financial year.
Consultation
- Consideration should be given to the need for wider
consultation regarding research priorities and specific research
topics. This might include representatives from the wider water
community, academia and major research consultancies.
- Defra should make greater use of existing advisors and
advisory groups, such as the Committee on Products and Processes (CPP)
and Health Protection Agency (HPA), in formulating research objectives,
including taking advantage of existing research programmes.
- Consideration should be given to the value of forming a
DWQH research steering group that could involve external stakeholders.
This body could scrutinise the proposed research programme and comment
on whether it covers the areas of importance, provides value for money
and is complementary to other research initiatives.
Collaboration
- Consideration should be given to closer collaboration with
other research funding bodies.
- Consideration should be given to whether it is possible to
encourage more collaboration between contractors to take greater
advantage of complementary skills. This could be included in the call
for tenders or by inviting contractors to put in a collaborative bid
for a particular specialist topic.
Project Management
- Consideration should be given to widening the number of
Project Officers to emphasise the importance of this role.
- Consideration should be given to the inclusion of a brief
statement on file as to the reason why the research has been
commissioned, and how this is expected to meet the objectives set out
in the ROAME statement.
- It is essential that the full details of project records
and files be maintained so that the project records are both complete
and up to date.
- Consideration should be given to increasing the time given
to potential contractors to provide tenders in response to an
invitation to tender. This could be four weeks for projects of six
months or greater duration.
- Consideration should be given to building closer and more
regular contacts between Project Officers and contractors into the
specification of research contracts.
- Consideration should be given to the involvement of an
independent expert to support the DWI to achieve the best outcome from
a project.
- A system of formal wrap-up meetings at the end of projects
should be instigated.
- The detail of project records and files needs to be
maintained so as to be complete and up to date. This should also
include all electronic correspondence of substance and will be aided by
the new project filing and control system being established.
Copies of this report may be available as an Acrobat pdf download under the 'Find Completed Research' heading on the DWI website.