Spatial Interpolation and Mapping of Rainfall (SIMAR) Volume 3: Data Merging for Rainfall Map Production
1151/1/04
December 2003
Executive Summary
The programme Spatial Interpolation and Mapping of Rainfall (SIMAR) was a three-year initiative encompassing three component projects, viz:
The final report on this programme, which was undertaken by scientists, researchers and engineers of the METSYS group of the South African Weather Service (SAWS) and the School of Civil Engineering of the University of Natal, in collaboration with the Department of Water Affairs (DWAF) and ESKOM, is contained in three volumes. The volumes are:
VOLUME 1. Maintenance and Upgrading of Radar and Raingauge Infrastructure
VOLUME 2. Radar and Satellite Products
VOLUME 3. Data Merging for Rainfall Map Production
Rationale for SIMAR
Water resources in South Africa are not well buffered against natural rainfall variability. Rainfall deficits and excesses readily translate to droughts and floods, respectively. Well-developed water resource infrastructure, which South Africa is fortunate to possess, has to be managed extremely skillfully to successfully balance water surpluses and deficits at an inter-catchment level, as well as to achieve best trade-offs between flood mitigation and storage maximisation at basin level.
The concept of water resource management is no longer restricted to regulating flows, storage and abstractions in and from rivers, dams and aquifers. Water resource management is increasingly becoming concerned with applying measures to ensure resource (ecosystem) sustainability and also with activities in the catchment which impact on both sustainability and availability of water for abstraction and use. A particular focus in the 1998 Water Act is on activities, termed streamflow reduction activities, the licensing and regulation of which are provided for in the Act. To do this objectively requires defendable information on water usage associated with entities such as forests, agricultural lands, natural veld, farm dams, soil conservation schemes, etc. Such water usage is tightly linked, through the catchment water balance, to catchment water availability and thus rainfall, a link which imposes an obligation on catchment management agencies to obtain detailed and accurate rainfall measurements.
Since natural disasters and fluctuations in agricultural production are also closely linked to rainfall, these sectors have a similar need for such detailed rainfall information.
Raingauges have traditionally provided the rainfall measurements required for water resource management purposes. Because the national raingauge network is rapidly becoming too sparse to meet existing and anticipated management requirements, a new rainfall monitoring/information system, incorporating the optimal use of remote sensing, has become necessary to satisfy the needs of South Africa.
This need was envisaged to be best satisfied through an umbrella research and development programme (SIMAR) having the ultimate goal of merging satellite/radar/gauge data to produce one field that is acceptable to the water-resources (and hence also agricultural and disaster-management) users.
The specific aim was to produce a daily rainfall map of 24 hour accumulated rainfall to a resolution of 2 km, over the whole subcontinent, accessible on the Internet. This has been accomplished. Furthermore, this primary product can and will be refined where needed to finer time scales over selected areas. This refinement will be of particular interest to the disaster-management users and those involved with the mapping of the tendency for, or forecasting of, flash floods. As improvements to the data streams and modelling techniques become available, they will be incorporated into the products emanating from the research. Follow-on research projects supported by the Water Research Commission are designed to bring about such improvements.
Results of SIMAR component projects
Details of the results deriving from the three component projects included under the SIMAR umbrella are contained in the three volumes which make up the SIMAR final report. Some background information pertaining to these results, and the results per se, are summarised below.
VOLUME 1. Maintenance and Upgrading of Radar and Raingauge Infrastructure
Since conventional meteorological infrastructure is dwindling at an alarming rate in South Africa, it became necessary to investigate the complementary use of conventional and less conventional infrastructure in sourcing rainfall data. The complementary sources here considered are surface networks and remote sensing sources, namely radar and satellite. The focus has fallen on maintaining current systems as well as using new technologies and techniques to upgrade systems, where necessary, with a view to securing and sustaining a reliable data flow from the above-mentioned data sources. The specific objectives of this part of the programme were as follows:
The raingauges of the Liebenbergsvlei and Durban networks played a vital role in investigations regarding elimination of ground clutter and in validation of radar-estimated rainfall on the ground. An investigation into the feasibility of using cell phone communication technology and infrastructure resulted in such technology being implemented in the Liebenbergsvlei and Durban networks, and gave rise to the vision of also implementing the technology at all the SA Weather Service second order stations.
Improvements and upgrades were introduced at the majority of radar installations within the network, while also ensuring a reliable power supply to the systems. A remote control and monitoring system, whereby the functioning of individual radar systems can be monitored from a central point, was implemented. This new capability has, for the first time, allowed objective assessments of the reliability of individual radar systems within the NWRN to be made. During the course of this project, the SAWS drastically increased its funding for maintenance and upgrading of the NWRN.
Unfortunately, the Meteosat Second Generation (MSG) satellite did not become operational during the lifetime of the SIMAR project as anticipated, as the MSG programme suffered lengthy delays prior to, and further problems after, the launch of the satellite. Nevertheless, research into utilising Meteosat 7 to fulfill the SIMAR project's objectives, continued. New techniques were developed which will be applied to MSG when it eventually becomes operational.
Close cooperation with the SAWS database developers led to provision being made for the archiving of real-time data generated by the SIMAR programme. The development of the new database has proceeded through different phases and will continue to be developed to accommodate such data products for routine applications and research purposes.
This component project succeeded in promoting data sharing between institutions, albeit on a small scale, and initially limited to operational exchange of data through collaboration with DWAF and Suidwes Agriculture. The pursuit of this objective of institutional collaboration will not end with SIMAR but will be carried on and expanded.
The training of personnel in the maintenance and upgrading of observational systems received high priority in the SIMAR project. The training initiative was also expanded to the international arena with training on SIMAR related subjects presented to students from other African countries such as Botswana and Tanzania.
VOLUME 2. Radar and Satellite Products
The focus of this component project was to provide the two remote sensing-based rainfall fields - the one derived from radar and the other from satellite - to be merged with yet another rainfall field, derived from daily reporting raingauges.
The specific objectives were as follows:
Radar products
Satellite products
Radar rainfall estimation
South Africa's NWRN represents a unique system based on a local solution to the complex problem of networking of several individual radars and merging of their individual data fields. This system, developed in-house, is a combination of South African innovation and shareware/freeware available from various sources in the world. Very few countries in the world operate successful weather radar networks and even fewer a network as elegant and modular as the one in South Africa. The report gives a summary of how this was achieved and highlights the data flow and product generation from the eleven radars within the network.
A major advance in radar rainfall estimation is the unique methodology that was developed to filter the negative impact of ground clutter. This technique, that uses the scan-to-scan coherence in the echo field to dynamically build up a slowly evolving clutter mask, has all but solved the problems of ground clutter contamination. This advance has had a positive impact on the resolution of the rain fields that are archived, displayed and used as input to create the integrated satellite-radar-raingauge rain fields.
The verification of radar performance, and independent procedures and tests to investigate the inter-calibration of network radars, have been fully documented. Of significance is the success with which the sun has been used as an independent calibration source.
Conversion of radar reflectivity into rain rate and the computation of rainfall depth accumulations have been refined considerably. Using the dense Liebenbergsvlei raingauge network as a basis for comparison, the mean reflectivity in the vertical column was found to be a more appropriate input to the Z-R relationship than the customarily-used maximum reflectivity in the vertical column. The mean reflectivity has a smoothing effect on the enhanced or erroneous reflectivity caused by the occurrence of hail, or by the so-called Bright-Band and Anomalous Propagation phenomena.
Furthermore, methods have also been developed to generate a merged rain field from rain fields generated at the individual radar sites instead of from a merged reflectivity field. This allows the use of different (and more appropriate) Z-R relationships for different regions and also allows use of data with the finest temporal resolution. Rainfall estimates over South Africa obtained in this way has been evaluated using rainfall data from 60 automatic weather stations across South Africa.
The above-mentioned studies and advances all provided a sound foundation for improvements already made to the NWRN rainfall estimation techniques, or for improvements due to be implemented in the near future.
Satellite rainfall estimation
Building on a review of literature on past South African and international experience, a technique (probably the most sophisticated satellite-rainfall estimation technique yet available for South Africa) that makes optimal use of all three channels (IR, Visible, Water Vapour) of the current Meteosat 7 satellite was developed and implemented operationally. Particular attention was also given to the characteristics of the MSG Satellite. Although it did not become operational during the project as originally anticipated, some of the first data examples from this satellite are shown.
The first stage in the evolutionary development of the MSRR (Multi-Spectral Rain Rate technique) was the ITR (Infra-red Power Law Rain Rate technique), which gave rise to the intermediate BSRR (Bi-Spectral Rain Rate technique). The use of all three channels leads to improved methods for filtering out non-precipitating clouds and enhances the estimation of rainfall from maritime clouds. Image processing techniques (including edge detection and speckle removal techniques) were also introduced to better identify rainy pixels from those that are cloudy but not rainy. Systems to reformat and communicate the satellite data in the same MDV format being used for radar data were developed. A novel development in the satellite rainfall estimation was the use of topographical slope to enhance estimated rainfall over mountainous regions. The advantages of using a Geographical Information System to display and process the data and products from the various sources was clearly demonstrated.
Methods to verify the satellite rainfall estimates in terms of their spatial extent and quantitative values through comparisons with radar and raingauge estimates were developed and applied. It became clear that the satellite rainfall estimation technique which was developed achieves the objective of providing useful, large-scale rainfall fields for the southern Africa region.
Rainfall data integration and product distribution
An analysis of the strengths and weaknesses of raingauge-, radar- and satellite-derived rainfall information provided the basis for additional measures to address the major weaknesses in each source. The accuracy (including human-introduced error factors) and coverage of daily raingauge data, in particular, has become a matter that needs to be addressed as a national priority.
The generation of the merged satellite-radar-raingauge field is a stepwise process, starting with the merging of the radar and raingauge fields. Thereafter the satellite and raingauge fields are merged before the two resultant fields are combined.
SIMAR has a dedicated section on the South African Weather Service (METSYS) web page (http://www.weathersa.co.za) which displays the various individual daily rainfall fields (radar, gauge and satellite) together with the integrated fields. Archived data relating to these fields are also presented.
VOLUME 3. Data Merging for Rainfall Map Production
This component project was initiated against the background of the following premises and objectives:
Theoretical development
Combining the precision of raingauge data with the coverage of satellite data and the detail of radar data was, in effect, an important objective of this research. The techniques initially envisaged as a means of achieving this were optimal spatial interpolation using a technique called Kriging and an associated one called co-Kriging.
It turned out that co-Kriging was not a good option because of the large computational load. This load comes from the fact that there are of the order of one million small areas (pixels) approximately 1.5 kilometers square (the typical spatial resolution of a weather radar) covering the subcontinent and the surrounding oceans. The challenge was to be able to map the country's rainfall routinely to that detail. Even the quadrupling of computer speed, between the year 2000 (when the project was proposed) and its end in 2003, did not diminish the need to find a better way to process data, which would be easy to automate. A method of Kriging, exploiting the efficiency of the Fast Fourier Transform, was consequently developed. The process of development necessitated having to deal with a highly technical subject, involving some difficult and advanced mathematical ideas and theory.
Outcome and Technology Transfer
The techniques developed for optimal integration (merging) of data fields and their implementation to date have been most fruitful. The daily rainfall maps on the SAWS:METSYS website bear testimony to this successful outcome.
The Fast Fourier Transform approach to Kriging provided the basis for the coding of an algorithm to accomplish the massive computing task efficiently and speedily. Speed is of the essence in the delivery of the daily rainfall maps in real time. Information on the accumulated rainfall for the 24 hours until 8:00 am SA time, derived from the recording raingauges around the country, arrives at METSYS (Bethlehem) by 9:00 am daily. By that time, the previous 24 hours' satellite and radar images will have been used to produce the best estimates, respectively, of the rainfall totals per pixel over the whole area. The merging of the three fields: gauge, radar and satellite is then done and the result posted on the METSYS web-site by 11:30 am. A thorough description of the practical implementation of this methodology is presented in the body of VOLUME 2 of the SIMAR final report. Some examples of the website output are reproduced therein.
Conclusions and recommendations
SIMAR has successfully met its objectives and laid the foundation for a national (and potentially regional) rainfall observing system which promises to meet all reasonable requirements regarding spatial and temporal resolution and real-time availability of data.
There are several areas in which the current SIMAR system, with further attention to data availability, research and development, can be improved. These are:
The SIMAR system would also benefit from certain infrastructural improvements, the most crucial of these being:
The above envisaged improvements build on the existing platform of work developed under SIMAR and will make a considerably more acceptable product. Follow-on projects already under way are addressing these issues with energy.
Capacity development
The capacity developed at both technical and professional levels through SIMAR has provided a sound foundation upon which further capacity can be built.
The training of technical personnel in the maintenance and upgrading of observational systems received high priority in the SIMAR programme. Four individuals from the previously disadvantaged groups were trained in maintaining the electronic observational infrastructure thus ensuring the long-term sustainability of the observing systems. Training was conducted in-house, through courses as well as self-development. The training initiative was also expanded to the international arena with training on SIMAR related subjects presented to students from other African countries such as Botswana and Tanzania. As SIMAR products are used routinely by institutions, training will continue well beyond the lifetime of this project. It is especially training in the utilisation and interpretation of SIMAR products where a strong need exists. Users should also be trained and educated in the use of remotely sensed data, its advantages as well as its limitations.
There are two aspects to professional capacity building which were achieved here - indirect (people being exposed to the ideas and concepts but not working on the project) and direct (those people personally involved with aspects of the project). In addition, there was a strong component of Competency Development as a direct result of the project.
Indirect Capacity Development.
In the Hydrology Section of Umgeni Water, where one of the researchers (Scott Sinclair) worked in 2001 and 2002, two PDIs were kept abreast of the developments of the project in both informal and formal (reports, presentations) ways. The 2002 final year class of 28 Civil Engineering Students in Hydrology at the University of Natal, Durban, contained 16 PDIs (of whom 5 were women) and 2 white women. The project co-leader (Geoff Pegram) made frequent reference to the SIMAR in class and repeated the oral presentations given in this regard at the European Geophysical Society in Nice in April 2002. These presentations tempted two students from previously disadvantaged backgrounds to undertake dissertations under the project co-leader's supervision during the second semester of 2001
Direct Capacity Development.
In the second semester of 2001, a female final year student, Deanne Everitt, undertook a dissertation study under the supervision of the project leader entitled "Flood Impacts: Planning and Management". This was an overview study making use of the output from SIMAR, with special focus on the Umlazi catchment in Durban. A later addition to the team was Nokuphumula (Phums) Mkwananzi, a practising Engineer, who registered for an MScEng at Natal University under the supervision of the project co-leader in 2002, worked on the WRC project "Extension of Research on River Flow Nowcasting to include Levels of Inundation" which depended on SIMAR input of rainfields, and completed his Masters in September 2003.
Competency Development
Because of the nature of the Research, a number of people in Umgeni Water, Durban Metro/eThekwini Municipality, SAWS: METSYS and the University of Natal have been exposed to new ideas and potentials for ameliorating flood damages using the ideas that are direct spinoffs from SIMAR; new technology has been developed and existing technology has been improved and refined. Every individual involved has grown in competence and benefited from the project; in the long run the wider community in the region will be beneficiaries.
Knowledge dissemination
Knowledge generated by SIMAR has been disseminated through peer-reviewed articles, conference presentations, workshops and during international visits. These include the annual South African Society for Atmospheric Sciences (SASAS) conferences. The SASAS conference that coincided with the World Summit on Sustainable Development (WSSD) in 2002 provided an international platform for three SIMAR presentations. Members of the SIMAR team also used opportunities during visits to Lesotho, Botswana, Mozambique, Burkina Faso and the Kenya Institute for Meteorological Training and Research as well as the Drought Monitoring Centre to present the progress within SIMAR. The potential agricultural applications of SIMAR products were presented by members of the research team at a workshop organised by an agricultural service provider.
An important means of relatively quick dissemination of the ideas that are the outcomes of research are via presentations at conferences and Symposia. Such presentations at National and International Fora include the following:
National:
International:
In yet other ways, SIMAR benefited substantially from international exchanges of knowledge. Initiatives to present data and results led to fruitful discussions and the pursuit of new ideas. In particular, Professor Geoff Pegram was active in fostering Australian and European links, as marked by the following personal invitations:
The knowledge gained by these interactions has benefited not only the participants in SIMAR but has already realized its potential to benefit the post-graduate students working on on-going projects which are out-growths of the Water Research Commission's investment in SIMAR.