The Process of Developing the DHIS
The DHIS project is born out of the political processes of change in South Africa following the fall of apartheid, and as a synergetic collaboration between public health activists from the anti-apartheid struggle and information system developers from the Scandinavian tradition. DHIS emphasises the use of information for action and improved health services, user participation and ‘live’ (in real contexts), agile and rapid prototyping. The DHIS software development effort was organised within the HISP network, and has since its inception been embedded in a synergetic mixture of public health and participatory design perspectives.
The HISP approach to action research and information systems design was initially influenced by a number of union-based action research projects in Scandinavia in the 1970s and 1980s. The focus in the earlier participatory design projects was on empowering workers who were affected by or threatened by new technology, by exploring ways in which their influence over technological solutions could be ensured. Later projects shifted toward actively producing technological alternatives by involving workers in cooperative design at the workplace. The philosophical grounding of the design approaches followed by HISP may be seen as a continuation of these perspectives, with a goal to explore ways in which disadvantaged communities, regions and countries could appropriate ICTs for their own empowerment. User participation and rapid prototyping in the context of use, combined with capacity building, all at multiple organisational levels of the health hierarchy, became the basis for the development approaches used to pursue this goal. The original key members of the HISP team had backgrounds as social/political activists in the anti-apartheid struggle and other social movements, and were thus explicitly political actors in a larger development process.
The context of South Africa and the process of rapid change following the fall of apartheid became decisive for the design and development of the DHIS, and its robustness through time. In South Africa at that time, the health system and data requirements were extremely fragmented and constantly changing. To give a picture of the fragmentation: During apartheid (1948–1993) and until May 1994, there were 14 departments of health at the central level: the ‘general’ Department of National Health and Population Development, 3 specific ‘white,’ ‘Asian’, and ‘coloured’ administrations, and 10 for ‘blacks’, ‘homelands’, and ‘self-governing states’. As one consequence of this fragmentation, there were numerous data collection tools, procedures and data definitions in use, and there were no national standards for data collection, and, as all the ‘homelands’ were included in the provinces; each province used different data standards.
As DHIS started out to support the decentralisation and integration of the health services, the design focus was on a flexible data structure; it had to be easy to add and change data elements, and to add and remove health facilities and districts, and to change organisational boundaries.
The first phase of DHIS development can be characterised as an intensive three-year evolutionary process of participatory design, which took the system from a district pilot to a country-wide standard for HIS in South Africa. Between the years 1997–1998, HISP developed a free (open-source) database application based on Microsoft Office 97/2000, selected mainly because it was the existing defacto standard amongst potential users.
Development efforts were carried out, in line with participatory design practices, and a series of increasingly refined prototypes were tested in close collaboration with users, to enable information for local action. To some extent, the prevailing post-apartheid reform goals of decentralisation and local empowerment were consciously ‘inscribed’ into the software. Given the agenda of supporting political change in South Africa, the software design process started out with a set of objectives and scenarios the design team wanted to inscribe in the software; that is to try to enable certain ‘desired’ actions through particular functionalities and to make it less for the user to perform ‘undesired’ actions:
- Shift of control of data and information handling from central towards local levels, that is, toward more equal control between central and local levels.
- Local flexibility and user orientation – it should be easy to adapt the software to local conditions.
- Support for health sector reform towards decentralisation and the development of health districts; that is integrating the vertical flows of data from various health programs at district level.
- Empowerment of local management, health workers, and communities – by providing access to their own data on their conditions.
- Horizontal flow of information and knowledge, based on the principle of free access to all anonymous, aggregated health data/information.
These objectives were translated into concrete inscriptions through key principles laid down during the development of the first prototype 1997/1998:
- The application must support the hierarchy of essential datasets, that is, allowing users to add, modify, or delete local data elements, indicators, and so forth.
- The application should be designed in such a way as to support the drive toward decentralised capture, analysis, and use of data – in particular, support the push toward having the facility staff responsible for data collection also doing data capture, quality checking, initial processing, and output.
- The application should be easy to use for new areas (provinces, districts), and should allow users to tailor the geographic scope of their datasets to their needs. This resulted in the use of a front-/back-end solution in Access, where the backend data files covered different geographical areas and the user could switch between them at will.
- The application should as much as possible rely on the flexible and powerful analytical and display tools already available within Office 97, such as Pivot Tables in Excel, even if this increased the learning curve.
- The application should be based on free (open-source) software – both gratis and with free distribution and redistribution of the source code.
Participatory Prototyping of the Early DHIS Application
The first DHIS application was developed in Visual Basic and Access by a team of two core developers and with a team of about 10 HISP members acting as mediators between users and the developers. The first DHIS prototype aimed at capturing and analysing routine monthly data (‘the MD module’), which was released for pilot testing in the HISP pilot districts in March 1998, and went through a series of very rapid prototype cycles during the next 4 to 6 months. New ‘builds’ were sometimes released on a weekly or even daily basis. The informal mechanisms for reporting bugs and requesting new functionality – all tightly integrated with user support – proved popular and encouraged users to provide feedback to the development team. This combined with the rapid deployment of new or corrected versions astounded many users, who previously had experienced many long drawn-out tender processes, fully pre-specified development projects that often ended in frustrating delays or fiascoes. Requests for new functionalities and/or new modules had to be filtered, prioritized and moderated by the HISP development team depending on the number of users making a request, its importance and team capacity, but all relevant requests were logged and prioritised even if they could not be implemented immediately.
The development process went through several phases, emphasising performance and progress and rapid response to user demands over any established prototyping model. Within the institutional framework in which HISP was operating, consisting of a variety of hierarchical levels and organisational and political structures, more formally organised user participation would have been impossible or inefficient. Formal user groups would easily become battlegrounds due to the ongoing large-scale political transformations of South Africa’s administrative structures. The methodology for participation and development used was, thus more informal and to a significant degree based on improvisation, whereby any interested or innovative user, regardless of his or her place in the hierarchy, had full access to the development team – a meritocratic approach. This access was either direct or indirect via the other DHIS trainers/facilitators, and users were encouraged to use whatever channels they preferred. Access was not regulated, but the development team would normally have to guide users to a significant degree in understanding their own requests and how they could be implemented in practice. Such guided user participation was obviously time-consuming and only possible with a limited number of users.
After the first phase of very rapid prototyping, the user base increased and the software and user requests stabilised, and releases of versions became more controlled, and super users in advanced and early districts and provinces were used to test new versions before national releases. By 2001, the DHIS was implemented in all provinces and districts in South Africa.
Expanding the DHIS Project
While the rapid prototyping and iterative design process during the first phase of the DHIS project produced a close fit with the needs for reforming the health system in South Africa, the system accumulated both rigidities and a messy overall architecture. This proved problematic when it was subsequently introduced in countries such as Mozambique, India, Vietnam, and Cuba, after the turn of the millennium. To address this situation, the project embarked on a completely revised and internationalized version of the software in 2004, called DHIS version 1.4. This new development was started from a full remodelling of the database. The developer team was still confined to Cape Town, and employed the same technology (MS Access), but this time, as in contrast with the first phase of development, the users were primarily elsewhere. South Africa kept the existing installations (DHIS version 1.3), which were stable and well-adapted to the local situation. Through extensive travelling of project staff, supplemented by e-mail communication, the new internationalised DHIS (version 1.4) was developed as a participatory process between the developers in Cape Town and implementation teams in Botswana and Zanzibar.
As the new DHIS1.4 was introduced to new countries, the two-person development team became a bottleneck to supporting the expanding network of users with specific requirements. While the technology enabled rapid prototyping, the DHIS version 1 architecture was not suited for distributed development, and because of the small and co-located team, no source code sharing tools had been employed. Another shortcoming of the architecture used for version 1 was the dependence on the Microsoft platform, which meant that even though DHIS itself was open source, it required the full MS Windows and MS Office stack to run. These factors triggered yet another and simultaneous redevelopment of the software. Development of DHIS version 2 began in 2004 under the leadership of the University of Oslo, but aimed at distributing development activities to a number of the countries in the HISP network, in order to bring software development closer to the contexts of use.
A stack of ‘bleeding edge’ Java-based technologies was selected for the development of DHIS2, and in parallel a distributed development platform similar to those employed by many FOSS projects was set-up. However, re-implementing DHIS as a modular web application proved quite difficult for various reasons. The radical break in technologies as well as too much reliance on the new online – electronic medium as in contrast to face-to-face communication presented a formidable obstacle to the involvement of existing technical staff in different countries and sites. The new flexible but complex architecture in fact hindered participatory design efforts in the field, as it took over a year and a half before DHIS2 could initially be deployed, first in a clinic in Kerala, India, and even then much important functionality was lacking. The system improved significantly through early use in India and Vietnam, and later also in Sierra Leone, as well as through the involvement of new software developers recruited locally in countries. While engaging with the global source code, their main task was to support local implementations, in the process trying to bridge the divide between users and developers.
The first real implementation of DHIS2 was in Kerala, India, January 2006, and from there, after a hectic period of development, it was implemented from 2008 onwards in more than 20 Indian states, as described elsewhere in the book. India is by far the largest DHIS2 open source implementation for the health sector in the world.
Open Source and Shared Development
Open Source Software development in the HISP context is intimately linked to the continuous development of the DHIS software for managing health data for decisionmaking in various countries in Africa and Asia over the last 15 years. Since 1997, over 15 years, a tremendous amount of man-hours have gone into the development of the DHIS application, its business logic and use cases, on-site in multiple countries in Africa and in Asia, since its inception in South Africa. For any developing country, to build their own such HIS data warehouse application from scratch, with equivalent levels of embedded knowledge and functionality of the DHIS would be a huge undertaking far beyond individual resources. This is one of the philosophical underpinnings of the HISP and DHIS project, namely that because the creation of software to support HIS in countries is so complex and so huge a task, while the requirements in many countries are quite similar, it makes a lot more sense to collaborate as a big virtual team, than to work in isolation and reinvent the wheel. Distributed development of open source software in collaborative networks of development, implementation and use, including regional and country health authorities, open source communities, and research institutions, make-up both the arena and methodology for the DHIS project.
This ideal aim of distributed and collaborative development of new functionalities and modules into a common shared toolbox, where innovations in individual countries are shared and spread in the network, is, however, a complicated project to achieve. Logically, it can be likened with the shared writing of a big document where a number of writers around the world, all with their own local experiences that they would like to capture and explore further in various subsections of the document. Despite all writers using the same draft_1 as their point of departure, bringing all contributions together in a shared draft_2, is a complicated process. It would require both appropriate tools for editing and a certain level of organisation and specification of responsibilities. While Google Doc would provide a tool for such writing, an editor combined with agreed rules for what to write and how update the document, would make-up the organisation. In open source and distributed software development, such as the DHIS project, similar challenges of coordination and management of the ‘master’ copy, is extremely critical. This is illustrated by the following example:
In 2003, the Ethiopian HISP team developed a module for International Classification of Diseases (ICDs) registration in the DHIS. In order to address the age and sex breakups of ICDs, they developed the option of multi-dimensional data elements, that is, the core data element is the singular ICD code and disease and age groups and male/female represent dimensions of it. This ICD module was elegantly implemented and well received. The problem, however, was that at that time the development of the DHIS was not organised as a distributed project, and the multi-dimensional functionality was not included in the master version. The Ethiopian DHIS version became a ‘fork’, not compatible with the ‘master’ source code. The drawback from this development was as a principle, twofold:
- The Ethiopian HISP team was stuck with the particular DHIS version in which they had hard coded the multi-dimensionality – the DHIS version 1.3.67, and they could not benefit from future improved versions of the DHIS.
- The innovation was not shared and distributed, it did not become part of the shared toolbox, and the global DHIS network was prevented from taking the Ethiopian innovation into use more broadly.
Forking is thus disadvantageous, both for the individual ‘forker’ and for the network as a whole, as sharing necessarily requires a common code base, which is not possible. It was this situation that triggered the DHIS2 development. The DHIS2 started out with a data model quite similar to the redeveloped DHIS1.4, which did not include multi-dimensionality. However, when a similar situation arose in the DHIS2 project; an Ethiopian developer implemented multi-dimensionality as a response to user requirements in trying to manage large and complex datasets in Tajikistan, considerable efforts were invested in making multi-dimensionality part of the DHIS2 meta data model. However, it takes more than a distributed development platform and an enabling architecture to solve the problems of shared and distributed development, as illustrated by the following example:
When DHIS2 was introduced in 2006, the HISP India team had already several years of experience with the DHIS version 1, and mastering the totally different and unknown technologies used in version 2 represented a huge challenge. As the DHIS2 at that time still did not have reporting and graphing tools, the India HISP team needed to develop such reports, but they had the problem of ‘finding’ their data. The database model in version 2 was quite similar to version 1, but the technology differed; version 2 included a database abstraction layer, inscribing the use of a Java Application Programming Interface (API) instead of directly accessing the database. As the Indian developers were unfamiliar with this technology, they bypassed the API and worked directly against the database in developing a range of ‘hard coded’ reports. An additional problem at that time was that DHIS2 had poor capacity to process the large amounts of data being managed in India, and the API was also here seen as a hindrance in providing yet another incentive for working directly on the database; why bother with a slow aggregation of a data mart for general use when you could make specific hard coded reports that were generated much quicker?
The dashboard module developed by the India team illustrates the problem facing the open source sharing philosophy. This module was developed in close collaboration with health managers and enabled ‘on-the-spot’ analysis and presentation of indicators, and became very popular among users. However, as it was a hard-coded workaround, it could not become part of the global repository and toolbox. To solve this problem, the Norwegian and Indian developers spent a considerable amount of time together in reprogramming the module, but also in developing a shared understanding of the DHIS2 architecture and to develop concrete procedures for how to manage development and decision-making in the core cross-country development team. Today the country teams are working more closely together on releases and the merging of code bases.