Programme Assessment through Social Accountability Tools (SATs)

Most governments are responsible for providing essential services to the people. They spend huge amounts of resources to provide services such as drinking water, education, health care and sanitation as well as electricity, roads and transportation. Citizens depend on these services for their security and livelihood.

When basic human rights of the people, guaranteed by the Constitution are not adhered to, there is no option for citizens but to engage in collective action to demand better governance and accountability from those in authority, through “voice” of the people. Citizens, both individuals and groups, (civil society) with the required resources, capacity and numerical strength are likely to exert pressure for change on the state through their voice.

There are several reasons why civil society groups began to proactively monitor government programmes and services in recent decades. First, there was growing evidence of public dissatisfaction with the performance of governments and their departments. In India and other developing countries, the vast majority of the people depend on government for essential services and entitlements. Failure of service delivery was a serious matter for them. Second, access to information exacerbated the problem and made people suspect that corruption and other abuses of power in government were on the rise. Third, lack of access to essential services adversely impacted the poor, and amounted to a denial of the basic human rights. And even those who had access found it cumbersome to interface with service providers and resolve their problems. It is a mix of these factors that led to the emergence of a variety of tools and approaches that helped civic groups to monitor the functioning of governments and demand increased accountability and transparency.

Tools that can be used to hold service providers accountable have been used by civil society activists at the sub-national level in India can be divided into four categories:

•  Social audits in the form of local public hearings
•  Community Score Cards – used at local levels
•  Citizen Report Cards – used at multiple levels
•  Public Expenditure Tracking and Budget Analysis

These tools are meant to strengthen the demand side of governance and focus on the use of public (user) feedback on the services and entitlements that governments provide to citizens. Though the scope and methodologies involved may vary, their primary goal is to use the feedback to seek remedial action or in a broader sense, demand accountability from public service providers or other agencies of government. In recent years, governments have also used some of these tools and have listened to their findings. A common feature of the tools is that they seek information on the experience people have with public services and programmes. The data thus collected are aggregated and analysed, and findings and recommendations for action are derived from such analyses of experience.

Citizen Report Cards (CRCs) and Community Score Cards (CSCs)

The Citizen Report Card (CRC), pioneered by the Public Affairs Centre (PAC), Bangalore, is a simple but powerful tool to provide public agencies with systematic feedback from users of public services. CRCs gain such feedback through sample surveys on aspects of service quality that users know best, and enable public agencies to identify strengths and weaknesses in their work. Some important characteristics of Citizen Report Cards are:

• A “bottom-up” approach to reform measures. This approach is effective in identifying key constraints that citizens (especially the poor and underserved) face in accessing public services, and benchmarking the quality and effectiveness of public services staff.
• The use of quantitative and statistical methods. Data is collected via a random sample, and is then aggregated and used as a basis to analyse public services. Quantitative data is used to assess overall service delivery, as well as specific aspects of public services.
• Simple and unambiguous measures of satisfaction. CRC surveys ask individuals to rate each public service in quantitative terms. But these surveys also glean information on a number of components of each service, in order to recognize the underlying reasons for collective opinion.
• The recognition of the importance of citizens’ thoughts on policy. The CRC survey asks questions not only about the current state of public services, but also about what policies they themselves would like to see implemented.
• Credibility and reliability. By using a carefully designed methodology and by surveying a diverse sample of users, CRC findings are met with a high level of confidence.

Purposes for which a Citizen Report Card can be used

• As a diagnostic tool. CRCs provide information on more than just total satisfaction and its components. It can help in identifying gaps and inequalities in service delivery, as well as in assessing citizens’ awareness of their rights and responsibilities.
• As a means to improve accountability: CRCs can potentially reveal areas where the institutions responsible for service provision have not fulfilled their obligations, and translate findings into ‘rights based’ advocacy statements and positions.
• To benchmark changes: When conducted periodically, CRCs can track variations in service quality over time, which can generate pressure on poor performers to improve the quality of services.
• To reveal hidden costs: A powerful outcome of CRCs is the generation of user feedback on hidden costs like bribes. Furthermore, the nature of corruption (whether bribes are paid voluntarily or extorted) and the size of payments can be effectively highlighted and tracked. Feedback can also be used to estimate the amount of private resources spent to compensate for poor service provision (e.g. water purifiers, voltage stabilizers, private tuition, etc.).
• A self examination on the part of government: CRCs provide valuable information to the government itself. Institutions undertaking a program to improve services could use such projects to evaluate their own performance and to determine the types of changes that are necessary.

Key stages of a Citizen Report Card study

• Assessing the applicability of Citizen Report Cards. There are certain enabling conditions which enhance the applicability of CRCs like existing political contexts, extent of decentralization, ability of citizens to voice opinions and experiences freely, local competency to carry out surveys and advocacy etc. Public Affairs Centre (PAC) has developed a structured assessment exercise involving various stakeholders to explore the “fit” of the tool to the context.
• Scoping & Planning. Once the feasibility of CRCs are established, the next step is to carry out series of pre-survey activities like (a) Assessment of institutional setting; (b) Identifying key sectors/services for the probe; (c) Mapping of service provisions; (d) Orienting local partners to the concept and methodology of CRCs.
• The design of the questionnaire. Focus Group Discussions (FGDs) involving both service providers and users are necessary to provide inputs useful in designing the questionnaire. Providers of services may indicate not only what they have been mandated to provide, but also areas where feedback from clients can improve their services. Users may give their initial impressions of the service, so that areas that need attention can be determined.
• Sampling. To collect feedback from the entire population would require too much time and resources. Sampling, when carried out accurately, gathers feedback from a sample group that is representative of the larger population. The appropriate type of sampling design must be determined. A group’s knowledge of statistics and its prior experience in developing a sampling plan is necessary, although it may also be useful to consult an expert on sampling techniques if the population in question is complex.
• The execution of the survey. First, a cadre of survey personnel must be selected and trained. Second, to ensure that recording of household information is accurate, random spot monitoring of question sessions must be performed after a certain proportion of interviews are complete. Third, upon completion of each interview, the enumerator must go over the information collected and identify any inconsistencies.
• Data analysis. Data is anlysed in term of averages, percentages and in some cases, regression is also carried out. A typical finding may look like this: “Girls (20%) tend to drop out of school more than boys (11%). Of those children who drop out of elementary school, 60% do so in grades 4 and 5.” Another example is “On an average, users of handpumps have to wait for 30 minutes to collect water”.
• Dissemination. There are three important points to consider with regard to the dissemination of CRC findings:
  • Findings should be constructively critical, and should not aim to either embarrass or laud a service provider’s performance. Toward this goal, these service providers should be involved throughout the process, in order to share with them preliminary findings, and to gain their feedback.
  • The media is the biggest ally for dissemination. Beyond the publicizing of findings via a high-profile press conference, coverage should be widely dispersed, such that these findings cannot be easily ignored. This may require the preparation of press kits with small printable stories, media-friendly press releases, and the translation of the main report into local languages.
  • A direct interaction between the two concerned parties provides the link between information and action. Following the publication of the citizen report cards, service providers and users should meet in a town-hall type setting. This not only allows for a constructive dialogue, but also puts pressure on service providers to improve their performance for the next round. If more than one agency is being evaluated, these settings can foster a sense of healthy competition among them.
• Advocacy & Service Improvements. The dissemination of the findings of the pilot Citizen Report Card is extremely critical to derive the maximum benefits from the effort. The usefulness of the Citizen Report Card will be quite limited if findings are not shared and used to bring about improvements in public service delivery. Key activities include carrying out stakeholder analysis, targeting audience for dissemination, building coalitions etc.

Applications of Citizen Report Cards

• CRCs have resulted in public service agencies discussing performance with citizens in open fora in Bangalore.
• CRCs have empowered citizens to play the role of a watch-dog to monitor local government functions in three cities in Ukraine.
• In the Philippines, CRC findings were used to streamline and prioritize budget allocations.
• CRCs were used as independent approaches to monitor pro-poor services in Zanzibar, Ethiopia & Tajikistan.
• In Mumbai , India, CRC findings were cited by lower level officials to seek more resources to function effectively.
• The Chief Minister of Delhi, India used CRC as a means to seek direct feedback from citizens.
• In Kenya , CRCs are being used to strengthen consumer voice in the water sector.
• CRCs have also been used to monitoring post- Tsunami rehabilitation interventions in Sri Lanka.

Community Score Cards (CSCs) represent a more structured version of the social audit approach. It comprises of a four step methodology that involves preparatory groundwork (collection of secondary data, user information regarding the service and preparing indicator matrix); community assessment (sharing of rights and entitlements with the user community, development of community generated indicators and scoring against those indicators based on user experience); provider self-evaluation (rating of their own performance against the same indicators developed by the community); and interface meetings (sharing of scores in the presence of both the community and the unit-level service provider, leading to dialogue, discussion and join action plan for monitoring of service delivery).

The location and scope of the meeting will be determined by the nature of the services/programmes under review. Village level meetings are an accepted medium for gathering user feedback. As village populations are relatively small, meetings are easier to organize. In a city, meetings can be easily organized in slum communities. A facilitator may guide the deliberations so that answers can be elicited on specific issues or aspects of services. In a CSC done by Public Affairs Centre in Bangalore on maternity home services of the Municipal Corporation, community members were asked to rate services on aspects such as availability of the medical services and medicines (access), access to ancillary services such as toilets, display of information, cleanliness of the facility, staff availability, staff behaviour and courtesy, counselling and advice, user fee payment. The rating was done by the users on a scale of 1-5. In the interface meetings that followed, monitoring committees were formed by the users with the approval of the providers, leading to regular monitoring of these maternity homes and curbing of corrupt practices such as demand for extra money for various services by the staff.

In Maharashtra, CSCs have been done on drinking water supply at the panchayat level. In Madhya Pradesh, NGOs have done CSCs on local health and education services. The small scale and local nature of CSCs has made their visibility rather low in comparison to CRCs and the widely publicized social audits. The CSC can be a useful diagnostic aid, if corrective actions can be taken at the village or community level for that unit of service delivery. But if the service provider has a wider catchment area or if corrective actions can be taken only at higher levels and for larger units, then similar CSCs will have to be prepared for all communities and aggregated. The positive feature of a CSC is that it offers scope for members of the local community to participate in a process that seeks their feedback and follows up on their problems and possible remedies. Their motivation to monitor and demand accountability from the service provider may thus also be higher.