VA
Insights

Bill Gates emphasized the important role of philanthropic evaluation in the open letter from Bill & Melinda Gates Foundation at the beginning of 2013. Coincidentally, social organization evaluation has become a hot topic in the past two years as being carried out at both national and local level. In October, Bill & Melinda Gates Foundation held a symposium with the theme of national evaluation. VA was invited to speak on national evaluation indicators. In this article, we will elaborate on this topic and share with you problems and trends of evaluation in China’s third sector.

Problems

The government has led evaluation research and pilot since 2005, but did not reach a national scale until public service procurement achieved breakthrough in the last few years. However, the evaluation system was flawed:

• The evaluation indicator system puts too much weight on standardization and process management but too little on program implementation and social impact. It also lacks systematic and customized methodology for the sophisticated landscape of NGOs in China.

• Local evaluation capacity fails to catch up with development in this field; regional evaluation institutions emerge too quickly with no regulation or certification, which leads to unqualified evaluations.

• Evaluation results are intended for public service procurement decisions in most cases and lack transparency. Without participation of multiple stakeholders, government rating lacks public accountability.

We believe that national evaluation has great potential as a remarkable political effort in public administration: to promote the development of professional evaluation institutions and standardized evaluation framework with government resource network and mobilization. However, there are still a lot to be done to realize that potential.

Compared with government-led evaluation, efforts made by nongovernment bodies started earlier (around 2002), mainly pushed by international NGOs and academic institutions. After 2008 – the “dawn of philanthropy era”- a number of independent evaluation organizations emerged and became main drivers in the field. Regrettably, after 10 years of development, there are still some gaps to be met:

• The number of professional evaluation institutions is still limited (no more than 10). And they rarely communicate with one another due to distance or lack communication platforms.

• Due to lack of accountability in the third sector of China, philanthropic evaluation is regarded mostly as a tool to demonstrate credibility to the public or donors while its potential in improving organizational capacity and strategy is yet to be fully recognized.

• The value chain remains to be completed and enhanced. Relevant research is still at nascent stage. There are few advocacy, communication and training in this field in contrast with less technical subjects such as promotion and marketing.

Trends and Highlights

FSG, a leading nonprofit consulting firm, recently published an article that reflected upon the status quo of philanthropic evaluation and shared 3 kinds of practices that highlighted characteristics of “next generation evaluation: 1) using big data; 2) involvement of multiple stakeholders; 3) developmental evaluation that informs and supports innovative and adaptive development of evaluation models in complex, dynamic environments. Back to the development of philanthropic evaluation in China: it is frustrating that there is no “generation” of evaluation theory, research, or cases. However, we would like to highlight some findings that may represent the trend of evaluation in China:

1. Use new media as a data collection and monitoring tool. Weibo, Wechat, BBS, SNS enable program teams, volunteers and beneficiaries to conveniently perverse all kinds of program traits and even specific scenes. These data can be used for summative evaluation as well as real-time coordination between evaluators and program team —— M&E (Monitoring and Evaluation) mechanism helps enhance the program management.

Take Dream Box of Adream Foundation as an example. Dream Box is a SNS of all teachers in Dream Centers (the customized learning centers for students) built by Adream. Rather than using traditional labor-intensive methods such as school visits or phone interviews to collect information, with Dream Box program managers can easily collect amounts of feedbacks from front-line teachers. Data stored in the Box is also a good source of information for future impact evaluation.

2. Capacity building for participatory evaluation. External evaluation is admired for its professional and fresh viewpoint but it also brings up the challenge of data collection. In most cases, the quality of data available largely depends on program team’s sense of data management and technique. This requires evaluators to actively communicate with the project team or even take the role of trainers to improve the project team’s capability.

An independent evaluation institute in Guangdong prioritized evaluation capacity building when they conducted evaluation for local government’s social investment. Plenty of interviews and evaluation skills training were carried out at the design stage of the programs. It resulted in more efficient data collection, reciprocal evaluation process and program management improvement.

3. Strategic evaluation. Evaluation is usually neglected or seen as a burden by many NGOs. This is mainly because they see evaluation as a must-do “homework” for the public and donors, not realizing its value to organizational capacity building and strategic planning planning.

In the evaluations that VA conducted, the connection between programs and organizational strategy was always emphasized. From the design of indicators to the presentation of evaluation reports, VA is concerned with how the evaluation process and results can guide NGO strategic planning, and hopes to help NGOs shift their mind, i.e., design projects according to strategy and refine strategy based on project feedback.

The existing problems and development trends of philanthropic evaluation in China prove the trial and error that practitioners are experiencing when driving this career forward. Let’s look forward to the “next generation of evaluation”: more innovative and professional practices, more communication and engagement, and more effectiveness and efficiency.