Monitoring and Impact Evaluation Working Group- Web2fordev
Monitoring & Evaluating Knowledge Management for Development (the notes below are based on an open space session held at the Web2fordev Conference in Rome, Italy. A group of about 15 individuals shared questions and answers about the impacts of KM for development. We agreed to share this information on the KM4dev Wiki so the group can continue to share their collective experience and/or in the name of the KM4Dev Wiki, mercilessly edit!!)
Challenge: How can one capture and measure the benefits of knowledge sharing? The participants in this open space session discussed the challenges associated with assessing the impact or results of a KM activity. We agreed that while one can tally web statistics, this quantitative measure does not speak to the quality of interactions and exchanges that a website might support. Participants shared ideas for how one might qualitatively and quantitatively measure the impacts of knowledge sharing activities.
Goals: • To provide to Donors, Clients and target/user groups information that demonstrates that funding for knowledge management (KM) is well spent. • To allow project teams to meet these groups’ need for quantifiable data.
Getting started: Measuring outputs or outcomes or both? KM efforts produce outputs (e.g. lessons learned, manuals, conference proceedings, workshops, listservs, websites etc.) and they also contribute towards achieving project outcomes (e.g. giving farmers access to market information, improving forest management practices, increasing awareness about health threats). Before setting up a plan for monitoring and evaluation it is important to decide why and how to measure outputs, outcomes or both.
Methods: Ways to measure your KM impact
Qualitative Methods “The Kudos File” A “Kudos File” or compliment file can capture qualitative information that demonstrates how KM efforts are addressing project needs for services (outputs) and goals (outcomes). A “Kudos File” can be kept over the life of the project to capture unsolicited feedback, or feedback in response to a targeted evaluation or survey. It is important to log the time, context of the comment and person giving the feedback. Benefits: This method is simple and low tech. It provides anecdotal evidence of project impacts. Limitations: This method, if not applied with a broad survey or evaluation will not represent larger target audiences.
Outcome mapping Outcome maps can be used to show change: behavioral; market; etc. It can be effectively applied to show project outcomes e.g. for capacity building; are groups trained able to more effectively influence their surroundings? By documenting relationships between actors before and after a project, outcome mapping can show how peoples’ abilities, confidence and behavior have changed. Benefits: This method provides information that demonstrates how the KM efforts in a program contribute to achieving the project goals (outcomes). Limitations: Requires substantial baseline and post-project information about target groups. Outcomes can be affected by forces outside the project (e.g. political or legal changes, other programs, etc.).
Quantitative Web stats Web statistics can be used to trace various output related information on project websites (e.g. users registered, hits, and document downloads). It is also possible to track RSS feed usage through services like FeedBurner. There are emerging standards for web stats that help website managers look at trends in use rather than raw numbers. Benefits: Web stats are quantifiable and show trends in web resource use. Limitations: If you are offering a site that is not mainstream, or not meant to be frequently visited, this is not as useful. Also, bots visit sites and this creates many ‘false’ web hits. Web statistics offer little information about project outcomes. Also, if project KM efforts do not involve a website this is of no use.
Mixed Tools: Survey Tools survey tools like Survey Monkey and others can help to provide qualitative and quantitative information by allowing survey designers to include numerical rankings and qualitative responses. It is important to define what your questions are before starting to design a survey. Are they utilization based (outputs) or objective based (outcomes)? Benefits: Survey can solicit feedback about specific program products (outputs) and progress towards goals (outcomes). Surveys can target larger groups. Electronic surveys provide information for easy trend analysis. Limitations: Surveys only give feedback from those who take the time to respond, potentially biased samples.
Social network Analysis maps connections between groups based on exchange of information and ideas. For web, you can incorporate user stats to map who was downloading different information. Social network visualization software used to map connections. Content is a reflection of the user. Mark access to documents between the actors. (map: notes for actors, color coded with characteristics, lines can have value that you want # of downloads, page hits) 1. Tools: Netdraw- free, other proprietary software. 2. Social network data over time can show behavior change. 3. Challenges: does not necessarily address quality issues or effects of information after download. Benefits: Shows connections between groups (outcomes) affected by the use of KM tools. Limitations: Requires substantial baseline and post-project information about target groups. Outcomes can be affected by forces outside the project (e.g. political or legal changes, other programs, etc.).
Peer groups or focus groups can contribute to finding out what has led to behavior change. Benefits: Focus groups can provide in depth information. Limitations: Small group samples and more work to organize.
Other approaches: Connecting with business goals may be a useful approach to ground the KM work. Donor M&E guidelines can also be useful for determining the outcomes of KM investments.
Resources: UN Secretariat Recommendations for Knowledge Management Netdraw Survey Monkey IDRC Framework for mapping and evaluation Solutions Exchange
Organizations represented: International Development Research Centre International Resources Group UN Biodiversity Initiative CABI - CAB International - Knowledge for Life Imperial College London - Imperial Website International Water Center Helvetas - Schweizer Gesellschaft für internationale Zusammenarbeit IUCN - The World Conservation Union
