Impact and M&E of KM

This is a first summary of a discussion that was launched previous to the KM4Dev Meeting 2008

Discussion starter

• What are your big questions re. impact or measurement of KM initiatives? Alternatively, what have you been trying to do to find out out how well your KM initiative is doing? How you been able to "prove" that it has born fruit?

Suggested Summary Graph

http://www.flickr.com/photos/8764209@N07/2536430327/

1) Impact of KM and KM as a complex, long-term process

• How you improve your own knowledge about these complex longer term impacts and how you convince those that you are working for of these impacts?
• Using narrative (stories / anecdotes)as a way to determine impact might be one way to uncover how KM (and project) initiatives actually work.
• How to ensure that the success communicated in the stories (e.g. the new way of doing something) becomes sustainable?
• Those who write good stories tend to get more funding.
• Participatory approaches: Design evaluation processes that involve as much as possible the planners and implementers of activities being studied. Collective reflection.
• Quarterly small group sharing: After one-year of sharing. the impact was the small group were able to lobby more on funding policy, donor design, project strategy and even network on advocacy for policy changes
• Social Return on Investment (SROI): a methodology which monetizes both the inputs and outcomes of a project which has social impact, to determine whether it made a contribution that justified its costs. See guide of New Economics Foundation: http://www.neweconomics.org/gen/z_sys_publicationdetail.aspx?pid=180 SROI analysis starts by identifying who are the relevant stakeholders for a project: those who are most significantly affected. Then it explores (through interviews or otherwise) what they put into the project (time, money, etc) and what they get out (e.g. improved performance in their work). By estimating the economic value of the inputs and the outputs for all relevant stakeholders, then calculating the present value of future costs/benefits it is possible to determine whether the social impact justifies the total expense.
• The Zooming Model in Chimanimani District of Zimbabwe: building on community knowledge and value rather than starting with what has worked elsewhere plus development of Stories of Change
• Most Admired Knowledge Enterprises (MAKE) Rankings methodology
• Process documentation (PDoc) is a tool that helps project staff and stakeholders to carefully track meaningful events in their projects. Process documentation not only describes what happened, but it also explains how it happened and investigates why things happened the way they did. These insights are crucial in taking innovation processes to scale and adapting them to other locations and contexts.
• Measure acquired knowledge (insight) and then applied (or adopted) knowledge. And measure the outcome on a 3-tier level: institution, trainee (the individual who accessed KM) and the satiesfaction of its client, individuals or groups, community level (if applicable)
• On measuring success (if not impact): http://www.greenchameleon.com/gc/blog_detail/how_to_use_kpis_in_knowledge_management/

2) M&E of KM efficiency and effectiveness

• Let’s not waste our time and rather focus on much more realistic monitoring of efficiency and effectiveness of KM in the KM4Dev context.
• The real question for us in assessing KM initiatives is whether this was a cost effective way of contributing to the goals of the organization
• I am getting the impression that in trying to demystifying KM, we are actually mystifying it.
• Demystifying KM. Basic needs and activities like - how to make sense out of information and present it in a form that is relevant , timely and quality assured.
• This discussion is extremely relevant for Almada and the Saturday discussions on the KM4Dev community's future: what are our objectives, what are our indicators, how are we going to monitor?
• Perhaps importantly we need to separate out in this discussion the issues to do with KNOWLEDGE and those to do with KM as in 'structured information processes that are generated for and stored within organisational boundaries'
• Need to look at the INTERVENTIONs and their effect. What kind of interventions have been tried and were successful or not in which organisations? Do you feel intranets have an important impact or networks or learning organisation scans? Did they deal with the obstacles in knowledge sharing or not? What ARE the most obvious obstacles in knowledge sharing in development? Does the preoccupation with evaluations (which is part of the DNA of knowledge processes in my opinion) inhibit knowledge sharing or contribute to it? And how are we linked to the three waves in knowledge management: 1. Focus on IT and databases 2. Focus on connecting people 3. Focus on knowledge as a strategic asset. May be we need more bridging mechanisms between north/south?
• Quantitative organizational and donor requirements: such Quantitative indicators are used to justify funding or projects or programs because they can present an appearance of cause and effect in a relatively quick and easy-to-follow way for busy decision-makers
• Relationship between KM and the management of an organization as such: Make sure that monitoring is actually USED in management at the earliest possible opportunity
• Who needs to know about impact? Why? Who needs to prove to whom what kind of impact is achieved? On the one hand one would want good ways to measure impact of KM, on the other hand it seems the KM that has the most impact is the one where "management" studiously ignores it -> role of organizational systemics
• I often have the impression KM doesn't drive institutional learning, its institutional politics that drive KM actions for reasons that have little to do with learning.
• I would love to hear from any donors on this list as to how they see all of this. Also, is anyone out there trying to do something to change the donor mindset?
• The challenge is to figure out what the donors want to hear and what not.
• Systemically speaking almost all reorganization, new policy, studies, KM etc. is undertaken by Donor in order to sustain its organization (ensure continued access to funding) and in order to not have to change the way it behaves and operates in the real world.
• Discussion also relevant for development communication. Communicators are forever searching for evidence that communication works, and therefore that their jobs are worthwhile.
• What does the funder want to know: Whether their money is being used EFFICIENTLY? or EFFECTIVELY e.g. linked to the project/programme purpose? or the IMPACT it CONTRIBUTES (among many other external factors) to (e.g. the overall goal) ?
• What are the criteria for "effective" and "efficient" KM, and who decides what those criteria ought to be?
• A needs assessment is an important aspect of seeing through a chain of causalities and results. This would also point to the direction of our donors who, most often, attempt to fulfill a need based on sound principles of development practices and requirements.
• Questions to ask: why do I want to measure this? Is it for reporting purposes (we delivered so many sacks of rice), or to quantify success (and what is success or failure), or because you have to (governance issue), etc. How will you measure (and what metrics?) Sometimes we use metrics because the metric exists rather than identifying what it is we really want to measure and why. Lastly, how will the metrics be used?
• The OECD-DAC use the following five criteria to evaluate development effectiveness. Could we mainstream KM4dev into evaluations of aid effectiveness?:

1. Relevance - the extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.
2. Effectiveness - A measure of the extent to which an aid activity attains its objectives.
3. Efficiency - Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results.
4. Impact - The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators.
5. Sustainability - Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable

• What is maybe still missing is how we can (if at all) ATTRIBUTE things?

3) KM and behavioral change

• Do we have an influence with KM/KS: At least we learn from our experiences and use that learning to design and implement future activities in better ways.
• Never mind any causalities, as long as we can show that our organizations actually behave differently the KM investments are already justified
• Isn't this exactly the main challenge, i.e. how to measure these insights and this change? Because one cannot directly measure it in a monetary way?
• Knowing about things is NOT enough to change behaviors/ resource flows but it is an important element. Behavioral change comes in part from knowledge
• Include the following in project planning: Downsize plans and agree to what is doable.
• When you can really show a change in behavior, you still have to be able to link the change to the investment because you have to be able to attribute the change to your effort.
• Behavioral change does not result from knowledge (or KM) that is internalized as new insight. But conversely it triggers new thinking that might be consolidated with supportive information.
• Behavioral change to me is more a side effect of applying methods and tools we introduce to achieve business results. the impact or result we are really interested in, is the knowledge getting to where it is needed.
• The behavioral change I was talking about as an impact of KM was the second level systemic change. Second level change is worth considering as an impact. First level I consider isn't.
• “Cognitive dissonance” (someone correct me if I’m wrong). It is based on the human desire to be (and to be seen to be) internally consistent. In other words, Behaviour -> Attitude -> Knowledge (not, as is often assumed, the other way round)
• It is useful to reconsider behavioural change as impact of KM. KM can be massively good, useful, effective at changing IDEAS but it is very possible that the external environment makes it impossible to ACT differently. So what do we do then? Where does that leave ‘behaviour change as indicator of real KM impact? And if there is no behaviour change, does this mean that KM was bad? No, perhaps not. Perhaps the analysis to invest in KM was poorly done and KM was well implemented but based on the wrong theory of change.

4) “Knowledge is power”

• Francis Bacon was right then. Today we are wallowing in so much information and knowledge, it's become a major problem to get anything sorted out and actually do something about it. Knowledge is increasingly powerless to change things. What is REAL power in today’s world is the capacity and competence to achieve changes in how things happen
• In this era of knowledge/information overload, we need to promote KM as a crucial practice that can help organizations and people to find the right needle in a haystack of data/information/knowledge.
• Perhaps we can edit Bacon to say 'knowledge CAN be power'? Not by definition but conditional to criteria such as relevance, in the hands of those who need new insights, linguistically and conceptually accessible, etc.
• Knowledge can help to build power to act individually and power with others to act collectively.
• We should be humble about knowledge (and KM) but not necessarily throw it all in the trashcan
• Friar Bacon should have said "knowledge is corporate power". His words are too often interpreted as "knowledge is personal power". This false interpretation has generated the armies of control-freak managers that stifle progress rather than the enlightened leaders who make the workplace a great place to be.
• Knowledge as personal power versus knowledge as people power
• Knowledge management for capacity development: What is critical is to manage the interface of proprietary and non-proprietary knowledge. Eric von Hippel's Democratizing Innovation would be worth referencing.
• The discussions on KM and power is really central when we think of structures and processes that (dis)empower (end)users

5) Importance of Knowledge Sharing

• Share it or perish (analogue to the old academic pun "publish or perish). Share what we know or become irrelevant. Fail to share and get bypassed in the next round of sharing. KM? Is sharing an important criterion for impact?
• Sharing not so much as an impact of KM but a condition or quality indicator of KM. See Dave Snowden’s Cynefin framework and his passion about the notion of open source. www.cognitive-edge.com.
• For development actors, making what we know more open and more openly available and accessible, through Knowledge Sharing (KS), should surely be a core measure of an organization's contribution to the development commons? I find the notion that what we 'share' is at least as important (to us as well as to others) as what we 'own'. In development, it is probably much much more important, perhaps close to the reasons why most of our organisations exist.
• An individual who refuses to share his/her knowledge is therefore useless to the organization
• Maybe we should aim more for wisdom based on experience?
• What do we have to do as far as the general management of development organizations is concerned in order to make KM effective and sustainable?
• Sharing is what defines us, particularly in the development world but also in the commercial spheres. How would I judge the quality of sharing? By what criteria?