KM4Dev Learning and Monitoring 2012-2013

This initiative is now completed - however the results of this plan

A suggested M&E approach for KM4Dev

The IFAD-supported plan of activities for KM4Dev 2012-2013 included monitoring and evaluation (M&E), formulated as 'Learning and Monitoring' (L&M) to emphasise the importance of learning from the activities funded, for the community, IFAD and beyond. This page explains how the L&M plan was implemented. The objective of this 'L&M plan was to understand how each of the planned activities contributed to the well-being and effectiveness of the KM4Dev community through the lens of specific learning areas and issues in relation - and crucially to learn where KM4Dev could expand good practices and minimise negative results'.

Documenting the L&M work

In this section of the wiki, the L&M team tried keep everyone up-to-date with ongoing L&M activities mainly through an analysis of some of the key activities as well as a log of L&M activities that was following all that L&M work.

Background information

The KM4Dev Learning and Monitoring (L&M) group was coordinating the monitoring and learning around the activities planned and implemented as part of the IFAD-supported overall plan for 2012 activities. 

It had the responsibility to:

• Validate the proposed monitoring approach as a way to capture the learning of the key activities and monitor detailed project milestones;
• Review the performance against the agreed milestones;
• Ensure the learning is documented and shared through KM4Dev and within IFAD.

The group had to comply with these responsibilities in a participatory manner. 

The learning and monitoring (L&M) group consisted of: Bertha Camacho (SKAT), Ewen Le Borgne (ILRI), Simone Staiger (CIAT) Sophie Alvarez (CIAT) and Willem Bettink (IFAD), joined in November 2012 by Melissa Bator, Saskia Harmsen and Peter Bury. It was composed of IFAD and KM4Dev core group members.

In addition to this group, an advisory 'tiger team' (of up to 10 people selected in the wider KM4Dev mailing list on the basis of interest) was supposed to provide technical advice to the L&M Team regarding the L&M approach and specific activities as they developed along the way - from the L&M group or spontaneously to check how things are going in the L&M work.
Finally, a baseline 'focus group' (of up to 100 people self-selected in the wider KM4Dev mailing list on the basis of interest and completed by a random selection for each activity) was also supposed to assess the baseline of the community in the light of the learning areas highlighted below. 

In practice, however, the Tiger team and the baseline focus group did not materialise - mostly because KM4Dev members seemed to be overwhelmed by information coming from all members running activities such as L&M - and the idea of having these support groups was abandoned along the way, although this did not impede the L&M team to do its work

The plan that follows below suggested an approach to reach these objectives.

The levels and focus of activities of learning and monitoring

In this framework of the IFAD-supported activities, the group focuses on a) three distinct focus areas that interest us generally to understand how KM4Dev is working and leading to results b) three distinct groups of people, interlinked in many ways, which (might) play a role in how we assess the three focus areas and c) three distinct learning areas:

The three focus areas are:

1. The community itself: KM4Dev as a community of practice, the people that make it and how they connect with each other and coalesce around their collective sense of belonging to the community. This is a focus area as we need to understand how the community itself governs and develops itself.
2. The domain: i.e. the field of knowledge management for development and how that domain is related to the community and to the wider domain community (see below). This is important to understand how the CoP influences this domain and draws from it, as it keeps on changing. 
3. The wider domain community: the people that are in some way related to or concerned by the field of knowledge management for development but are not part of the KM4Dev community of practice, although they could potentially be very interested by what the latter does). This is an interesting focus area to understand how the work of the community on the domain affects others invested in that domain but not necessarily part of KM4Dev.

The three groups of people are:

1. The core group - whose role is to steer the community;
2. The wider KM4Dev community - all other members that contribute to making the community alive;
3. The wider world - which has connections with either core group members or KM4Dev community member and could indeed be influenced by what the former two groups do. This group relates to the wider domain community but could also go beyond.

The three learning areas that matter for the L&M group are:

1. The identity and governance of the KM4Dev community of practice. The identity is defined by all members together;
2. The governance is mostly taken care of by the core group but in close interaction with the wider KM4Dev community.
3. The activity of the community of practice: the level of interaction, discussion, action etc. that characterises KM4Dev as a community of practice. The (learning and) outcomes of KM4Dev: the outputs developed by the CoP but also and much more crucially the use of these outputs and other signals of influence that revolve around KM4Dev at either of the three levels and with/between either of the groups mentioned.

These learning areas form the essence of the L&M plan’s contents. The operationalisation of the plan takes into account the different levels and groups of people. These two aspects are developed in the next two sections.

Unpacking KM4Dev learning areas

The three learning areas above relate to several specific issues that matter to understand the effectiveness (or not) of KM4Dev as a community of practice. These issues are specified in the table below:

Learning area	Specific issues in this area	Description
Governance	Transparency	Systematic sharing of and accessibility of results of conversations, decisions, initiatives, reification activities etc. also including the selection of core group members
	Vision, values and principles	Existence, clarity, understanding and acceptance of general vision, principles and values for the community of practice by and for its members
	Leadership	Demonstrated and particularly accepted leadership of the core group (and occasionally others) by other members of the KM4Dev community. Is there any dissonance between the two groups?
	Active leadership building	To what extent existing leadership seeks feedback from the community and incorporates it into activities and encourages new leaders to step in leadership.
	Mobilisation and commitment	See below
Identity and activity	Diversity and expansion	Profile of members of the community and the core group (language, region, type of organisation etc.); Growth and expansion (frequency of new members, how etc.) and ties with external networks
	Conversation	Frequency and quality of conversations around the domain or the community
	Reification (1)	Tendency (quality and frequency) of the community to ‘reify’ conversations into tangible outputs e.g. blog post, wiki entry, journal article etc. Also has a bearing on learning and outcomes
	Mobilisation and commitment	Capacity of the core group members and other KM4Dev members to mobilise themselves and commit to activities (which activities? to what extent/degree of involvement?) and indeed deliver according to the plan and with strong learning. This also has bearing on the governance
	Participation	Degree of participation of different members to conversations and other activities
	Reflection	Evidence of social learning, development and sharing of new insights as part of activities (and results – this has bearing on learning/outcomes)
	Cohesion	Evidence that the relationship between members of the community is good and that everyone finds their place in the community while feeling they are part of a whole
(Learning and) Outcomes	Reification / outputs	See above. Production of outputs (quality / frequency?) - planned or spontaneous
	Reflection / changed thinking and discourse	See above. Evidence that reflections from the KM4Dev community have achieved change in thinking and/or discourse among others e.g. citations, semantic analysis.
	Inspiration / changed behaviour	Evidence of change as a new way to proceed, inspired by KM4Dev activities
	Innovation / changed artefact or approach	Evidence of KM4Dev influencing development of a new artefact or method, codified concretely
	Impact	Evidence of larger changes (in autonomy and livelihood-related well-being) where KM4Dev activities have inspired/influenced others within community and particularly beyond. Caveat: attribution
Engagement channels	Suitability for participation	The different KM4Dev channels (mailing list, wiki, ning community, annual meetings) foster dialogue and engagement, and learning
	Ease of use / Availability of KM4Dev outputs	The different channels are easy to use and complement each other. They make KM4Dev activity outputs visible, and available. Identity Governance of Km4dev is clear in all engagement channels

(1) codification of conversations into concrete outputs / items e.g. blog post, wiki entry etc.

Operationalising the Learning and Monitoring plan

The suggested approach to make this plan operational is the following:

• Develop a baseline survey on SurveyMonkey asking a test group (see below) to assess KM4Dev from 1 (very low/inexistent) to 5 (very high/fully achieved) on each of the issues above and add a couple of open questions about where KM4Dev has a great legacy/offers a great example that it should capitalize on and where it should improve. 
• After each activity from the IFAD-supported activity plan, submit a group of people to this questionnaire to understand to what extent the activity reached any of the learning area issues - adding an open box for comments under each issue; feed back these results to a) concerned organisers, b) core group and c) wider community (on the wiki). Feedback could thus come from:
  • The L&M team prompting the baseline focus group (on the survey) as mentioned above;
  • From spontaneous reactions and testimonies collected;
  • From reactions and testimonies sought from activity organizers (about three things that went well and three things that didn't go so well).
• Twice a year, chase stories of change (guidelines for these will have to be organised) from the wider KM4Dev community, either directly inspired from conversations on the mailing list or from any other opportunity;
• Twice a year take stock of the results (from activities, based on the assessment made with the focus group and with the activity organisers) and see where improvements need to take place to adjust as and when and document good practices.
• Take advantage of face-to-face conversations among L&M group members and core group members to check insights from this work through focus group discussions and interviews.
• Continually (at least once every two months) monitor quantitative indicators: amount of members / discussions /contributions / reification activities (writing blog posts, writing wiki summaries, journal articles etc.), etc.if easy to pull out)
• At the end of the project, a final full survey will be done with all baseline focus group members to collect a final set of data to integrate and draw lessons from for a final report which will be shared with everyone in the community (the focus and contents of that final report will be discussed at a later stage).
  

In practice, the baseline questionnaire was conducted (see report on top of the page), short interviews were held with the initiators of the activities but not all of it has been documented on this wiki. No feedback has been sought from other KM4Dev members (the beneficiaries of these different activities) to avoid overburdening them. The L&M group did have regular conversations for some time although after the completion of the baseline survey these conversations progressively stopped too.

The final activity involving L&M, which seems to replace the final survey, is to look at the results from the L&M survey, social network analysis and community technology lab for overlapping pointers and recommendations to drive KM4Dev forward.

Baseline survey May 2013

2013 Baseline Report of the KM4Dev Learning and Monitoring initiative

• Retrieve the baseline report from the L&M group File:KM4Dev Learning and Monitoring final baseline report.pdf

Geographical distribution of respondents in the 2013 Baseline Report

One of the questions addressed in the IFAD Synthesis project is how the different studies complement each other – what one study offers to the matters addressed in the other. Specifically it considers the assumption that email is more accessible in the South than more bandwidth-intensive platforms like Ning (or to a lesser extent, a wiki). With that in mind, the IP address in the survey was used to get a rough country identification of survey respondents. Because KM4Dev members could well have been traveling when they responded, this is just a rough snapshot, but it's interesting to note how spread-out respondents are. Here are the countries represented in the survey according to this rough-and-ready method:

Distribution of respondent location based on the IP address collected by Survey Monkey.

Many responses per country:	United States: 29; United Kingdom: 17; Netherlands: 10; Switzerland: 7; Canada: 6; Ethiopia: 6; India: 5.
Four responses per country:	Colombia, Lithuania, and Uganda.
Three responses per country:	Brazil, Nepal, South Africa, and Spain.
Two responses per country:	Bangladesh, Belgium, Costa Rica, Ecuador, France, Germany, Kenya, Malaysia, Mexico, Nigeria, and Philippines.
One response per country:	Australia, Botswana, Burkina Faso, Chile, Denmark, Djibouti, Fiji, Indonesia, Jordan, Pakistan, Paraguay, Peru, Senegal, Trinidad and Tobago, Tunisia,
	Three responses had blank IP addresses so no country information is available.