Revision as of 18:05, 26 September 2013

This initiative is now completed - however the results of this plan

A suggested M&E approach for KM4Dev

The IFAD-supported plan of activities for KM4Dev 2012-2013 included monitoring and evaluation (M&E), formulated as 'Learning and Monitoring' (L&M) to emphasise the importance of learning from the activities funded, for the community, IFAD and beyond. This page explains how the L&M plan was implemented. The objective of this 'L&M plan was to understand how each of the planned activities contributed to the well-being and effectiveness of the KM4Dev community through the lens of specific learning areas and issues in relation - and crucially to learn where KM4Dev could expand good practices and minimise negative results'.

Documenting the L&M work

In this section of the wiki, the L&M team tried keep everyone up-to-date with ongoing L&M activities mainly through an analysis of some of the key activities as well as a log of L&M activities that was following all that L&M work.

Background information

The KM4Dev Learning and Monitoring (L&M) group was coordinating the monitoring and learning around the activities planned and implemented as part of the IFAD-supported overall plan for 2012 activities. 

It had the responsibility to:

• Validate the proposed monitoring approach as a way to capture the learning of the key activities and monitor detailed project milestones;
• Review the performance against the agreed milestones;
• Ensure the learning is documented and shared through KM4Dev and within IFAD.

The group had to comply with these responsibilities in a participatory manner. 

The learning and monitoring (L&M) group consisted of: Bertha Camacho (SKAT), Ewen Le Borgne (ILRI), Simone Staiger (CIAT) Sophie Alvarez (CIAT) and Willem Bettink (IFAD), joined in November 2012 by Melissa Bator, Saskia Harmsen and Peter Bury. It was composed of IFAD and KM4Dev core group members.

In addition to this group, an advisory 'tiger team' (of up to 10 people selected in the wider KM4Dev mailing list on the basis of interest) was supposed to provide technical advice to the L&M Team regarding the L&M approach and specific activities as they developed along the way - from the L&M group or spontaneously to check how things are going in the L&M work.
Finally, a baseline 'focus group' (of up to 100 people self-selected in the wider KM4Dev mailing list on the basis of interest and completed by a random selection for each activity) was also supposed to assess the baseline of the community in the light of the learning areas highlighted below. 

In practice, however, the Tiger team and the baseline focus group did not materialise - mostly because KM4Dev members seemed to be overwhelmed by information coming from all members running activities such as L&M - and the idea of having these support groups was abandoned along the way, although this did not impede the L&M team to do its work

The plan that follows below suggested an approach to reach these objectives.

The levels and focus of activities of learning and monitoring

In this framework of the IFAD-supported activities, the group focuses on a) three distinct focus areas that interest us generally to understand how KM4Dev is working and leading to results b) three distinct groups of people, interlinked in many ways, which (might) play a role in how we assess the three focus areas and c) three distinct learning areas:

The three focus areas are:

1. The community itself: KM4Dev as a community of practice, the people that make it and how they connect with each other and coalesce around their collective sense of belonging to the community. This is a focus area as we need to understand how the community itself governs and develops itself.
2. The domain: i.e. the field of knowledge management for development and how that domain is related to the community and to the wider domain community (see below). This is important to understand how the CoP influences this domain and draws from it, as it keeps on changing. 
3. The wider domain community: the people that are in some way related to or concerned by the field of knowledge management for development but are not part of the KM4Dev community of practice, although they could potentially be very interested by what the latter does). This is an interesting focus area to understand how the work of the community on the domain affects others invested in that domain but not necessarily part of KM4Dev.

The three groups of people are:

1. The core group - whose role is to steer the community;
2. The wider KM4Dev community - all other members that contribute to making the community alive;
3. The wider world - which has connections with either core group members or KM4Dev community member and could indeed be influenced by what the former two groups do. This group relates to the wider domain community but could also go beyond.

The three learning areas that matter for the L&M group are:

1. The identity and governance of the KM4Dev community of practice. The identity is defined by all members together;
2. The governance is mostly taken care of by the core group but in close interaction with the wider KM4Dev community.
3. The activity of the community of practice: the level of interaction, discussion, action etc. that characterises KM4Dev as a community of practice. The (learning and) outcomes of KM4Dev: the outputs developed by the CoP but also and much more crucially the use of these outputs and other signals of influence that revolve around KM4Dev at either of the three levels and with/between either of the groups mentioned.

These learning areas form the essence of the L&M plan’s contents. The operationalisation of the plan takes into account the different levels and groups of people. These two aspects are developed in the next two sections.

Unpacking KM4Dev learning areas

The three learning areas above relate to several specific issues that matter to understand the effectiveness (or not) of KM4Dev as a community of practice. These issues are specified in the table below:

(1) codification of conversations into concrete outputs / items e.g. blog post, wiki entry etc.

Operationalising the Learning and Monitoring plan

The suggested approach to make this plan operational is the following:

• Develop a baseline survey on SurveyMonkey asking a test group (see below) to assess KM4Dev from 1 (very low/inexistent) to 5 (very high/fully achieved) on each of the issues above and add a couple of open questions about where KM4Dev has a great legacy/offers a great example that it should capitalize on and where it should improve. 
• After each activity from the IFAD-supported activity plan, submit a group of people to this questionnaire to understand to what extent the activity reached any of the learning area issues - adding an open box for comments under each issue; feed back these results to a) concerned organisers, b) core group and c) wider community (on the wiki). Feedback could thus come from:
  • The L&M team prompting the baseline focus group (on the survey) as mentioned above;
  • From spontaneous reactions and testimonies collected;
  • From reactions and testimonies sought from activity organizers (about three things that went well and three things that didn't go so well).
• Twice a year, chase stories of change (guidelines for these will have to be organised) from the wider KM4Dev community, either directly inspired from conversations on the mailing list or from any other opportunity;
• Twice a year take stock of the results (from activities, based on the assessment made with the focus group and with the activity organisers) and see where improvements need to take place to adjust as and when and document good practices.
• Take advantage of face-to-face conversations among L&M group members and core group members to check insights from this work through focus group discussions and interviews.
• Continually (at least once every two months) monitor quantitative indicators: amount of members / discussions /contributions / reification activities (writing blog posts, writing wiki summaries, journal articles etc.), etc.if easy to pull out)
• At the end of the project, a final full survey will be done with all baseline focus group members to collect a final set of data to integrate and draw lessons from for a final report which will be shared with everyone in the community (the focus and contents of that final report will be discussed at a later stage).
  

In practice, the baseline questionnaire was conducted (see report on top of the page), short interviews were held with the initiators of the activities but not all of it has been documented on this wiki. No feedback has been sought from other KM4Dev members (the beneficiaries of these different activities) to avoid overburdening them. The L&M group did have regular conversations for some time although after the completion of the baseline survey these conversations progressively stopped too.

The final activity involving L&M, which seems to replace the final survey, is to look at the results from the L&M survey, social network analysis and community technology lab for overlapping pointers and recommendations to drive KM4Dev forward.

Baseline survey May 2013

2013 Baseline Report of the KM4Dev Learning and Monitoring initiative

The baseline report from the L&M group File:KM4Dev Learning and Monitoring final baseline report.pdf

Geographical distribution of respondents in the 2013 Baseline Report

One of the questions addressed in the IFAD Synthesis project is how the different studies complement each other – what one study offers to the matters addressed in the other. Specifically it considers the assumption that email is more accessible in the South than more bandwidth-intensive platforms like Ning (or to a lesser extent, a wiki). With that in mind, the IP address in the survey was used to get a rough country identification of survey respondents. Because KM4Dev members could well have been traveling when they responded, this is just a rough snapshot, but it's interesting to note how spread-out respondents are. Here are the countries represented in the survey according to this rough-and-ready method: