Talk:Conference Impact analysis

See the original thread of this E-Discussion on D-Groups

Brad Hinton, 2010/2/23

I think there is a great opportunity to use the KM4dev network and the other professional networks of KM4dev members to alert us to upcoming KM events and conferences.

The challenge is how to organise this - is it by individual email alerts; the risks being cross-postings and an even greater email volumes, or a centralised web page that we can access and perhaps get alerts using RSS?

Or perhaps we could have a wiki page on KM4dev where members can upload conference alerts and the conferences could be organised in date order?

The other thing I'd like to see is to get a summary or some feedback from people who have attended a KM/development conference and *how it made a difference to the person when going back to the workplace*. To me, attending a conference is only part of the journey- what you do afterwards with the knowledge/learning/experience/contacts is just as important.

Whatever the case, it would be good for our distributed intelligence to be used for conference alerts and conference "impact statements" without adding to the workload of the KM4dev administrators!

Brad Hinton, 2010/2/24

As to the "conference impact analysis" idea, I really think this is something important. Many times people attend conferences and get all enthused about possibilities but then the reality of the workplace hits them. How can we take or adapt what we hear from good conferences into our workplace setting and what difference will it make. What are the practical steps and outcomes in our workplace as a consequence of attending a conference (can be an actual presentation or just a comment from a person in the corridor or over lunch).

The other benefit is the next time one asks to attend an expensive conference, a previous "conference impact analysis" report will provide the evidence as to value.

I'd be interested in hearing what other people think about this "conference impact analysis" idea - either here on the list, or directly.

Ewen Le Borgne, 2010/2/25

I agree strongly with you Brad, that it would be excellent if we could have a small report / series of impressions (doesn't have to be long - in fact better short) about the value and happenings of these events. The blog function on KM4DEV's Ning group is perfect for this (add content [top right hand corner], select blog post) although anyone is free to do it the way they want.

At a meta level, and this could perhaps be a topic for IKM-Emergent to look into, if we had those impressions about a number of events over time, we could probably draw out some patterns of discourse and practice that would give us an indication of where our field (KM for development) is moving onto and what are the trends, buzzes, fads and promises that animate our community - a crucial indication of where our collective efforts could focus on to encourage what we see as positive developments, to dim those we may not agree with and more than anything to discuss these issues among ourselves to unpack them further and realise how we can use this information.

Tarit Kumar Datta Gupta, 2010/02/27

Dear Colleagues

The discussions are generating encouraging interests among the Km4Dev members. Since my entry into the discussion, though I am occupied with preparing a document titled: New World Culture for the Future UN Project, and fortunately I succeeded today to submit the first part namely “Election Economics”, the topics of discussions so far are quite interesting. Brad Hinton in his mail to Lucie on last 26 February suggested the possibility of documenting “Conference Impact Analysis”. I would like to draw the attention of Johannes working with UNDP in this regard because I myself work with impact study of program or project and I follow the UNDP Hand Book on monitoring and evaluation.

To my understanding, the very word impact is highly technical and is difficult to measure of an activity only. The impact comes much more late. Moreover, it is not clear which conference Brad Hinton is talking about. As I understand (please make necessary corrections) impact is the objective of the project. One activity, here, organizing and holding conference by Km4DGroup can not be identified as a project. A number of activities will identity a project and will produce certain results at the end of the project, then after a certain period will lead to the objective of the project. In fact, objective is nothing but the way how the deliverables (output and outcome) will be used in future or why the project has been opted for, so the activities are implemented. Thus, the impact measurement also depends on the age, that is, how many years ago the project or the activities of the project were implemented. Moreover, the achievement of the project objective also depends on other projects implemented in the same locality or surrounding localities by the same agency or other agencies. Considering the above, the word “impact” shall not be used.

I hope more pragmatic phrase can be “conference analysis” or “conference report” or any other words instead of using the word impact. Please react so that it becomes learning your understanding.

Brad Hinton, 2010/02/28

Tarit,

Thanks for your comments. Let me articulate what I am trying to propose with respect to "conference impact analysis".

I really don't think there should be any problem with the phrase "conference impact analysis". The idea is to go beyond just writing a conference summary or a conference report which is only part of the process.

Impact is not a technical word as it is. There might be technical connotations within a certain context, albeit I find even the phrase "impact assessment of project work" to be more akin to a check list of outputs than whether any real difference was made for the recipients of a project. I therefore do not believe that the impact of a project is *just* the objective, since I believe that impact is something that actually happens and not something *only* to be wished for.

To me, impact is about change - making a difference. If nothing happens, is there any impact? Impact is not process in my opinion.

I am interested in what change, what difference, what resulted from, what impact attending a conference or a seminar had on one's actual job or responsibilities. It is too easy to write a summary of a conference, feel nice and enthusiastic about it, and then do nothing when going back to the workplace.

There are plenty of conferences and seminars available that we can attend. But what REALLY is the impact on our work as a consequence of attending these events?

Dave Snowden, 2010/02/28

My own view is that "impact" should be used to define measures that do not require a forecast or definition of outcome

Brad Hinton, 2010/02/28

Dave,

Yep - agree with that!

We need to convince the development aid sector though......

Nilanjana Bhaduri, 2010/03/01

Hello All,

Introducing myself, I am Nilanjana working for Bosch in India. I am few days old in this forum and have found the discussion threads quite stimulating.

Regarding the conference impact analysis, I feel that it might be tough to capture the true "impact" as it is purely a perception factor. A person attends a conference, makes some notes & a report for submission and then that report gets in management circulation. Some of the contents might get discussed and adapted to organization interest, but still the perception might remain that these thoughts have been culled from multiple other sources and not essentially traced back to this report. Hence, the "impact" gets diluted. Thus, a quantitative value capture of impact might be quite a challenge because of the extensive breadth of stakeholders involved in realization of the impact factor.

Looking forward to your thoughts & opinions,

Peter J. Bury, 2010/03/01

There is that 'impossible' word again: impact.

Back in the late 90's I was pleasantly surprised to see my colleagues Kathy Shordt and David Saunders struggling in IRC International Water and Sanitation Centre to 'get rid of impact'.

Why? Because trying to assess impact is in the overwhelming number of cases a total waste of time and money. Impact is about causing a real lasting change, and the problem is with measuring this.

1. measuring is mostly impossible because impact happens often later than you, in your typical funded project, are allowed to spend money to measure anything; 2. so many other factors influence the thing you try to have an impact on DURING your work; 3. so many other factors influence the thing you try to have an impact on AFTER your work; 4. so if anything, how do you attribute what change to whom's work?

So rather than wasting precious resources and pretending, why don't you focus on:

1. measuring and monitoring *efficiency* during and at the end of your work; 2. *effectiveness *preferably during your work and immediately afterwards (eventually outsourcing to your inheritors monitoring effect beyond your intervention).

A small example: improving health by hygiene education and introducing sanitation services:

1. efficiency: cost versus benefit or actual versus planned meeting your objectives in terms of number of people reached through hygiene education and cost of sanitation facilities in line or lower than typical local costs for equivalent; 2. effectiveness: % people actually USING the latrines; 3. impact: possibly a decrease in hygiene and sanitation related diseases, which could well have been influenced by better similar interventions in the same area by others (go and see in Makarere III, Kampal: the density of latrines / sanitary facilities funded by a plethora of different interventionists). Other factors that influenced: fill in yourself.

Does that make sense?

Peter J. Bury, 2010/03/01

Dave can you explain this better please, see what my understanding is of efficiency, effectiveness and impact monitoring.

Vu Bang Pham, 2010/03/01

Dear all,

To my understanding , as a monitoring & evaluation officer, Tarit mentioned a very professional impact chain analysis that all development projects use. The following graph can clearly summarize Tarit's point.

It takes time to evaluate the impact of projects as it appears to be a big work.

Brad, I also agree on your point that we need to see what resulted from attending conferences, workshops, etc. But it should be more logical to agree that these results are somewhere around the output level. Then, they will become outcomes only if we apply and put in use what we have learnt and been shared in conferences and workshops. Now, wait and see if there is any differences made accordingly, we are confident to say that attending to conferences and workshops create certain impact.

Looking forward to hearing and learning from you all.

image002.png

Md Santo, 2010/03/01

Therefore, according to Peter, effectiveness and efficiency come up as OUTPUT, and impact come up as OUTCOME. Does it make sense? And for those who’re suffering from “Love is blind until their eyes are opened” is the impact due to “do not require a forecast or definition of outcome” previously. Thanks to Dave for citing his consideration

Peter J. Bury, 2010/03/01

Md Santo

Effectiveness certainly doesn't come up as OUTPUT, if anything the use of latrines in the example is an OUTCOME.

Marc Steinlin, 2010/03/02

Dear all,

Interesting issue, to discuss the impact of conferences. There are now two threads in this discussion: a) what means "impact" and b) what do conferences serve for? Regarding the latter:

I feel that while conferences (of all sorts) continue booming, the question of value and benefit of all these events are in a bad way. How can we justify all these literally millions of development money going in countless conferences? What is the return for all this time, money and energy going into them? In many situations I experience the unquestioned assumption, that they in some (miraculous?) way will make an important difference.

Having specialised on designing and facilitating events of all sorts with the purpose of igniting change through (peer) learning and sharing (from smaller workshops, trainings and meetings up to large international conferences), we have found they all have a few things in common:

a) we indeed ask organisers, sponsors or initiators what impact their envisioned event should have. In general they are very quick in formulating the outputs they all sound pretty much the same: well, people should share their experiences, they should hear about what's new, they should network and and learn from each other; maybe we want a declaration or an action plan. But most importantly, we need a product like a conference reader, a report, a video, a website (possibly with some nice Web2.0 functionality) - that's for the sponsor. Our question then is: please let's forget all of that at this particular stage. Rather tell us: if six months or one year after the conference (workshop, ...) you look back over this period, what do you want to see? What do you want having happened *after* your conference so that you will conclude it was worth all the money and time (which usually is a lot if you sum up all the time and expenses of all the individual participants)? This is usually more difficult to answer (if the response should not remain in vague terms) and I had more than one instance where a client after reflecting about this question decided it was not worth it and dropped the idea altogether. But only when this question is answered, we can start reflecting about outputs that most likely serve this particular impact - and the report will be the least of those things (because in many cases it mainly serves not much more than the bureaucratic requirements of many institutions).

b) At this stage facilitation becomes the key issue. Facilitation in this understanding is not just about implementing an agenda of an event, welcoming participants, introducing speakers, ensuring time management, summarising statements, giving people the floor in a discussion and in the end saying goodbye to everybody. Facilitation in this context means first and foremost to have an understanding how human beings interact, how processes can be shaped and platforms be structured so that people - on the basis of learning and sharing - are likely to transform (to shift a little but significant bit, become a slightly different person, if this is not too pathetic) and to do something different afterwards. You can only lead horses to the river, but they must drink themselves. Therefore facilitation becomes the art of hosting, knowing the path and the terrain, knowing the quality of the water, creating an environment and way that is inviting and encouraging for people to engage and commit, to develop ownership and accomplish genuine learning.

Needless to say we're not talking about the wretched sequences of PowerPoint presentations and talking heads (with some pseudo breakout groups or plenary Q&As) that make up many of the conferences - needless to say these do not provide an occasion for people to take something from it and have a real impact (to use the word again).

c) The key to conferences with impact lies in three elements: I. defining the purpose well ahead (in the sense described above) - and in many instances they don't have a real purpose, they are just done because it's in some project document or in an annual plan or because it has always been done like that. II. Creating a design, a structure, a process that leads from the beginning to the end (which may sound obvious but is not) - an architecture (or a "choreography" to use the metaphor from dance theatre) that picks up participants where they are, leads them through a logical process of engaging, learning, concluding and committing and "dismissing" them with something different, yet significant, like some specific idea of what to do, some clarity, some useful connections to pursue, some immediate action to go about. There are some simple yet powerful ways and design principles to do so. III. Use relevant and effective methods of convening, conversing and engaging - and there are countless out there (starting with simple things like World Cafés, ... and ending with much more complex approaches like Future Search Conferences or Open Space). The KM4Dev list has been full of them...

By the way I feel that many of these new forms of dialogue and sharing have a clear edge in dealing with complexity in the sense Dave Snowden has mentioned earlier in this thread.

Brad Hinton, 2010/03/02

Marc,

Thanks for this. You have explained what I was driving at with my thinking on "conference impact analysis" far better than I. In particular, you say "What is the return for all this time, money and energy going into them? In many situations I experience the unquestioned assumption, that they in some (miraculous?) way will make an important difference". In other words, what is the "difference" attending a conference makes - or what is the impact at the workplace from your conference attendance?

This is the crux of my discussion point around conference impact analysis.

How can we extend what we learn, take up, consider, wish to apply from a conference and use this in real ways when we return to the working environment? By acting and reflecting on what we do with our conference experience back at the workplace. I am suggesting some form of conference impact analysis (and it doesn't have to be some mammoth formal document inside some development conference framework). We need to find where the real value of the conference has occurred, beyond just the personal interest of conference attendances.

As Marc goes on to say, perhaps we need to be considering what impact a conference has had on us six to twelve months after the event. It is in this way of thinking that I have proposed conference impact analysis.

Caitlin Bentley, 2010/03/02

Hi All,

I think the difference between the two viewpoints is that Brad is talking about individuals and others are approaching it from the event angle, as if it were the conference organisers that were measuring the impact of their conference in its entirety.

If we're talking about individuals, I would say it's just learning that happens.

Learning could be transformed into workplace impact depending on the type of person and organisational management structure. But how is this a result of the conference?

Unfortunately, conferences I don't find are very educational in terms of instructional or educational design, which is why we find it hard to retain and apply learning. Generally we attend presentations at conferences. Depending on the presenter, presentations can be fairly cognitive (i.e. present an outline to prime attendees, organise information into chunks, use interesting examples, use images and text for double coding, etc.). Cognitive learning depends on the extent that information is stored into long term memory, which takes attention, prior knowledge, practice, etc. Even workshops that could ideally be designed better using constructivist approaches, generally aren't.

If we wanted conferences to be educational for individuals so that they can learn and transform their workplaces then we would need to consider the format of conferences.

In my experience, conferences have been more useful for networking and filtering knowledge. It remains my own responsibility to mark down the studies or ideas that I found pertinent to then go back later and read more about it to actually learn about it to use it in my work.

All this to say that I think impact is maybe not the perfect word because it's not the conference that does this it's the motivation of the individual or the mandate of the organisation. Semantics.. Perhaps conference usefulness analysis or something LOL.

Matt Moore, 2010/03/02

Caitlin,

I have to agree that from an instructional design / learning theory perspective, most conferences are woeful.

The NSW KM Forum did a session on the Future of Conferences about a year ago: http://nswkmforum.wordpress.com/2009/05/04/future-of-conferences-redux/

We're certainly seeing a move from presentation-based conferences to more immersive experiences and other formats: World Cafe, BarCamp, Open Space, Ignite/PechaKucha. Most of the innovation is not coming from traditional commercial conference organizers.

Also at the NSW KM Forum, we ran a Most Significant Change activity early last month around the impact of the Forum's events for knowledge managers in Sydney. To be honest I'm not sure the "impact" of any one event has been huge. I see it as more of a cumulative thing.

Tony Pryor, 2010/03/02

This discussion on conference impacts is really very exciting (not that the words “exciting” and “conference” are often found next to each other…). The word “conference” can cover all sorts of things; events where the annual social gathering of like minded individuals is core (annual conventions of dentists,. etc., etc), one where the presentation of research papers is key, one where the actual exhibit hall is as important if not more important than any workgroup or plenary, and of course conferences that are designed to get a result (such as those built around a treaty negotiation, or one where sales people gather to get trained on the latest products).

The problem often occurs because the actual impact, purpose and objective of the conference is not very clear (“we always have one this time of year”), and given the size of such events it’s often just too hard to target the energy around specific “deliverables”. Second, results of Conferences are often static (meeting reports, summaries and presentations,. etc). rather than being organic steps along a much longer continuum, with work going on before, during and after the event, including (gasp) the assignment of responsibilities for following through afterwards.

Online tools per se may or may not address these constraints, although they do permit the more active participation of folk before and after the event, plus it permits expanding the pool of participants to include virtual participation (although this is not as easy as it appears).

I am more taken by the use of KM tools face to face, such as open space and the like. They certainly can capture the enthusiasm of the participants; there is nothing like a loooong series of lectures to put the participant into a stupor. But more important, a lecture-laden approach often equates the participant with as a knowledge consumer wanting to be told things, rather than a peer coming to the meeting with ideas equal to, and in some case better than, those of the presenters. But there is nothing that can beat actually thinking through goals and objectives ahead of time, and defining impact before the conference is planned. In fact, if done properly you MAY decide that a conference is inappropriate.

Marc Steinlin, 2010/03/02

Caitlin, interesting point, the distinction between usefulness (to avoid the word impact) on the level of individual participants versus the level of the (generally institutional) organisers.

I think to some extent this may also apply to many trainings. Development organisations hold trainings and pay for those. Participants are expected to learn something and subsequently to change something (ie. to apply what they learned). But still there must be an interest and an expectation on the side of the organisers, otherwise they would not organise/ pay for it. Yet the big question is whether the participants actually do "transform" in some way or another during those trainings. Sadly enough, the worst case in the development sector is when certain people mainly attend because then the receive the daily allowances... Of course it's an entirely different story when people pay a training from their own pocket. I just think there are a few parallels in terms of assessing impact.

In conferences we have designed recently, we try to actually harness the potential of participants to to jointly achieve something. It also helps aligning the two levels, the interest of the individual with the expectation and need of the organiser. This can have different consequences:

• on the individual level, the learning may be deeper, because the individual participant is actively involved in the "construction" of the knowledge; moreover this leads to genuine ownership over the results of the conference and thus to commitment to follow-up and apply insights.
• on the organiser's level an outcome beyond the mere report can be achieved - the conference can really create something with a certain degree of novelty.

As an example I attach an annotated agenda of a conference that we organise for the "Donor Committee for Enterprise Development" DCED (a coalition of donors like UNDP, ILO, GTZ, SDC, DANIDA, SIDA, World Bank and many more). Not to advertise to conference but to illustrate concretely how this may look in reality. In the past, the DCED held three conventional international conferences (presentation of papers and cases, plenaries, panels, break-out groups), and they felt the urge to do something different now that has more impact; the "old" model became unsatisfactory and they felt the "return of investment" was not good enough.

The current programme follows the design pattern of "divergence-emergence-convergence". On the three main days of the conference, the first day serves the actual learning and sharing of experiences (using methods like World Café, Fish Bowls, Speed Geeking). Day two is about "creating", ie. all the participants are actively involved in a process of jointly elaborate recommendations on the basis of the manifold experiences - a consolidation and extraction process; ideally we will achieve a kind of consensus on what to concretely do; since everybody is actively working on it, it also leaves a sense of achievement with every single participant. Day three is about implementing: in Open Space, all the participants jointly explore what the recommendations from day two mean in practice and what each of them can contribute concretely.

For illustration purposes, you can also watch the following video clip of a similar conference: http://i-p-k.co.za/wordpress/2009/08/03/making-conferences-attractive-through-a-mix-of-methods/

For our training courses in facilitation, we have also written our own handbook on " Knowledge Sharing for Change: Designing and Facilitating Learning Processes with a Transformative Impact". It's the first edition, we need to have a professional (native English-speaking) editor to work on it and to layout it. But there is an early version, which you can have if you like. You can download a free copy under: http://i-p-k.co.za/wordpress/resources-downloads/

CapeTownPreliminaryProgrammeFeb2010.pdf

Tarit Kumar Datta Gupta, 2010/3/2

Dear Brad and my other colleagues

It has become really interesting to see the encouraging discussion gradually increasing concerning the word “Impact”. To me the differences in opinion are the creative tensions instead of being the negative attitudes and the differences in opinion shall be unearthed and addressed to reach a common consensus. This is one form of knowledge management.

Any way, so far up to 2 PM of March 02, 2010 the responses that I have received are more or less close to my understanding, peter is more transparent, more clear understanding I see from Vu Bang Pham working for IFAD in Vietnam. I have come again for a number of reasons. First: Brad second comment in response to Dave “we need to convince the development aid sector though … “ frustrates me perhaps he could have rephrased to express his discontent with the development aid sector. Second: as I also see the confusion in Dave between output and outcome. And Third, I expected: Johannes working with UNDP (I repeat) and Lucie who is working with Km4Dev Secretariat (as I understand), Tammie Alzona who is silent for a long time and others will give their valuable comments about the use of the word Impact, not in this case only, but in general.

Let me please begin with what is the system model of an organization. Then on questions of Impact analysis I would like to quote UNDP Handbook on Monitoring and Evaluation which is lying in front of me while I was preparing my response to my colleagues. This handbook is the latest UNDP Book on M&E published in 2002 from the Evaluation Office, United Nations Development Programme, One United nations Plaza, New York, NY 10017, USA. I would humbly request all that this book can easily be downloaded from UNDP website and is highly rich and useful for any developmental practitioner. Then I would also like to quote some of my thirty years professional experiences directly and indirectly linked to Impact Study.

We need to keep in mind that what we are discussing we are discussing for the Future UN Project, that is, for the future UN Development Management System. As such, we shall not deviate ourselves from what have been practiced by UNDP for long, the latest development in this regard is 2002 because Impact is clearly imbibed in Monitoring and Evaluation. This further suggests that we shall be careful in wording our understanding about the subject concerned because UNDP or Future UN Development Management System will not accept whatever we think. We need to write those things which are logical. If my deliberations appear like counseling, I seek apology to Brad and my other colleagues of Km4Dev.

Brad, let us look at the system model of an organization. An organization is divided into three compartments: (1) people, (2) task, and (3) environment with small group of people as the nucleus working in the form of a team within an organization. The people consume resources from the environment (both internal and external), produces goods and/or services through accomplishing certain tasks, and delivers the same products and/or services to the same environment (internal and external). And to produce the products and/or services an organization adopts programs and projects. The system model of an organization further tells that an organization has vision, mission, strategy, programs/projects, and legal status sequentially to read reverse way (that is, from legal status toward opposite direction). (Ref.: MIM Handbook on Management, Manitoba Institute of Management).

Now let us see what is a project or program. When we say “project office or program office” we see the same picture of an organization. That is: people, task, and environment doing the same thing what has been suggested.

What I have written vision, mission of an organization, in the old days people used to write organizational goal and objective. Then what are the valiant aspects of a project or program: these are program/project activities, results, objective and goal. Brad, see how similar is the behavior between an organization and the program/project.

Brad, now I would like to draw your attention toward a different direction to understand me more clearly. That is logical framework popularly known as LFA or Project Planning Matrix (PPM). So far my knowledge goes, the LFA or PPM has been discovered by the American Corporate Sector, and since then it has been used by many development agencies, particularly the Scandinavian Countries whether private corporate or state developmental agencies. Germany has brought certain adaptation but using the same structure because it is well established hypothesis and we can not change the structure. In Bangladesh, the GtZ – state development agency has the mandatory conditions to use LFA or PPM from either the government or the private agency for any financial assistance. I am working as a certified moderator on LFA. It has got two parts: (1) situation analysis and (2) deriving PPM from situation analysis. The situation analysis that deals with cause-effect and means-end relations begins with problem tree, objective tree and alternate tree. Once the tree is selected from alternate analysis the tree is being placed in the LFA/PPM chart. The LFA/PPM is a four by four planning matrix, that is, 16 squares. The lowest row is the activities, the second row is the result (core means), the third row is the objective and the fourth row goal level statement. It is read from bottom to top. Brad, it is not possible for me to discuss LFA/PPM in detail but now you understand while the crux of the word “impact”.

Brad, Dave, Peter, Bang and my other colleagues, earlier we used to talk about measuring result but not enough longtime ago development practitioners identified that result is also difficult to measure at the end of the project, as such the practitioners reached to such an unanimous understanding that result shall be divided into two parts: output and outcome to facilitate measuring the achievement of the progress. To note that in the past, it was objective oriented project management but now-a-days almost all the development agencies (for Brad it is not all necessary to segregate development aid sector, non-aid sector etc.. because we are tired with the use of all these jargons which simply creates confusion instead of simplifying the characteristics of the agency) follow result based management (RBM) and UNDP is the Pioneer of RBM. UNDP defines RBM (quoting): RBM is a management strategy or approach by which an organization ensures that its processes, products and services contribute to the achievement of clearly stated results. Results-based management provides a coherent framework from strategic planning and management by improving learning and accountability. …” (Ref. Chapter 2, UNDP Handbook on Monitoring and Evaluating for Results).

Brad in the same Handbook UNDP clearly clarifies its understanding about result. It has given result chain: (1) inputs, (2) Outputs, (3) outcomes and (4) impact. The inputs are nothing but the activities. But UNDP again clearly articulates result: “overall purpose of monitoring and evaluation is the measurement and assessment of performances in order to more effectively manage the outcomes and outputs known as the development results”. (Ref.: A. Purposes of Monitoring and Evaluation; Chapter 1). UNDP further defines outputs as the specific products and services outcomes are the changes in development conditions (Ref. Ibid). And then where lies the impact, obviously it is the objective level statement the reasons that the program/project has been opted for or how the deliverable (result) will be used in future by the same agency or other agency. There is another important reason why the objective is the impact of the project. Brad, if you look into the LFA the most important column is the fourth column, the way it is read, which states the assumptions at different levels of PPM. Assumptions are the positive statements of a negative condition that must exist in the project location but beyond the control of the project holder. For example: government policies remain favorable to the private sector. And the assumptions are always cross-linked, as such no assumption is written at Goal because then it is necessary to write super goal statement which means that we will write five rows instead of four rows, as such does not fall into the category of LFA. To note that, as the LFA/PPM is derived from the objective tree of situation analysis, objective is also one form of deliverable but after a long time.

For peter, I agree with. Latrine is the product (output) being produced, and the use of latrine is the changes in development conditions (outcome). What about the impact. The impact is the reduced water or air borne diseases which will not occur only with the use of latrine but there will be some other activities and/or project to be implemented to reduce the water or air borne diseases.

I had the opportunity to conduct an impact study of the rural growth centers after eight years of developing the growth centers with the financial assistance by Swedish Government. The objective of developing the growth centers was to rejuvenate the rural economy at particular location in Bangladesh. On the other hand I had the opportunity to learn in 2000 from one of my friends working with Plan International Bangladesh that a mission from Denmark which was visiting Bangladesh to conduct impact study of a project was searching my friend after 18 years of a project of Danida that he worked with to take his interview.

Now Brad, Dave, Peter, Bang, Johannes, Tammie and my other colleagues I think now I am clear what I wrote in my first discussion about impact.

With warm greetings

Thanking you and with very best regards

Sincerely yours

Tarit Kumar Datta Gupta Development Specialist Dhaka, Bangladesh

Dave Snowden, 2010/3/2

Your detailed response deserves more than a quick reply! However I wanted to pick up and isolate one point.

The linear concept of input, leading to outputs, leading to outcomes which in turn leads to impact is I think at the heart of the problem, It implies (and I can see why people would want this) a causal chain that can be replicated.

However if the system is complex (in the sense of complex adaptive) then any input is a stimulus or modulator which influences but does not determine impact. That means we need to start measuring the sensitivity of a system to different stimuli, and the way in which some stimuli produce a disproportionate effect in that they catalyze other inputs. This is newly developing area which has not hit the development sector yet, but we are working on it in related fields, loosely termed modulator mapping. It also leads us to evolutionary representations (such as fitness landscapes) and measure based on stability of landscapes. In all those cases mathematics are simplified by representation and linked micro-narratives. There is no point in measuring anything if the results do not convince both donors and recipients alike to take action

All of that moves the "impact" agenda on. I didn't confuse outputs and outcomes, I conflated them as the model means there is no real difference in what is measured in practice.

Dave Snowden Founder & Chief Scientific Officer Cognitive Edge Pte Ltd

Peter J. Bury, 2010/03/03

Dear Brad and Marc

Interesting, but sorry I'm not convinced. I'm 56 going 57 soon and have attended and facilitated workshops and similar countless times since at least 1983.

If I'm honest I cannot say what the* measurable* impact is of any of those. At times I realize only years later what the one or the other event has brought me, made me change things and hence having an influence on what I do. Is that so different to YOU, dear reader of this message?

That certain events have had a specific effect on me, yes that I can remember much more vividly. But effect is not yet impact, in my understanding of these terms.

Curious to keep reading reactions to this conversation.

Peter J. Bury, 2010/03/03

Conferences?

The question is how people use the term. My suspicion is that many people call events conferences without really thinking about what a conference is in terms of objectives, strategy, methods, programme, roles of all attending.

Earlier I replied a bit quickly to this conversation on conference impact, thinking about them in terms of workshops. I guess most conferences are not meant to be workshops. But then hey, I'd have to say what I understand by 'workshop' ;-)

Peter J. Bury, 2010/03/03

Tony

you write (quote) The problem often occurs because the actual impact, purpose and objective of the conference is not very clear (“we always have one this time of year”), and given the size of such events it’s often just too hard to target the energy around specific “deliverables”.(unquote)

Isn't this discussion about conference impact analysis?

How would one know in advance what the impact is of a conference? May be i'm completely wrong, but I don't think there is any way to know in advance what the impact of a conference will or even could be!

Maybe a (measurable) outcome or (un)concrete output (as in deliverables - weird word) yes, but an impact???

Ian Thorpe, 2010/03/03

Hi all

I always find it interesting to see how a small suggestion by KM4DEV participant branches out into so many broad and stimulating discussion topics.

I think original suggestion by Brad that participants events also share something about how they benefited from the event is a good one not to be lost in this discussion. At the same time I think it's important to recognize that this experience as a participant in a conference or a training is highly personal and subjective. I've often heard widely differing opinions about the value of the same meeting coming from different participants - probably based both on their expectations and their pre-existing opinions and knowledge of the topic being discussed or the people discussing it. I also know that I've attended some workshops which I felt were lacking in clear purpose and poorly facilitated where nevertheless I gained something useful, often not what I was expecting or looking for. Many of the incidental gains are related to the connections you make and the side conversations you have, or the personal insights you gain from seeing what doesn't work.

I think Brad's suggestion of having participants write something about a meeting they attended which goes beyond content and looks to what was the effect of the meeting on their thinking or action is a great way to share insights about the learning process as a kind of "storytelling" device, and it is something that I intend to do more often, ad I hope others will take it up. At the same time it's not a fair way to assess the usefulness of a meeting as a whole, or to judge whether next year's conference will be worth attending. It's just a bit of personal flavour that you might find helpful, or not.

Paul Mundy, 2010/03/03

Conference evaluation forms can be very long and tedious to fill in. What did you think of the facilitation on the first day on a scale from 1 to 7? The second day? How about the accommodation? The food? The discussions?

In an attempt to be nice, most participants give a score of 6 or 7 to everything - except maybe the food.

To evaluate conferences (or workshops, or training courses for that matter), I use a very simple form. I get participants to take a blank A4 sheet of paper, and draw a horizontal line to divide it into halves. Then they draw a vertical line to divide the top half into two. Draw a smiley face in the top left quadrant, and a frowny face in the top right. Like this:

Under the smiley face, participants write what they liked. Under the frowny face, they write what they didn't like. On the bottom half of the sheet, they put any other comments. They then hand the sheet in to a neutral person.

This is a lot easier to prepare and complete than a standardized evaluation form, and gives the organizers all the information they need. By forcing participants to think of something to put under the frowny face, you learn what really were the problems.

To measure immediate impact, you can adapt this by having people write what they learned, or what they will do differently, in the bottom half of the sheet.

Arthur Shelley, 2010/03/03

Marc,

Thank you for sharing your resources. I greatly enjoyed browsing your website and am amazed at how different people with different backgrounds end up doing similar things for others in different places/situations. I am currently assisting an indigenous group here is Australia to set up a network of community mentors who can work within their own community to enhance relationships and develop a path towards a greater wellbeing for their people. This is very personally rewarding and enjoyable for us all.

I am now looking to see which of your concepts may be useful to use with these mentors to help them along their journey. As I believe in reciprocity and sharing, you and other forum members may be interested to see the free on-line profiler I have already successfully used with them at www.organizationalzoo.com <http://www.organizationalzoo.com/> This tool uses animal metaphors to enable people to self assess and reflect on their own (and others) behaviours and the impact these have on their actions and outcomes. It is simple and fun and leads to some rich “Conversations that Matter” (see:http://organizationalzoo.blogspot.com/2010/01/conversations-that-matter.html).

Good luck with these, I will keep you in touch with the outcomes of the facilitations using some of your methods.

Brad Hinton, 2010/03/03

Peter,

Perhaps I have "over-formalised" what I am driving at with respect to "conference impact analysis". I don't necessarily need a "measure" as such, although I am sure there are people higher up the organisational chain where such mesaures might prove useful.

I am really just trying to get people to think about what they do back at the workplace as a consequence of attending a seminar or conference. The "consequence"(is that a better term than impact?) of attending a conference or seminar back at the workplace is.....what? Better informed is the usual response but informed to do what? I am really driving the point that at some point after the seminar or conference a positive *action* takes place; an

• action* that might not otherwise have taken place had the conference or seminar been igniored.

I am well aware that such a post-conference "practice" or "reflection" or "analysis" may not give a *causal relationship* to what later transpires, but if we can identify something that *happens" as a consequence of conference attendance then perhaps we are able to recognise some *benefit* that *translates* into an actual worklplace action. Here, I am assuming that the puropose of attending a conference or seminar is to actually "take action" afterwards, but I acknowledge this is just an assumption on my behalf. I am sure there are conferences and seminars we go to just for the lunch!

As an aside, due to the imperfections of written language in a listserv environment, it would be great to workshop something like this to see how we can develop a more cogent approach and assessment of attending conferences and seminars beyond the feel-good factor. If anyone is really interested in this topic and wants to workshop the issue (and is located in Sydney, Melbourne or Canberra) please email me off-list.

Patrick Lambe, 2010/03/04

First, there are, surely, different levels of impact/effect/value for a conference.

There's the effect on an individual attendee, which is usually a combination of sub events - workshops, presentations, conversations, experiences at the conference, some of them designed, some of them emergent. There are different kinds of valuable effect too - whether it's doing something differently because of something learnt, an expanded network of contacts, a tool picked up and used, greater confidence in tacking a subject, better appreciation of an area. Some are more discernible than others.

Into this equation comes the degree of effort by an attendee to exploit the conference ("impact" is often a two way street, which is one of the several reasons why it's difficult to get to grips with it in a systematised way).

Then there's slow effect and immediate effect, which Peter hinted at. What happens next week? Do themes and learning and experience and contacts resurface years later in different contexts?

Then for regular events there's collective/cumulative-level effect - in Singapore we have an annual KM practitioners' conference which now functions as an important part of the rhythm of events in the KM community, and this value is over and above the value of individual conferences, ie it has gradually built over a number of years.

Second, I think we are constantly making the mistake of reducing a number of activities into the one concept of measuring. Measuring is only one way to gauge value, most commonly against established standards, or as a way of monitoring changes in a system. There are other evaluation and accounting (in the sense of "accounting for value") techniques for emergent and unpredicted/able value. MSC is one of them.

I have a great deal of sympathy with Peter's position in that we are constantly trying to over simplify the evaluation of complex phenomena, and using "impact" as a piece of jargon to cover up a multitude of ambiguities. I know why we do it, it's a design and communication challenge to communicate effectively with stakeholders; simplicity helps. However, we are so eager for simplicity that we cease to respect or reflect the complexity of the phenomena we're dealing with. I don't go so far as to abandon the term "impact", though; I think we just have to get out what we're measuring/monitoring/evaluating, why, and how. We need more precision in our language.

Shalini Kala, 2010/03/04

We do a similar thing - ask participants to write/share about "what worked" and "what could have been improved". It gives us sufficient information to make future improvements.

Ian Thorpe, 2010/03/04

Hi all

I always find it interesting to see how a small suggestion by KM4DEV participant branches out into so many broad and stimulating discussion topics.

I think original suggestion by Brad that participants events also share something about how they benefited from the event is a good one not to be lost in this discussion. At the same time I think it's important to recognize that this experience as a participant in a conference or a training is highly personal and subjective. I've often heard widely differing opinions about the value of the same meeting coming from different participants - probably based both on their expectations and their pre-existing opinions and knowledge of the topic being discussed or the people discussing it. I also know that I've attended some workshops which I felt were lacking in clear purpose and poorly facilitated where nevertheless I gained something useful, often not what I was expecting or looking for. Many of the incidental gains are related to the connections you make and the side conversations you have, or the personal insights you gain from seeing what doesn't work.

I think Brad's suggestion of having participants write something about a meeting they attended which goes beyond content and looks to what was the effect of the meeting on their thinking or action is a great way to share insights about the learning process as a kind of "storytelling" device, and it is something that I intend to do more often, ad I hope others will take it up. At the same time it's not a fair way to assess the usefulness of a meeting as a whole, or to judge whether next year's conference will be worth attending. It's just a bit of personal flavour that you might find helpful, or not.

David Gurteen, 2010/03/04

Many of you will be familiar with the Knowledge Cafe process and a version I have developed that I have dubbed the Gurteen Knowledge Cafe. In the early days my process went like this:

+ speaker + question + small group conversations + whole group conversation

Quite early on I modified the process..

1. where possible I asked people to form a circle with their chairs for the final whole group conversation

2. I added a fourth step at the end. I now go around the circle and ask each person to share with the group just one thing they plan to do when they get back to the office that is a result of what they have learnt in the Kcafe. I tell them if they don't have an action then to share a key learning. And to take any pressure off them I also say it is OK to pass though few people do that.

I am not too concerned whether they follow up on the action or not. I tell them about the "round robin" before we start so they think about actions they might take throughout the conversations. I think creating just that attitude of mind is key.

I am also experimenting with other variations of the Kcafe process where subsequent meetings to the initial KCafe are used to follow up on people's actions.

Here is one of my favorite quotes :-)

"The great aim of education is not knowledge but action." Herbert Spencer http://www.gurteen.com/gurteen/gurteen.nsf/id/X000193DE/

I hope this adds a little to what is turning out to be a fascinating discussion.

Gurteen Knowledge United Kingdom

Julia Cleaver, 2010/03/04

Greetings,

I would like to ask a question of the group.

Currently our organization does not have an executive level person who is responsible for KM type of cross-organizational priority setting and policy making. Until we have a person to do that we think that it would be helpful to have a committee that we could use to help us push our KM agenda. Actually, lately it has been that we have done such a good job of promoting the value of our work to the organization that we need help to prioritize all of the new work we are being given (without any new staff to help do the work of course). We also need to have strong support for developing and rolling out KM related policies.

Does your organization have a KM Steering or Advisory committee? If you do, what is the mandate and a typical agenda of this group? Do you set KM policy, priorities for activities? Are you the head or facilitator of the group? Do you have representation from across the agency. Do you have a member from your executive team?

If you can suggest any good models for a committee of this sort I would really appreciate seeing them.

Ewen Le Borgne, 2010/03/05

Hello all,

I've also been enjoying this conversation *very much* and agree with a number of things:

• Yes conferences should plan ahead what they envisage as a success or a positive outcome and it is indeed a shame that so much money is spent on

what could be labelled 'conference tourism' if the point is just to cash in on per diems or travel to have fun, rather than take sthg useful out of the workhsop.

• Yes that appreciation at personal level is very subjective and may/will not be the same to each and all; and yes sometimes we may not appreciate

just yet how much an event is valuable to us, although as Peter suggested we can already 'feel' when we are in a ground-breaking or stimulating event.

• Yet there is indeed a case for reporting about the event and what seen as (subjectively assessed) as useful points, using narratives as a way to link

embodied (personal and practical) and abstract symbolic knowledge (see this post by D. Snowden: http://www.cognitive-edge.com/blogs/dave/2010/02/narrative_as_mediator.php#moreabout this).

• Yes the facilitation is definitely key and it should, also in my view, help participants find and develop their learning space by co-designing or co-implementing the conference/workshop.
• The evaluation form that Paul suggested and the evaluation method that David are promoting seem like useful ways to prime our minds towards the

learning we expect getting out of the event. I very much agree with David that a first step in the right direction is to stimulate that attitude shift towards the 'what's the learning for me, how can I (share some of it) and apply it? By the way, if you haven't yet read this great article about the 2-5-1 evaluation method that Robert Swanwick blogged about check it out: http://swanthinks.wordpress.com/2009/11/23/2-5-1-storytelling/. It's an alternative to after action review (AAR), is easy to remember and very meaningful. I'm afraid it's not useful for large conferences though as it takes time.

• And anyway yes even with a badly designed event there will be valuable learning (and conversely a well designed event may still leave some hungry

for more).

Would anyone on the list volunteer to summarise this chat? It's a very worthwhile investment: for you as an individual to structure the learning and thoughts in the way a blog post would do, and for others on this list and beyond to have a clear entry on this valuable topic. I can support anyone volunteering to take this on. You just need a wiki account, go to http://wiki.km4dev.org/wiki/index.php/Community_Knowledge and add a new entry using (or adapting or ignoring) the template (http://wiki.km4dev.org/wiki/index.php/Community_Knowledge_Template). There is already a skeleton for this entry that was started and never completed: http://wiki.km4dev.org/wiki/index.php/Making_Conferences_Valuable

Thank you all for your fantastic insights, week in week out KM4DEV proves its value to me thanks to these discussions, a real fruit of our collective intelligence.

Joitske Hulsebosch, 2010/03/05

Hi,

Haven't had the time to read all the responses in depth (and not volunteering to summarize Ewen unfortunately :) but there is one perspective I haven't seen yet: you could also see conferences as one event in time in a large community of practitioners. In that case the conference may serve as a key catalyzer for contacts, new entrants and summarizing the state-of-the-art. It may be short-sighted to evaluate the event as a one-off, separate thing. It may also be too much asked of people to put a new idea they gain in a conference that leads to a whole new journey and needs greenhousing straight into practice. It seems too simple.

There's an experience that made we thinking about this. We had a community of practice with all kind of activities and connections going on. But what was evaluated was the book that the community produced...

John Smith, 2010/03/05

I really agree with you, Joitske. (And I haven't read this thread carefully, either/) A conference is a platform - with all kinds of things that attach to it. So we have to assess its value against "platform-sized" metrics. For example, one thing I've been thinking of is how conferences can be big (even "commercial") platforms where sub-communities form. CPsquare tried offering its own conference in 2002 but it almost caused nervous breakdowns. So ever since we've attached to other conferences of convenience. For us it has been a successful strategy - keeping conversations alive and moving them forward. (I wrote up some thoughts on how tagging can play a part in making our "sidecar" events possible on my blog: http://learningalliances.net/2010/01/tagging-and-face-to-face-events/)

So I think that conference organizers should ask, "Did we facilitate the spawning of appropriate or desirable sub-communities?"

If the communities that sponsor or benefit from a conference are the stakeholders of the conference impact analysis, there might be a whole other series of questions that are asked. Among other things, the impact is initially invisible. Also, the question is not whether to hold a conference or not, but "what else could have been done to move the community (or communities) forward?"

Peter J. Bury, 2010/03/11

Julia

I seem to be the first to react. A bit surprised about that. Actually I almost overlooked your query! Wonder if my response will trigger some reaction, among other from my colleagues at IRC.

No, IRC does not have any formal grouping of people that guide our either internal or external KM activities.

The closest we came to that was probably a multi-year Dutch Government funded programme called "Resource Centres for Development" (aka RCD), which evolved into the current WASH (for Water Sanitation and Hygiene) "Sector Learning" thematic work, about which concept we recently published a 'factsheet' (not sure what to call it), attached.

Though over the years we had various attempts to create IRC internal bodies to guide ourselves in KM, my view on it is that we keep searching and are not prepared to seriously invest in any 'body' more formal.

Jaap? Ewen? Viktor? comments?

WASH Sector Learning e-version IRC10.pdf

Manuel Flury, 2010/03/11

Dear all,

our Division (Knowledge and Learning Processes) has decided to award a prize for "good learning" to collaborators of SDC (the "Eléphant d'Or" ). We intend to invite collaborators to propose colleagues with outstanding and extra-ordinary merits related to "institutional learning" and bringing forward SDC's institutional learning. The proposals will be submitted to an open rating in the SDC IntraWeb.

Our query: We'd like to illustrate in the call for proposals what "good learning" is. E.g. "Bringing experts to sharing their experiences in a regular cafeteria meeting", "Writing a lessons-learnt paper that is being downloaded by xyz number of people", "Successfully replicating (scaling-out) a successful approach in another country/region".

What are your examples of a "good learning" practice to be honoured?

Looking forward!

Roxanna Samii, 2010/03/11

Dear Manuel

What an interesting initiative. We're currently conceptualizing something similar.... We want to honour early adopters who have put their "good learning" to use to trigger a shift from business as usual to adopting and using knowledge sharing methods and tools - which believe me sometimes is considered as an innovation!

Your msg has helped me to clear the cobwebs in my head and finally put pen to paper and write our concept note!!! Thank you!

Charles Dhewa, 2010/03/11

Manuel,

Thanks for these ideas. I am trying to imagine how you can capture learning through mistakes. Not sure about the phrasing.

May be some of your collaborators who have implemented ideas badly can innovatively share what and how they learnt from it.

Julia Cleaver, 2010/03/11

Manual,

We are doing something similar in our organization. We have recently completed the process of redesigning our intranet and moving it from SharePoint 2003 to 2007. It took over a year to get all of our headquarters units moved because we were incorporating great KM principles in the design and doing lots of training along the way. Now we want to recognize the staff that are really 'stars' in using this new technology for knowledge exchange. We call our intranet Luna. So the Luna team find "Luna stars" on a regular basis. They are chosen for diverse reasons. Some examples are below. While these are maybe small steps, they are building the corporate culture and getting staff to talk to each other about better ways of sharing information across the organization. I know this is focusing on a technological tool and there are many other KM practices to encourage, this is the big initiative we want to encourage right now. The hokey part of it is that the team actually puts a star with the person's picture outside their office and also on a public bulletin board in addition to the Luna announcement page. The goal is to get staff to ask them what they did to get the recognition.

Allison was tasked with creating a wiki version of the Resource Center's Procedure Manual. Allison exceeded expectations by creating a wiki that is so intuitive, user-friendly and easy on the eyes that units all over have replicated the design. Thanks to Allison for planting the seeds on which many wiki pages are built! You're this week's Luna Star!

Not only has Merrill done an excellent job of keeping the Stategic Planning site up-to-date with the latest information, she also has taken the Single Source of Truth credo to heart! Merrill has been diligent in emailing links to documents (instead of sending attachments) and creating links to documents within Luna sites instead of saving second (and third and fourth) copies of the same item. We are proud to make her this week's Luna Star!

Allison Hewlitt, 2010/03/11

Hi Manuel

What about a team award going to a group of people who have been able to design and implement a learning oriented approach to their regular team meetings ie. they don't just create a list of topics to be covered in unreasonable amount of time but put effort into thinking about the desired outcomes and considering which methods and tools will help them get there.

I can't say that I have ever met someone who gets excited about taking part in their regular team meeting but maybe there are a few within SDC's walls.

Carl Jackson, 2010/03/11

Like this tangent Charles, how about an award for “listening to and including critical voices from beneficiaries”. I think its a tough award to even get shortlisted for but surely an important one.

Manuel Flury, 2010/03/17

Dear Allison, Julia, Carl, and Charles,

dear all,

thanks a lot for your suggestions what "good learning" in our practice would include. In brief what I have learned so far:

• Charles Dhewa proposed to award "good" learning from a "bad" practice, to honour the (good) way these bad experiences have been translated into a success.

• Julia Clever told about collaborators using the organisation's new intraweb in an innovative way: championing by creating wiki conversation; championing in sending links (to the document management system) instead of attaching copies of documents.

• Carl Jackson considers "listening to and including critical voices from beneficiaries into policies" good learning.

• Allison Hewlett proposes to focus the award on teams, groups of people who have been able to design and implement a learning oriented approach to their regular team meetings (ie. they don't just create a list of topics to be covered in unreasonable amount of time but put effort into thinking about the desired outcomes and considering which methods and tools will help them get there)

All proposals deal with an individual or group learning performance that may incite others to follow, to copy. These proposals do not yet transform the organisation. They, however, prepare transformation towards more conscious learning. They all represent a "bottom-up" way to change. I am convinced about the transforming momentum such initiatives can create.

Nevertheless my second query:

What initiatives do you know that strengthen the transformative momentum of such "grassroots' initiatives"?

Tarit Kumar Datta Gupta, 2010/03/17

Dear Manual

Perhaps you have missed the one I have written on "good practice" that was the outcome of my study on "good practice" for Regional River Basin Programme of Oxfam GB in Bangladesh, India, and Nepal. To conduct the study I visited Bihar of India. I hope this will give some more understanding, and the question that you have raised today.

Hannah Beardon, 2010/03/20

Dear all,

I have just come out of a workshop looking at what it means to use the information and perspectives generated through participatory work more widely and effectively and my head is still pretty jumbled with the ideas and issues which came out of it. However one of them was definitely learning, and how difficult this is to do within the organisational structures and cultures of most large INGOs/ development organisations.

Someone commented to me at the workshop that she realised it would be much easier for her to go back to work and clear a day to write an article, than to spend 5 minutes a week reflecting on her practice in relation to the issues we had discussed. This is because the latter would require a different approach to work fundamentally, although little time in reality. We are all quick to say that TIME is an issue for good learning, sharing etc, and yet we spend so much time on activities which could be transformed into learning or reflection opportunities.

Recently I was at an organisation in Peru who have to write about 15 6-month reports to their UK office to enable them to report to DFID and other donors. In practice the 4 directors responsible for this reporting would spend a whole day, or more, sitting alone with their laptops writing from memory according to what they had in their heads... and in practice noone in the organisation really read any of the reports. They noted to me that it would be much more useful if they talked to each other beforehand about lessons learned, impact, change etc, to reflect together, identify common issues, structure their thoughts before writing the reports, and generally share learning. And they already have a meeting every six months, although it had been much more managerial in nature. What's more the director I was talking to suggested he would like to use meetings with partners and communities to think together about some of the questions on the report - lessons learned, impact etc. So in fact with no more time, there could be a lot more learning and reflection, and the whole process could be less extractive and most likely the information arriving at the UK could be a lot more useful.

Although, as I said, the workshop is still a jumble to me a few things are already top of my mind which inspired me to repsond to your email:

we need to use the spaces that already exist (donor reporting, meetings etc) to promote reflection and learning learning is about change - how do we listen to others and how do we respond to that - and change is a very difficult process. Underestimating the impact of real learning is dangerous as initiatives can become threatening to existing power structures and individual people and teams. the bigger picture of changes to organisational structure and culture can be overwhelming, but I felt much more positive when we were thinking about how we can empower ourselves to make change happen. This helps me to understand how small things we do, changes in practice or development of new products and processes, are contributing to much larger changes.

I realise this is not an answer to your question, but we were talking about transformative (in small ways) practice - both past experisneces and ideas for the future - at the workshop and may be able to give you some examples as the notes are written up.

Tarit Kumar Datta Gupta, 2010/03/22

Dear Manual

I am very happy to keep me abreast of. Awaiting your follow-up message for KM4Dev.

Peter J. Bury, 2010/03/29

Manuel and others on awarding as in encouraging, stimulating

IRC is also on the awards paths *"Tell us a story - for pride and prize" * more on http://www.irc.nl/page/51946

Manuel Flury, 2010/03/29

Many thanks Peter, this is highly helpful! I'll speed up summarisin all ideas and proposals made by community members.

Tarit Kumar Datta Gupta, 2010/03/02

Dear Brad and my other colleagues

It has become really interesting to see the encouraging discussion gradually increasing concerning the word “Impact”. To me the differences in opinion are the creative tensions instead of being the negative attitudes and the differences in opinion shall be unearthed and addressed to reach a common consensus. This is one form of knowledge management.

Any way, so far up to 2 PM of March 02, 2010 the responses that I have received are more or less close to my understanding, peter is more transparent, more clear understanding I see from Vu Bang Pham working for IFAD in Vietnam. I have come again for a number of reasons. First: Brad second comment in response to Dave “we need to convince the development aid sector though … “ frustrates me perhaps he could have rephrased to express his discontent with the development aid sector. Second: as I also see the confusion in Dave between output and outcome. And Third, I expected: Johannes working with UNDP (I repeat) and Lucie who is working with Km4Dev Secretariat (as I understand), Tammie Alzona who is silent for a long time and others will give their valuable comments about the use of the word Impact, not in this case only, but in general.

Let me please begin with what is the system model of an organization. Then on questions of Impact analysis I would like to quote UNDP Handbook on Monitoring and Evaluation which is lying in front of me while I was preparing my response to my colleagues. This handbook is the latest UNDP Book on M&E published in 2002 from the Evaluation Office, United Nations Development Programme, One United nations Plaza, New York, NY 10017, USA. I would humbly request all that this book can easily be downloaded from UNDP website and is highly rich and useful for any developmental practitioner. Then I would also like to quote some of my thirty years professional experiences directly and indirectly linked to Impact Study.

We need to keep in mind that what we are discussing we are discussing for the Future UN Project, that is, for the future UN Development Management System. As such, we shall not deviate ourselves from what have been practiced by UNDP for long, the latest development in this regard is 2002 because Impact is clearly imbibed in Monitoring and Evaluation. This further suggests that we shall be careful in wording our understanding about the subject concerned because UNDP or Future UN Development Management System will not accept whatever we think. We need to write those things which are logical. If my deliberations appear like counseling, I seek apology to Brad and my other colleagues of Km4Dev.

Brad, let us look at the system model of an organization. An organization is divided into three compartments: (1) people, (2) task, and (3) environment with small group of people as the nucleus working in the form of a team within an organization. The people consume resources from the environment (both internal and external), produces goods and/or services through accomplishing certain tasks, and delivers the same products and/or services to the same environment (internal and external). And to produce the products and/or services an organization adopts programs and projects. The system model of an organization further tells that an organization has vision, mission, strategy, programs/projects, and legal status sequentially to read reverse way (that is, from legal status toward opposite direction). (Ref.: MIM Handbook on Management, Manitoba Institute of Management).

Now let us see what is a project or program. When we say “project office or program office” we see the same picture of an organization. That is: people, task, and environment doing the same thing what has been suggested.

What I have written vision, mission of an organization, in the old days people used to write organizational goal and objective. Then what are the valiant aspects of a project or program: these are program/project activities, results, objective and goal. Brad, see how similar is the behavior between an organization and the program/project.

Brad, now I would like to draw your attention toward a different direction to understand me more clearly. That is logical framework popularly known as LFA or Project Planning Matrix (PPM). So far my knowledge goes, the LFA or PPM has been discovered by the American Corporate Sector, and since then it has been used by many development agencies, particularly the Scandinavian Countries whether private corporate or state developmental agencies. Germany has brought certain adaptation but using the same structure because it is well established hypothesis and we can not change the structure. In Bangladesh, the GtZ – state development agency has the mandatory conditions to use LFA or PPM from either the government or the private agency for any financial assistance. I am working as a certified moderator on LFA. It has got two parts: (1) situation analysis and (2) deriving PPM from situation analysis. The situation analysis that deals with cause-effect and means-end relations begins with problem tree, objective tree and alternate tree. Once the tree is selected from alternate analysis the tree is being placed in the LFA/PPM chart. The LFA/PPM is a four by four planning matrix, that is, 16 squares. The lowest row is the activities, the second row is the result (core means), the third row is the objective and the fourth row goal level statement. It is read from bottom to top. Brad, it is not possible for me to discuss LFA/PPM in detail but now you understand while the crux of the word “impact”.

Brad, Dave, Peter, Bang and my other colleagues, earlier we used to talk about measuring result but not enough longtime ago development practitioners identified that result is also difficult to measure at the end of the project, as such the practitioners reached to such an unanimous understanding that result shall be divided into two parts: output and outcome to facilitate measuring the achievement of the progress. To note that in the past, it was objective oriented project management but now-a-days almost all the development agencies (for Brad it is not all necessary to segregate development aid sector, non-aid sector etc.. because we are tired with the use of all these jargons which simply creates confusion instead of simplifying the characteristics of the agency) follow result based management (RBM) and UNDP is the Pioneer of RBM. UNDP defines RBM (quoting): RBM is a management strategy or approach by which an organization ensures that its processes, products and services contribute to the achievement of clearly stated results. Results-based management provides a coherent framework from strategic planning and management by improving learning and accountability. …” (Ref. Chapter 2, UNDP Handbook on Monitoring and Evaluating for Results).

Brad in the same Handbook UNDP clearly clarifies its understanding about result. It has given result chain: (1) inputs, (2) Outputs, (3) outcomes and (4) impact. The inputs are nothing but the activities. But UNDP again clearly articulates result: “overall purpose of monitoring and evaluation is the measurement and assessment of performances in order to more effectively manage the outcomes and outputs known as the development results”. (Ref.: A. Purposes of Monitoring and Evaluation; Chapter 1). UNDP further defines outputs as the specific products and services outcomes are the changes in development conditions (Ref. Ibid). And then where lies the impact, obviously it is the objective level statement the reasons that the program/project has been opted for or how the deliverable (result) will be used in future by the same agency or other agency. There is another important reason why the objective is the impact of the project. Brad, if you look into the LFA the most important column is the fourth column, the way it is read, which states the assumptions at different levels of PPM. Assumptions are the positive statements of a negative condition that must exist in the project location but beyond the control of the project holder. For example: government policies remain favorable to the private sector. And the assumptions are always cross-linked, as such no assumption is written at Goal because then it is necessary to write super goal statement which means that we will write five rows instead of four rows, as such does not fall into the category of LFA. To note that, as the LFA/PPM is derived from the objective tree of situation analysis, objective is also one form of deliverable but after a long time.

For peter, I agree with. Latrine is the product (output) being produced, and the use of latrine is the changes in development conditions (outcome). What about the impact. The impact is the reduced water or air borne diseases which will not occur only with the use of latrine but there will be some other activities and/or project to be implemented to reduce the water or air borne diseases.

I had the opportunity to conduct an impact study of the rural growth centers after eight years of developing the growth centers with the financial assistance by Swedish Government. The objective of developing the growth centers was to rejuvenate the rural economy at particular location in Bangladesh. On the other hand I had the opportunity to learn in 2000 from one of my friends working with Plan International Bangladesh that a mission from Denmark which was visiting Bangladesh to conduct impact study of a project was searching my friend after 18 years of a project of Danida that he worked with to take his interview.

Now Brad, Dave, Peter, Bang, Johannes, Tammie and my other colleagues I think now I am clear what I wrote in my first discussion about impact.

Dave Snowden, 2010/03/02

Your detailed response deserves more than a quick reply! However I wanted to pick up and isolate one point.

The linear concept of input, leading to outputs, leading to outcomes which in turn leads to impact is I think at the heart of the problem, It implies (and I can see why people would want this) a causal chain that can be replicated.

However if the system is complex (in the sense of complex adaptive) then any input is a stimulus or modulator which influences but does not determine impact. That means we need to start measuring the sensitivity of a system to different stimuli, and the way in which some stimuli produce a disproportionate effect in that they catalyze other inputs. This is newly developing area which has not hit the development sector yet, but we are working on it in related fields, loosely termed modulator mapping. It also leads us to evolutionary representations (such as fitness landscapes) and measure based on stability of landscapes. In all those cases mathematics are simplified by representation and linked micro-narratives. There is no point in measuring anything if the results do not convince both donors and recipients alike to take action

All of that moves the "impact" agenda on. I didn't confuse outputs and outcomes, I conflated them as the model means there is no real difference in what is measured in practice.

CHD6px <a href="http://zpxkhymktwkl.com/">zpxkhymktwkl</a>, [url=http://vxzpwabvzgmi.com/]vxzpwabvzgmi[/url], [link=http://wakyzkvhazxc.com/]wakyzkvhazxc[/link], http://lhulegktpsjg.com/

CHD6px <a href="http://zpxkhymktwkl.com/">zpxkhymktwkl</a>, [url=http://vxzpwabvzgmi.com/]vxzpwabvzgmi[/url], [link=http://wakyzkvhazxc.com/]wakyzkvhazxc[/link], http://lhulegktpsjg.com/

comment4, http://gigarupu.comuf.com/wholesale-handbags-in-the.html wholesale handbags in the, jncja, http://gigarupu.comuf.com/dropship-authenic-designer-handbags.html dropship authenic designer hand

comment4, http://gigarupu.comuf.com/wholesale-handbags-in-the.html wholesale handbags in the, jncja, http://gigarupu.comuf.com/dropship-authenic-designer-handbags.html dropship authenic designer handbags, bucqdh, http://gigarupu.comuf.com/fair-lakes-shopping-center.html fair lakes shopping center, gnrff, http://gigarupu.comuf.com/shopping-outlet-in-florida.html shopping outlet in florida, hlvdjn, http://gigarupu.comuf.com/victorinox-first-class-neck-pouch.html victorinox first class neck pouch, 047267, http://gigarupu.comuf.com/new-shopping-mall-in-san-francisco.html new shopping mall in san francisco, 281292, http://gigarupu.comuf.com/find-fake-gucci-purses-in-the-united-kingdom.html find fake gucci purses in the united kingdom, >:-PP, http://gigarupu.comuf.com/hsn-home-shopping-network.html hsn home shopping network, 951, http://gigarupu.comuf.com/tracy-zych-stylish-handbags.html tracy zych stylish handbags, cnad, http://gigarupu.comuf.com/etro-knock-off.html etro knock off, 412385,

comment6, http://leulankili.tk/lakme-fashion.html lakme fashion, 76798, http://dodglessranexc.tk/culture-and-shopping.html culture and shopping, 983, http://supzamibu.tk/apa-fashion-recruitment.html

comment6, http://leulankili.tk/lakme-fashion.html lakme fashion, 76798, http://dodglessranexc.tk/culture-and-shopping.html culture and shopping, 983, http://supzamibu.tk/apa-fashion-recruitment.html apa fashion recruitment, 8-((, http://subscheckfeslau.tk/argentina-fake-handbags.html argentina fake handbags, >:-[, http://exseninghong.tk/makola-shopping-mall.html makola shopping mall, >:-(, http://abscenabre.tk/toronto-fashion-designers.html toronto fashion designers,  :-[[[, http://campfultile.tk/sitemap.html Site Map prada knock off handbags, 564, http://fickrihara.tk/metallic-purse-in-the-devil-wears-prada.html metallic purse in the devil wears prada,  %PPP, http://supzamibu.tk/sewing-and-fashion.html sewing and fashion, irt, http://moppociti.tk/shopping-online-websites.html shopping online websites, >:O,

comment1, http:///laperla-fashions.html laperla fashions, 720, http:///summer-winter-fashion.html summer winter fashion, frvyu, http:///clutch-handbags-on.html clutch handbags on, 57741, http:///it

comment1, http:///laperla-fashions.html laperla fashions, 720, http:///summer-winter-fashion.html summer winter fashion, frvyu, http:///clutch-handbags-on.html clutch handbags on, 57741, http:///italys-fashion.html italy's fashion, duq, http:///francesca-biasia-handbags.html francesca biasia handbags, 09935, http:///center-festival-shopping.html center festival shopping, 8-))), http:///handbags-by-nine.html handbags by nine, 219, http:///disigner-fashion.html disigner fashion, qpewfs, http:///center-rhodes-shopping.html center rhodes shopping,  :PPP, http:///pink-prada-purses.html pink prada purses, 55593,