Home

Integrating digital libraries and VLEs in higher & further education

Monitoring and evaluation

There is a certain lack of clarity in the community about the meaning and function of 'evaluation' activities. All projects appear to have engaged in some evaluation of their work - evidence of evaluation activities includes the use of familiar tools such as questionnaires, interviews, focus groups, user group meetings, scenario building and testing, and the development of demonstrators. There are many examples of the involvement of end-users in commenting on projects' work, although it is noted that where the project is primarily concerned with the production of middleware an end-user perspective is problematic. Indeed it could be argued in such cases that were end users to be able to identify the project's work it would effectively have failed!

Projects do not always understand the distinction between 'product user testing' and 'project evaluation' and projects may wish to focus on how they would make this distinction. The key issue concerns the differences between:

• The testing of products, which in essence is designed to check the conformance of a product (or a service) to its specification. For example, the project may have developed a functional specification in the early stages of its work; later both project staff and users recruited for the purpose will be engaged in checking that the product does indeed meet the specification. This is an essential step, and is closely related to quality assurance procedures (though the latter will be broader).
• The determination of the effectiveness of the project and, most critically, of its outcomes and impact. CSALT, as part of the LinkER team, has presented a methodology to enable projects to focus on outcomes from the earliest stages of their work. The emphasis here is on the effects which the project is having or has had on target and other audiences.
• Some projects commissioned external evaluators to assist them or use their Steering Group to guide and advise. When this is the case, they report great value from their activities, and that an 'outside' viewpoint helps them to focus their thinking, especially upon their project outcomes. Generally, we would commend an element of independence of view in project evaluation.

Additional Resources

Links:

• The EFX Project Evaluation Toolkit which has been set up for the FAIR and X4L projects contains extensive links to resources covering most relevant aspects of evaluation. (Notes on the above 'outcomes' methodology will also be found in the EFX Toolkit.)