Impact evaluation of development projects: A case study of the netherlands development organization’s project supporting small and medium enterprises

IMPACT EVALUATION OF DEVELOPMENT PROJECTS: A CASE STUDY OF THE NETHERLANDS DEVELOPMENT ORGANIZATION’S PROJECT SUPPORTING SMALL AND MEDIUM ENTERPRISES by Do Thi Dong A research study submitted in partial fulfillment of the requirements for the degree of Master of Business Administration Examination Committee Dr. Do Ba Khang (Chairman) Dr. Fredric William Swierczek Dr. Sununta Siengthai Nationality Vietnamese Previous degree Bachelor of Industrial Business Management Hanoi National Economics University Vietnam Scholarship Donor Government of Switzerland/ Swiss Development Cooperation (SAV) Asian Institute of Technology School of Management Bangkok, Thailand April 2001 TABLE OF CONTENTS ACKNOWLEDGMENT IV ABTRACT V CHAPTER 1: INTRODUCTION 1 1.1. Rationale 1 1.2. Problem statement 1 1.3. Research objectives 1 1.4. The scope of the research 2 1.5. Organization of the Research 2 CHAPTER 2: LITERATURE REVIEW 3 2.1. Development project 3 2.1.1. What is a development project? 3 2.1.2. Types of development project 4 2.2. Project evaluation 5 2.2.1. What is project evaluation? 5 2.2.2. Types and purposes of project evaluation 5 2.2.3 The project evaluation framework 5 2.2.4. Evaluation approaches 7 2.3. The impacts of development projects supporting SMEs 7 2.3.1 Social impacts 8 2.3.2. Economic impacts 8 2.4. The current profile of NGOs in Vietnam 11 2.3.1. What is an NGO? 11 2.3.2. Types of NGOs 12 2.3.3. Advantages and disadvantages of NGOs 13 2.3.4. Current situation of NGOs in Vietnam 14 2.3.5. Roles of NGOs in Vietnam 14 CHAPTER 3: METHODOLOGY 17 3.1. Research design 17 3.2. Sample design 19 3.3. Data collection 21 3.4. Data processing 21 CHAPTER 4: (SNV)’S PROJECT SUPPORTING SMES IN NINH BINH 23 4.1. SMEs in Ninh Binh province 23 4.1.1. Overview of Ninh Binh province 23 4.1.2. Existing problems of SMEs in Ninh Binh 23 4.2. Overview of SNV 24 4.3. Project description 25 4.3.1. Introduction 25 4.3.2. Project objectives and beneficiaries 25 4.2.3. Project activities 25 CHAPTER 5: FINDINGS AND DISCUSSIONS 28 5.1. Characteristics of the sample 28 5.1.1 Sample of enterprises 28 5.1.2. Sample of employees 29 5.2. Impacts of the project 29 5.3.1. Social impacts 29 5.3.2. Economic impacts 36 5.3. General Attitude of selected SMEs toward the project 41 CHAPTER 6 CONCLUSIONS AND RECOMMENDATIONS 43 6.1. Conclusions 43 6.2. Recommendations 44 REFERENCES 46 APPENDIX 49 Appendix A: Questionnaires 49 Appendix B: List of selected enterprises 53 Appendix C: Data 55 ACKNOWLEDGMENT I would like to express my gratitude to people, who have given me great support during my MBA study, especially when I conducted the research. This is my sincere thanks to Dr. Do Ba Khang, my advisor, for his guidance, support and recommendation to this work as well as his encouragement during my research study. I greatly acknowledge the advice and support of Prof. Fredric William Swierczek and Dr. Sununta Siengthai for serving as committee members of the examination together with useful comments to improve my works. I would like to express my acknowledgement to Swiss Government for the generous financial support. My special thanks are also extended to Dr. Hans Stoessel, SAV faculties and staffs in Swiss AIT Vietnam Development Management Program who gave me a lot of support, guidance and advice during the first year of my MBA study. My special thanks go to Mr. Toon Keijers, Coordinator and Mr. Heino van Houwelinggen, Advisor of Financial Service Department of the Netherlands Development Organization for providing me with the project data. My thanks are also extended to Mr. Duong Bien Thuy, Director of the Council of Cooperatives and Non- State Enterprises in the Ninh Binh province Vietnam and all SNV staffs including Mrs. Quynh Anh and Ms. Cam Tu for their help and kind support. I also would like to thank the management of selected enterprises for their enthusiasm and valuable support at the period of data collection. My special thanks are extended to Mr. Tran Duong Minh, my student, relative and dearest friend for his help and encouragement during the time I conduct the research. I also would like to extend my thanks to my colleagues at School of Management who gave me a great chance to participate in MBA program and also gave me great support during my MBA study. Also to be included in my list of acknowledgment are circle of my friends especially SAV6 members for sharing my happiness and sadness through my study and shaping my unforgettable memories of my life. Last but not least, I would like to express my deepest gratitude to my family who were tremendously scarified to my growth education, gave endless encouragement and support and provided spiritual strength for me. ABTRACT This study attempts to develop a framework to measure the impacts of development projects supporting small and medium enterprises (SMEs) by Non governmental Organization (NGO). It then tries to apply that framework to evaluation of a development project, which was carried out by Netherlands Development Organization (SNV), an Dutch NGO to support SMEs in Ninh Binh province Vietnam. Based on the sample of selected SMEs, data were collected to measure the impacts of the project in terms of Social and Economic aspects. The comparison between small size and medium size SMEs and among different SMEs in different industries were made whenever there is a difference on order to give insights for the donor, the project management and managers of the partner organization and related institutions. The results revealed that supported SMEs and their employees made a positive change in economic as well as social aspects. Among five industries given priority, embroidery industry appeared to be the best in making use of the project. However, changes seem not to be found in some social impacts such as health care and insurance. Some suggestions for further study are also given. CHAPTER 1: INTRODUCTION Rationale In the spirit of the “Doi Moi” (renovation) policy adopted by the government of Vietnam since 1986, Vietnam shifted from a centrally planned system to a market- oriented socialist country. Although the role of Small and Medium Enterprise (SME) in economic and social development was recognized, the business environment in this economy in transition presented great challenges to their development and growth. External constraints such as the transitional legal and administrative framework, scarcity of financial resources, intensifying competition as well as the internal constraints of company management, all exerted great pressures on SMEs. Despite the political willingness that existed to promote SMEs, there are still difficulties due to many problems such as lack of money, lack of knowledge, etc. Support from developed countries, thus, is very important to Vietnam. One kind of support from developed countries requiring less procedures than those approved by government agencies is in form of non-governmental organization- NGO. NGOs often target the poor section of the population. The utilization of the aid from these organizations in form of development project often brings prompt results. For this reason, it is important to evaluate these projects both in terms of economic and social aspects for donors, recipients and related institutions to not only know whether the project objectives have been achieved but also to draw lessons for relevant decision- makers for decision making about follow up activities, about other projects of the same type, or about other projects with the same institutional partners. As a result of the open door policy implemented in 1986, Vietnam is home to a lot of national NGOs. Most of the evaluation on NGOs’ projects so far took into account both economic and social impacts, but the framework is not sufficient. Donors, receivers and interested institutions find it difficult to have an overview on what the receivers have gain so far as well as what should they have dodge to achieve their objectives. Problem statement This research attempts to answer the questions “ How to know how much impacts of NGO’s development project on SMEs are” Research objectives With the above problems, the objectives of the research are as follows: Develop a framework to measure the impacts of development projects by NGOs. Attempt to apply the framework to a Netherlands Development Organization’s development project supporting SMEs in Ninh Binh province Vietnam with data from the research jointly carried out by the organization and the author. Draw conclusions and make recommendations on what the donor should do with another development project in the future. The scope of the research The research focuses on evaluation of NGOs’ development projects supporting SMEs. At first, a framework to measure the impacts of project is developed. Then, the study tries to apply that framework to explore the worth of SNV’s development project that supported SMEs in Ninh Binh province Vietnam. Aiming at improving capacity of the Council of Cooperatives and Non- State Enterprises of the Ninh Binh province to provide services to SMEs and of the SMEs, the project was started from October 1998 and terminated in December 2000 (Figure 1.1, project life cycle). Total 125 enterprises in five different industries (embroidery, reed weaving, agriculture, industry and service) in four districts of Hoa Lu, Gia Vien, Nho Quan and Kim Son were given the support. The survey was conducted in two periods when the project was in the termination stage. In the first stage, one survey to explore some issues including employment, income and revenue of the selected enterprises was carried out by SNV staffs in October 2000. Remaining issues were found out by the effort of the author in December 2000. Among recipients, studied enterprises have received support for at least one year. Due to the fact that the author have only 6 weeks for contacting and conducting interview at the mid of December when most of the enterprises in Vietnam are busy with their fiscal year report and with TET holiday, enterprises were selected by convenience. That means their willingness with the research is given priority in choosing respondents. 1.5. Organization of the Research The research is presented in six chapters as following: Chapter 1: Introduction including rationale, problem statement, objectives, scope and organization of the research Chapter 2: Literature review that presents the literature about development project, project evaluation including its impacts. Then more specific information about situation of NGOs in Vietnam is also introduced. Chapter 3: Methodology, this is the most important part where the framework to measure the project’s impacts is developed. In this part, the sample and data collection procedures are designed and finally data processing procedure is identified. Chapter 4: The project description. This part gives an overview on the donor organization and current situation of SMEs in the province. The main purpose of this part is presenting the description of the project in terms of objectives, size and activities. Chapter 5: Findings and discussions, the most interesting part. In this part, descriptive information about the sample characteristics is presented. The impacts of the project are shown. Chapter 6: Conclusions and recommendations. The chapter gives some conclusions about the evaluation of NGO’s development projects. Some suggestions for further study are also given. CHAPTER 2: LITERATURE REVIEW This part presents the literature about development project, project evaluation and project impacts. Then more specific information about situation of NGOs in Vietnam in terms of their activities and roles is also introduced. 2.1. Development project 2.1.1. What is a development project? What is a project? Many different answers have been given to the question, what is a project. In the manual of Commission of the European Communities 1993, project Cycle management: Integrated Approach and Logical Framework, prepared by the Management for Development Foundation, a project is defined as a dynamic and multi-dimensional intervention which is intended to: Remove or reduce the impact of certain constraints (e.g. constraints on urban waste management rural transport delivery of agriculture supplies production of construction equipment diet diversification food production scarcity of foreign currency). And/ or develop human physical and other potentials (labor, coastal zones, historic sites, hydraulic energy, etc.) What is a development project? The logical framework used by anonymous1 in Project Cycle Management, 1993 describes all development projects as having a hierarchy of objectives as follows: The project purpose: a specific objective to be achieved by the end of the project The results: a series of outputs The activities: the combination of a wide range of means (e.g. studies, investments, training, institutional changes) All projects also contribute to the achievement of broader policy or sectoral objectives, known as the overall objectives, and thus fit within a sectoral strategy. The intervention logic of a project is illustrated in figure 2.1. Regarding to this figure, a project involves combining resources including physical and non- physical means, which are carefully defined, together with a careful plan, they lead to some new situation of the objects. We call these situations results. Projects are designed to bring about changes. In addition to their direct results, they generally have wider impacts on the economy, environment, communities, and institutions. Figure 2.1. The intervention of logic of a development project Source: Manual of Commission of the European Communities, 1993 2.1.2. Types of development project In terms of the outputs, there are three types of development projects Project with tangible products, i.e. products which can be valued in monetary terms. Such projects generally aim to increase the production of goods and services: Which are sold or not sold but for which a market exists (such as food grown for home consumption). E.g. industrial projects, power development projects, or credits for firms projects Which are provided without direct payment by beneficiaries, or whose benefits can be “easily valued”. Examples include road projects, most of whose economic consequences concern transport costs and existing (or induced) traffic volumes or even savings in insurance costs, and can thus be calculated. Examples include road infrastructure, professional training and agricultural extention projects Projects with non- tangible products, i.e. products which can not be accurately valued in monetary terms without carrying out research which exceed the time and resources. Examples of these projects are projects dealing with health, education, urban development (e.g. sewerage), social services, institutional reform or environmental conservation. Combination e.g. after finishing, the project brings both tangible products and some intangible results. In this research, the project to be examined belongs to this type. One side, it facilitates SMEs by establishing an information database to provide information about market, technology, economic opportunities. On the other side, it includes training activities, policy development and services to improve ability of small sized enterprises to access financial services. 2.2. Project evaluation 2.2.1. What is project evaluation? According to Reidar Dale, author of the book “Evaluation Frameworks for Development Programs and Projects”, project evaluation is thorough examination, at specific points in time, of a project or parts of them, usually with emphasis on impacts and additionally commonly on efficiency, effectiveness, relevance, impacts and sustainability”. This definition resembles numerous other definitions of evaluation relating to societal development work. For instance, Rossi and Freeman (1993) specify it as “the systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of social intervention programs”, and Carlsson, Kohlin and Ekbom (1994) consider it “an activity for finding out the value or results of something, ... answering the information needs of various actors” 2.2.2. Types and purposes of project evaluation Depending on how the provided information is to be used, Reidar Dale (1998), distinguish between two main types of evaluations: formative and summative. Basic features and general purposes of the two are as follows: Formative evaluations aim at improving the performance of the program or project which is evaluated, through learning from experiences gained. In addition, it serves as a mechanism to ascertain the accountability of people with responsibility in the respective program or project. Formative evaluation are commonly done more than once. Each exercise may not be very comprehensive. The evaluations may be done at set intervals or according to the needs, as assessed by the responsible agencies, in the course of program or project implementation. Summative evaluations are under taken after one part of or a whole program or project has been completed. Their general purpose is to judge the worth of the program or project and the appropriateness of its design and management. Summative evaluations has largely been triggered by a need among foreign donor agencies to prove their accountability vis-a-vis their government and/ or other money providers as well as the general public in the donor country. For this reason, summative evaluations have mostly been undertaken by persons who are considered independent of the responsible program or project organizations and the donor agencies. In this research, summative evaluation framework is conducted to explore the worth of NGOs’ projects supporting SMEs and is then illustrated by an evaluation of one SNV’s project. 2.2.3 The project evaluation framework Reidar Dale (1998) built a basic evaluation model as illustrated in figure 2.2. The five main evaluation categories are efficiency, effectiveness, relevance, impacts and sustainability. They are specified as follows: Efficiency is defined as the amount of outputs created and their quality in relation to the resources (capital and personnel) invested. Efficiency is a link between ‘inputs’ and ‘outputs’. Effectiveness: This expresses to what extent the planned outputs, expected effects and intended impacts (development objectives) are being or have been produced or achieved. Relevance: This issue here is to what extent the program or project is addressing or has addressed problems of high priority mainly as viewed by stakeholders, particularly the program’s or project’s beneficiaries and any other people who might have been its beneficiaries. Impacts: these are the longer- term, largely indirect consequences of the program or project for the intended beneficiaries and any other people. Sustainability: this means the maintenance or augmentation of positive changes induced by the program or project after the later has been terminated. Figure 2.2. Evaluation Framework for Development Program and Project Notes: Design link Means- end link Other influence links Sources: Evaluation Framework for Development Program and Project, 1998, Chapter 3. Evaluating effectiveness, efficiency, relevance and sustainability need a long time and huge effort to study and contact with stakeholders. Given the time and resources constraints, this study only focuses on impacts of the projects. 2.2.4. Evaluation approaches The procedure to conduct an evaluation depends on the e