Designing & evaluating an english reading test for the non-Majors of civil engineering at haiphong publicrivate university

Vietnam national university, hanoi College of foreign languages --------–²—------- Designing & evaluating an English reading test for the non-majors of Civil Engineering at Haiphong publicrivate university Thiết kế và đánh giá một bài kiểm tra tiếng anh chuyên ngành cho sinh viên xây dựng dân dụng tại trường đại học dân lập hải phòng M.A. minor thesis Field: methodology Code: 50702 Course: k11 By : Nguyen Thi Phuong Thu Supervisor : Tran Hoai Phuong, MEd. Hanoi - August 2005 Acknowledgements During the process of further studying and conducting this research I was really honored to receive guidance, assistance, and encouragement from various lecturers as well as supervisors among whom I would like to acknowledge my sincere thanks to the leaders of the College of Foreign Languages who have given me permission and created favorable conditions for study and research. I would also like to thank my supervisor, Mrs.Tran Hoai Phuong, Med, who really sympathized with me and also gave me great help as well as invaluable guidance and encouragement from the very start to the end of my research. It is also my pleasure to give my special thanks to the students of classes XD 501, XD 502 and XD 503 at Hai Phong Publicrivate University who enthusiastically took part in doing the test and helped me collect the results of the test. I also benefited greatly from talks and discussions with my colleagues so let me thank all of them for what they have directly or indirectly contributed. And finally I really want to thank my beloved husband who always gives great support to my further study. Nguyen Thi Phuong Thu List of abbreviations 1. HPU Haiphong Public University 2. CE Civil Engineering 3. CEE Civil Engineering English 4. ESP English for Specific Purposes 5. MCQ Multiple Choice Question 6. T True 7. F False 8. M Mean 9. S Sum of 10. N The number of the scores 11. x The raw score 12. f The frequency with which a score occurs 13. H The highest value 14. L The lowest value 15. SD Standard Deviation 16. FV Item difficulty 17. R The number of the correct answers 18. ve very easy 19. e easy 20. d difficult 21. vd very difficult 22. D Iitem discrimination 23. CU The number of the correct asnwers of the upper half 24. CL The number of the correct asnwers of the lower half 25. gd good discrimination 26. md bad discrimination 27. bi bad item 28. p Spearman rho correlation coefficient 29. SU Score on the upper half 30. SL Score on the lower halfTable of contents Acknowledgement List of abbreviations Part I: Introduction 1 1.Rationale 1 2.Aims of the study 2 3.Scope of the study 2 4.Methods of the study 2 5.Design of the study 3 Part II: Development 4 Chapter one: Literature review 4 1.1.Language testing 4 1.2.Communicative language tests 6 1.3.Testing reading skills 7 1.3.1.Multiple choice questions 7 1.3.2.Short answer questions 7 1.3.3.Cloze 7 1.3.4.Selective deletion gap filling 8 1.3.5.C tests 8 1.3.6.Coloze elide 8 1.3.7.Information transfer 8 1.3.8.Jumbled sentences 8 1.3.9.Matching 8 1.3.10.Jumbled paragraphs 8 1.4.Major characteristics of a good test 9 1.41.Reliability 9 1.4.2.Validity 10 1.4.2.1.Content validity 11 1.4.2.2.Face validity 11 1.4.2.3.Criterion-related validity 12 1.4.2.4.Construct validity 12 1.4.3.Practicality 12 1.4.4.Discrimination 13 1.5.Achievement tests 13 1.5.1.Class progress test 14 1.5.2.Final achievement test 14 Summary 14 Chapter two: Methodology 16 2.1.A quantitative study 16 2.2.The selection of participants 16 2.3.The materials 16 2.4.Methods of data collection and data analysis 17 2.5.Limitations of the research 17 Summary 17 Chapter three: Discussion 18 3.1-The content area of the test 18 3.2-The relative weights of the different parts of the test 19 3.3-Constructing the test 19 3.4-Administering the test 24 3.5-Marking the test 25 3.6-Test scores interpreting and evaluation 25 3.6.1.The frequency distribution 25 3.6.2.The central tendency 26 3.6.2.1.The mode 26 3.6.2.2.The median 27 3.6.2.3.The mean 27 3.6.3.The dispersion 28 3.6.3.1.The low-high 28 3.6.3.2.The range 28 3.6.3.3.The standard deviation 29 3.7-Test item analysis and evaluation 30 3.7.1.Item difficulty 30 3.7.2.Item discrimination 32 3.8.Estimating reliability 34 Summary 36 Part III: Conclusion and recommendations 37 References 39 Appendices Part I: Introduction 1.Rationale Testing is a matter of concern to all teachers - whether we are in the classroom or engaged in syllabus/ materials, administration or research. We know quite well that good tests can improve our teaching and stimulate student learning. Although we may not want to become a measurement expert we may have to periodically evaluate student performances and prepare reports on student progress. Haiphong PrivatePublic University (HPU) is a university in which there are a number of classes of Civil Engineering (CE) for students of Construction Department. Generally speaking, non-majors, especially the students of this department, lack background knowledge of English. The non-majors of CE have chances to learn General English (GE) during their first three terms to prepare for their 120 periods of English for Specific Purposes (ESP) in the fourth term. In fact, this type of English is quite demanding for them and many had to admit that they could not learn it well. As a result, many students failed after each final examination. The causes for the above situation are various. It might be because some students are either too hesitant or too lazy to learn anew subject. It might also be because some students could not overcome the difficulties they usually meet during their study, for example their ESP is too new or too demanding for them, or they have to learn many periods per week to leave time for other subjects. However, the reason which is no less important and which needs taking into account is the matter of testing. In general, teachers at HPU are well-qualified and when teaching they are quite enthusiastic with good teaching methodology. However, the results of their students’ tests are not always satisfactory, the scores they gained were often lower than expected. Moreover, we teachers cannot deny the fact that sometimes the test results do not accurately reflect the testees’ language competence. According to Brown (1994a: 373) and Hughes (1989: 1) “A great deal of language testing is of very poor quality. Too often language testing has a harmful effect on teaching and learning and too often they fail to measure accurately whatever it is they are intended to measure.” For all the above reasons the author of this research study would like to take this opportunity to undertake the study entitled “Designing a reading test for the non-majors of Civil Engineering at Haiphong Public University” with a view to evaluating the students’ reading abilityBai doc thi chac la chi kiem tra duoc kha nang doc thoi, chu khong phai la toan bo language ability dau em. after one term’s study last school year (2004-2005) as well as to gaining some knowledge and experience of foreign language testing for herself after completing better the study. 2.Aims of the study The minor thesis is aimed at designing an achievement test of ESP reading which would be conducted in a class of Civil Engineering English at HPU. The test was considered as a final examination. Then the results of the test will be analysed, evaluated, and interpreted. The test takers are non - English - majors. The specific aims of the research are: to assess the learners’ achievement in improving reading skill with English of Civil Engineering after 120 period reading course. to measure their aptitude for the reading skill. to diagnose their strength and weakness in reading the subject matter. to find out whether or not the test satisfies the qualities of a good test. From there the test will measure the effectiveness of the teacher’s teaching. If the test is not a good one, some suggestions will be made for a better test form. 3.Scope of the study “Not all language tests are of the same kinds. They differ with respect to how they are designed, and what they are for; in other words, in respect to test method and test purpose.” (Mc Namara, 2000: 5). For example, in terms of method, there are paper-and-pencil language tests, performance tests, ect. And in terms of purpose, there are achievement tests, proficiency test, and so on. In fact, the same form of test may be used for different purposes, although in other cases the purpose may affect the form. Due to the limitation of time and ability, it is impossible for the author to design tests of all these types or of all the four language skills (speaking, writing, listening and reading). Therefore, this minor thesis is limited to designing and evaluating an achievement test of ESP reading for the non-majors at HPU and the reading tested was for communicative purposes. 4.Methods of the study In this minor thesis the author designed an achievement test of reading, administered it and then evaluated it, so the method adopted is quantitative. The data will be collected through testing the students’ reading ability of Civil Engineering English. 5.Design of the study The study is composed of three parts: *Part I is the presentation of basic information such as the rationale, the scope of the study, the aims of the study, the methods of the study and finally the design of the study. *Part II includes three chapters: + Chapter one is the literature review in which the literature that is related to language testing and major characteristics of a good reading test is presented. + Chapter two is concerned with research methodologies including the methods adopted in doing the research, the selection of participants, the materials, the methods of data collection and data analysis. + Chapter three is the discussion, which is the main part of the study. This chapter reviews how a reading test of Civil Engineering for the non-majors at HPU was designed, administered, and then evaluated. *Part III includes the conclusion and recommendations for further research on the topic. Following these parts are the references and appendices. Part II : Development Chapter one : Literature review This chapter will provide an overview of the theoretical background of the research. It is composed of five small sections. Section 1.1 brings a significant insight into the concept of language testing. Section 1.2 is the introduction of communicative language tests. Testing reading skills will be discussed in section 1.3 which is followed by section 1.4 with the investigation into major characteristics of a good test. The final area to be mentioned is a brief review of achievement tests which is presented in section 1.5. 1.1.Language testing An understanding of language testing is relevant both to those who are actually involved in creating language tests, and also to those who are involved in using tests or the information tests provide in practical research contexts. For this very reason, this section wishes to take a close look at what a language test is. Most researchers agree that language tests play many important roles in life. Firstly the moment one does a test can be considered an important transitional moment in his life, for example, a pupil wishing to enter a university has to pass the entrance tests, or a job seeker has to do a certain test so that the employer will know whether he is competent, or if somebody needs to drive a motor or a car, he or she has to pass a driving test, ect. Secondly, language tests are also important to many occupations. We teachers rarely teach without testing our students’ performance in the subjects. Tests will help us to put them in right places; therefore, language tests, if used properly, can be considered a valuable teaching device for any teacher, and they will contribute positively to the development of both teachers and learners. Last but not least, any researcher who needs measurement of the language proficiency of the subjects cannot do it without using an already existing test or designing his or her own test. As for Caroll (1968) a test in general will certainly tell something about a testee’s characteristics. Thanks to the results from his test, it is possible for a teacher to judge whether this student is good or bad at the subject tested. Caroll provides the following definition of a test: “a psychological or educational test is a procedure designed to elicit certain behavior from which one can make inferences about certain characteristics of an individual.” (Caroll, 1968: 46) According to Hughes (1989: 9), tests can be classified as follow: Proficiency tests Achievement tests Class progress tests Final achievement tests Diagnostic tests Placement tests Aptitude or Prognostic tests Direct tests versus indirect tests-Discrete- point tests versus intergrative tests Norm-referenced tests versus criterion-referenced tests Objective tests versus Subjective tests Communicative tests Generally there are some approaches to tests, for example the essay-translation approach, the structuralist approach, the integrative approach, or the communicative approach. However, in this minor thesis, I would like to choose only the communicative approach to testing. This approach focuses on how the language is used in communication ( Em vua noi la bai nay focuses on how the language is used in communication, xong roi lai mo ngoac giai thich la “what” rather than “how” la sao? ‘meaning’ rather than ‘form’). This attempts to obtain different profiles of a learner’s performance in the language. The development and the use of language tests involve an understandingof the nature of communicative language use and language ability, on the one hand, and of measurement theory, on the other. Each of these areas is complex in its own right. In short, like teaching, testing is important to any teacher as well as for any student. It is difficult to deny that testing cannot be separated from teaching, testing can even be seen part of teaching. Therefore, we teachers should pay great attention to the issue of testing in our teaching. 1.2.Communicative language tests There is one thing that is essential to the activities of designing a test and interpreting the meaning of test scores. It is the view of language and language use embodied in the test. The term ‘test construct’ refers to these aspects of knowledge or skill possessed by the candidate which are being measured. To define test construct it is important to be clear about what knowledge of language consists of and how that knowledge is used in actual performance (i.e. language use). It is also essential to understand what view the test takes of language use because if the view the test takes is different, then the test will be different. As a result, the reporting of score will be different, and the test performance will be interpreted differently. Therefore, the difference of format between tests is not just incidental; it implies a difference between views of language and language use. Accordingly, communicative language tests are different from other types of tests such as discrete point test or integrative and pragmatic tests in the following aspects: According to Mc Namara (2000: 17) discrete point test focuses on students’ knowledge of the grammatical system, of vocabulary and aspects of pronunciation and tends to test these aspects of knowledge in isolation. With this type of test, multiple choice questions are most suitable. This discrete point tradition of testing is seen as focusing too much on knowledge of the formal linguistic system for its own sake rather than on the way the knowledge is used to achieve communication. Aslo as for Mc Namara using integrated tests is a new orientation in which integrated knowledge of relevant systemic features of language (pronunciation, grammar, vocabulary) with an understanding of context is deployed. Yet, these tests are regarded as time consuming and difficult to score. For example for an oral interview, the test will involve comprehension of extended discourse (both spoken and written), and as a result besides the disadvantages mentioned above it also requires trained raters. Because of those disadvantages another type of test, pragmatic test, replaced the old ones. It focuses less on knowledge of language and more on psycholinguistic processing involved in language use. With this type, a cloze test was seen the most suitable and was once believed to be easy to construct, relatively easy to score. However, it soon turned out to be measuring the same kinds of things as discrete point tests of grammar and vocabulary. It also failed to test communicative skills. In the early 1970s thanks to Hyme’s theory of communicative competence (an understanding of language and the ability to use language in context, particularly in terms of the social demand of performance, i.e. knowing a language is more than knowing its rules of grammar) communicative language tests developed and it has the two following features: ’They are performance tests which require assessment to be carried out when the candidate is engaged in communication, either receptive or productive, or both. They see language as a sociological phenomenon, focusing on the external, social functions of language while integrative and pragmatic tests see language as an internal phenomenon. With this test, the use of authentic texts and real world tasks may be developed.’ (Mc Namara, 2000: 16). One of its distinguishing feature that supersedes other types of tests is that besides systemic features of language, it requires students’ careful study of the communicative roles and tasks. All the reasons discussed above are regarded as a strong impetus that initiates this minor thesis into designing a reading test of ESP for communicative purpose, i.e. it is a communicative language test. 1.3-Testing reading skills In a reading test, test items are often set basing on the text itself. And often within the same test more than one typed of item, maybe two, three or more types of the following items are used: 1.3.1. Multiple-choice questions (MCQs) This is one of the most popularly used types for setting a reading comprehension test. When doing this test the candidate is required to select the answer from a number of given options, only one of which is correct. The marking is totally objective. Selecting and setting items are, however, subjective processes, and the decision about which is the correct answer is a matter of subjective judgment on the part of the item writer. 1.3.2. Short answer questions In the test there are questions which require the candidates to write down specific answers in spaces provided on the question paper. 1.3.3. Cloze This type is also familiar with students. In the cloze procedure, words are deleted from a text after allowing a few sentences of introduction. The deletion rate is mechanically set, usually between every fifth and eleventh word because deleting too many or too few words can cause problems with test validity. Candidates have to fill each gap by supplying the word they think has been deleted. 1.3.4. Selective deletion gap filling It is selecting items for deletion based upon what is known about language, about difficulty in text and about the way language works in a particular text. 1.3.5. C-Tests In C-test every second word in a text is partially deleted. In an attempt to ensure solutions, students are given the first half of the deleted word. The examinee completes the word on the test paper and an exact word scoring procedure is adopted. 1.3.6. Cloze elide In cloze elide test, words that do not belong to the original text are inserted into a reading p