A study on the reliability of the achievement writing test for the second year English major students at N.A.JTTC

 INTRODUCTION 1. RATIONALE There is no doubt that testing is an essential part of language teaching and learning. A language test in general can be a “ sample language behavior and infer general ability in the language learnt.”(Brown D.H, 1994:252). In other words, from the results of the test, depending on different kinds of tests with different purposes as well, the teacher infers a certain level of language competence of his students in such different areas as grammar, vocabulary, pronunciation, or speaking, listening, writing and reading. It is obvious that the teacher plays a very important role in the process of assessment and measurement which is conducted through testing. It is said that “ language testing is a form of measurement. It is so closely related to teaching that we cannot work in testing without being constantly concerned with teaching.”(Heaton, 1988:5). There are various types of test which serve different purposes in foreign language teaching and learning. Among the kinds of tests and testing, writing tests are said to be less reliable from the point of both scorer and testee. This situation can be seen clearly at Nghe An Junior Teacher Training College. For many years, English writing has been considered the most difficult skill to be tested among teachers. Teachers have found it difficult to mark the achievement writing tests accurately, in particular mark compositions, as they blame that there is no rating scale for scoring compositions, or the provided rating scale is too general. Apart from this, many students are still worried about the results of the writing achievement tests, especially the task of writing a composition as they wonder if their writings are accurately evaluated by raters. That is the reason for choosing the topic of the research: A study on the reliability of the achievement writing test for the second year English major students at N.A.JTTC. It is hoped that the study will be helpful to the author, the teachers at the English department of

































































 N.A JTTC and to those who are concerned with language testing in general and the study of the reliability of writing achievement tests in particular. 2. AIMS OF THE STUDY The major aims of this study are: - to explore the relevant notions of language testing - to analyze the achievement writing test for the second year English major students on the basis of the syllabus, purposes of teaching and testing; and available data such as test scores and scores of sample compositions for evidences on its validity and reliability with a focus on reliability. - to provide some suggestions for test- designers as well as raters. 3. SCOPE OF THE STUDY Evaluating an achievement writing test consists of complex procedures and needs a number of criteria to be set up. However, due to the availability of data and limitation of time, this study focuses mainly on the reliability of the achievement writing test for the second year English major students at N.A JTTC. The results can be seen as the basis for providing some suggestions for test designers as well as raters. 4. METHODS OF THE STUDY On the basis of analyzing the teaching aims, and syllabus for the second-year English major students as well as the content of the writing test (term 1) as the practical base for the study, the quantitative method, which focuses on analyzing the test scores of 156 second year students and the scores of 15 sample compositions collected randomly, is used to measure the reliability of the test.

































































 5. DESIGN OF THE STUDY The study is comprised of three parts: Part I: Introduction provides information on the rationale for choosing the topic, the aims, the scope, and the methods of the study. Part II: Development ( divided into 3 chapters) *Chapter one: reviews the literature related to language testing (definitions, approaches, roles, purposes and relationships between teaching-learning- testing), testing writing (types of writing, criteria of testing writing and problems in testing writing) and the criteria of a good test with a focus on reliability in which factors affecting language test scores, methods to determine the reliability of a language test and measures for improving test reliability are mentioned. * Chapter two: presents an overview of the teaching, learning and testing situations at Nghe An Teacher Training college including a description of the second year English teaching aims, the writing syllabus /course book as well as the content of the writing achievement test. * Chapter three: presents the methodology of the analysis of the format of the writing achievement test, of the data collected from the test scores of 156 second year students and the scores of 15 random sample compositions in order to find out “to what extent is the marking of the writing achievement test reliable? Part III: Conclusion presents a summary and some recommendations/suggestions for test- designers and raters.



































































 DEVELOPMENT CHAPTER 1: LITERATURE REVIEW In this chapter, the theoretical background for the study is established. Firstly, the term’ language testing’ including the approaches, roles, purposes as well as the relationships between language testing, teaching and learning will be explored. Then testing of writing will be discussed and followed by an examination of the criteria of a good language test with a focus on the reliability. 1.1 Language testing 1.1.1 Definitions of language testing Testing is an important part of every teaching and learning experience and becomes one of the main aspects of methodology. Many researchers have given out definitions of testing with different points of view. Allen (1974:313) emphasizes testing as an instrument to ensure that students have a sense of competition rather than to know how good their performance is and in which condition a test can take place. He says ‘test is a measuring device which we use when we want to compare an individual with other individuals who belong to the same group.’. According to Carroll (1968:46), a psychological or educational test is a procedure designed to elicit certain behavior from which one can make inferences about certain characteristics of an individual. In other words, a test is a measurement instrument designed to elicit a particular behavior of each individual. According to Bachman (1990:20), what distinguishes a test from other types of measurement is that it is designed to obtain specific sample of behavior. This distinction is believed to be of great importance because it reflects the primary justification for the use of language tests and has implications for how
we design, develop and use them to their best use. Thus,
language tests can provide the means for more focus on the specific assure of interest. Besides, Ibe (1981:1) points out that “a sample of behavior under the control of specified conditions aims towards providing a basis for performing judgment.”. The term a sample

































































 of behavior used here is rather broad and it means something else rather than the traditional types of paper and pencils. Read (1983) shares the same idea with Ibe in the sense that a sample of behavior suggests language testing certainly includes listening and speaking skills as well as reading and writing ones. However, Heaton (1988:5) looks at testing in a different way. In his opinion, tests are considered as a means of assessing the students’ performance and to motivate the students. He looks at tests with positive eyes as many students are eager to take tests at the end of the semester to know how much knowledge they have. One important thing is that he points out the relationship between testing and teaching. In short, from the above descriptions, testing is an effective means of measuring and assessing students’ language knowledge and skills. It is of great use to both language teaching and learning. In order to understand more about language testing, we should have a look at the different approaches to language testing in the following part. 1.1.2. Approaches to language testing According to Heaton (1988:15), there are four main approaches to testing: (i) the essay translation approach, (ii) the structuralist approach, (iii) the integrative approach, and (iv) the communicative approach. (i) The essay approach is considered as an old method in which tests often focus on essay writing, translation and grammatical analysis. This approach requires no special skills or expertise in testing, but the subjective judgment of the teacher is considered to be of paramount importance. (ii) The structuralist approach places the main focus on testing concrete skills without relating them to the context. The skills of listening, speaking, reading and writing are separated from one another as it is considered essential to test each one at a time. The learners’ mastery of the separate elements of the target language (phonology, vocabulary and grammar) are also tested using words and sentences completely out of contexts so that a large number of samples of language forms can be covered in the test in a comparatively short time. Thus, the results of the students’ tests completely depend on the accurate forms of separate language aspects or skills tested rather than the total meaning of the discourse or ability to use the language appropriately and effectively.

































































 However, this approach is still valid for some types of tests for specific purposes as it is considered to be objective, precise, reliable and scientific. That is the reason why the typical type of test following this approach, multiple choice, is still widely used nowadays even though there is still a limited use for multiple choice items in many communicative tests. (iii) The integrative approach, in contrast, involves the testing of language in contexts and is thus concerned primarily with meaning and the total communicative effect of discourse (Heaton, 1988). Integrative tests, instead of separating the language into different aspects, are designed to test two or more skills at the same time (especially focusing on reading and listening or language components in integration as grammar and vocabulary. In other words, this type of tests is concerned with students’ global proficiency, not their mastery of separate elements or skills. The typical types of tests following this approach are cloze tests, dictation, oral interviews, translation and essay writing. However, according to Heaton (1988:16), integrative testing involves ‘ functional language’ but not the use of functional language, and thus they are weak in communication. (iv) The communicative approach is considered to be interactive, purposive, authentic, contextualized and should be assessed in terms of behavioral outcomes. Although both integrative and communicative approaches emphasize the importance of the meaning of utterances rather than their form and structure, communicative tests are concerned primarily with how language is used in communication ( Heaton, 1988:19). The communicative approach emphasizes the evaluation of language use rather than usage (‘use’ is concerned with how people actually use language for different purposes while ‘usage’ concerns the formal patterns of language). However, the communicative approach is claimed to be less reliable because of various real- life situations in different areas/ countries and according to Heaton (1988), in order to increase the reliability, especially in scoring, a very carefully drawn -up and well- established criteria must be designed. In brief, each approach to language testing has its weak points as well as strong points. Therefore, in Heaton’s point of view, a useful test will generally incorporate features of several of these approaches (Heaton, 1988:15).

































































 1.1.3. The roles of language testing According to Mc Namara (2003:4), language tests play a powerful role in many people’s lives, acting as gateways at important transitional moments in education, in employment, and in moving from one country to another. Language testing is used for the assessment, employment, selection and considered by raters as a means of placing students on particular courses. Moreover, language tests are also considered as the criterion to evaluate the language proficiency of the researchers who want to carry out their research in language study. F. Bachman (1990:2) has also mentioned to the importance of language testing. Language tests can be valuable sources of information about the effectiveness of learning and teaching (language teachers regularly use tests to help diagnose students’ strengths and weaknesses, to assess students’ progress, and to assist in evaluating students’ achievements); language tests are also frequently used as sources of information in evaluating the effectiveness of different approaches to language teaching, as sources of feedback on learning and teaching; and thus language tests can provide useful input into the process of language teaching. Conversely, insights gained from language learning and teaching can provide valuable information for designing and developing more useful tests. 1.1.4. The purposes of language testing Language tests usually differ according to their purposes. Hughes (1989:7) points out the different purposes of testing based on the different kinds of tests such as: - to measure language proficiency regardless of any language courses that candidates may have followed. - to discover how far students have achieved the objectives of a course of study. - to diagnose students’ strengths and weaknesses, to identify what they know and what they do not know. - to assist placement of students by identifying the stage or part of a teaching program most appropriate to their ability. According to Henning (1987:1-2-3), there are six major purposes of language testing which can be represented as follows: (i) Diagnosis and Feedback ( in order to find out students’ strengths and weaknesses)

































































(ii) Screening and Selection ( in order to decide who should be allowed to participate in a particular program of study) (iii) Placement ( in order to identify a particular level of the student and place him or her in a particular program of study) (iv) Program Evaluation ( in order to provide the information about the effectiveness of the programs) (v) Providing Research Criteria ( in order to provide a standard of judgment in a variety of other research contexts) (vi) Assessment of Attitudes and Sociopsychological Differences 1.1.5. Relationship between Language Testing and Language Teaching & Learning Though a large number of examinations and tests in the past tended to separate testing from teaching, Heaton (1988:5) emphasizes that teaching and testing in some ways are so interwoven and independent that it is very difficult to tease apart “Both testing and teaching are so closely interrelated that it is virtually impossible to work in either field without being constantly concerned with the others”. Heaton (1988:5) also notes: “Tests may be constructed primarily as devices to reinforce learning and motivate the students or as a mean of assessing the students’ performance in the language”. In the former case, testing is geared to the teaching, whereas in the latter case, teaching is often geared largely to the testing. According to Hughes (1989:1), the effect of testing on teaching and learning is known as backwash which can be harmful or beneficial. He puts more focus on the harmful test. If the test content does not go with the objectives of the course, then the backwash can be really harmful and it leads to problems of teaching in one way but testing in another way. In his view, some language tests have harmful effects on teaching and often fail to measure accurately whatever they are intended to measure. That is proved in the case of a writing test with only multiple choice items in which learners only concentrate on practicing such items rather than practicing the skill of writing itself. In general, testing and teaching -learning process have a very close relationship. Tests can be effective for both teaching and learning process and vice versa. Therefore, good tests

































































are useful, desirable and can be used as a valuable teaching device. For educational purposes, educators should improve both language tests and language teaching methods to have beneficial backwash. 1.2. Testing writing In this section, to understand what testing writing is, it is crucial to discuss the writing types, criteria to test writing accurately and problems in testing writing. 1.2.1. Types of writing Writing is defined as the productive skill in the written mode. It is complex and difficult to teach and test as well since it requires the mastery not only of grammatical and rhetorical devices but also of conceptual and judgmental elements in which the use of judgment is considered to be the most important in teaching and testing of writing. Normally, after clarifying the purpose of writing, the particular audience and register, an essay or a composition may be written in accordance with the four main types: narrative, descriptive, expressive, or argumentative. However, Heason (1988:136) divides the types of writing according to learners’ levels. In his point of view, there are three main levels: basic, intermediate and advanced. At the basic level, learners learn how to write letters, postcards, diaries, and forms. With the already -learnt types in the basic level, learners should be taught the ways to guide and set instructions at the intermediate level. And at the advanced level, learners have to learn more how to write newspaper reports and notes. 1.2.2. Criteria of testing writing There is a wide variety of writing tests which is needed to test many kinds of writing tasks that we engage in. According to Madsen (1983:101), there are three main types of writing tasks and tests as well: controlled, guided and free writing which correspond with their testing criteria. He points out the advantages and limitations of each type in which scoring in guided and free writings is considered to be subjective. Thus, McNamara (2000:38) says in the past, writing skills were assessed indirectly through examinations of control over the grammatical system and knowledge of vocabulary because of the problem of subjectivity. However, nowadays educators place more emphasis on teaching and testing

































































 learners’ communicative language abilities. That is the reason why there is a current trend in shifting from testing isolated items to testing writing compositions in which the managing of the rating process became an urgent necessity. John Boker indicates the disadvantages of testing writing compositions such as limited amount of content sampled, time- consuming of score, and low reliability in scoring.