Definition and Principles in Materials Evaluation [PDF]

Zitiervorschau

Definition and Principles in Materials Evaluation 1. Definition Materials evaluation may be defined as a procedure or a systematic appraisal measuring the potential value(s) of materials on learners in relation to their objectives (Tomlinson 1998, 2003). In other words, materials evaluation means a principled process of providing useful information about the targeted materials in order to select and/or develop them in a reliable and valid approach. Material evaluations can be impressionistic or empirical (Ellis 1997) and the impressionistic way of materials evaluation has come under criticism for being unempirical or unscientific (Mukundan 2006). Moreover, the evaluation practices have not been examined critically to determine the effectiveness and value in teaching-learning environments and this is likely the main potential reason why the literature suggests that selected coursebooks have been more of a hindrance than a benefit to teaching (Mukundan 2004, 2009).     

Materials evaluation can refer to either before the program begins or after or both. Here, materials evaluation refer to a before program evaluation of published textbooks. This is motivated by the need to choose materials that will be relevant and appropriate for a particular group of learners and also by the need to identify specific aspects of the materials that require adaptation. In materials evaluation the evaluator identifies a set of criteria that are used to reach a decision regarding which book to adopt and how it needs to be adapted. Instruments used in evaluating materials. The instruments used in evaluating materials or textbooks are questionnaires, checklists and guides.

Principles in Materials Evaluation There are two principles in materials evaluation: effectiveness and efficiency (Ellis 1998 in Tomlinson 1998). Effectiveness principle: Is a course book effective in meeting the needs of the learners? To answer this question, the evaluator needs to compare what the learners knew and were able to do before they used the course book with what they know and are able to do after they have used the book. Efficiency principle: Does a course book meet the needs of the learners more effectively than some alternative course books? To investigate efficiency, it is necessary to compare the learning gains evidenced by using one course book with the gains evidenced by another course book.

Principles in Materials Evaluation What is Materials Evaluation? It involves making judgments about the effect of the materials on the people using them and it tries to measure some or all of the following: 1. The appeal of the materials to the learners. 2. The credibility of materials to learners, teachers, and administrators. 3. The validity of the materials. 4. The reliability of the materials. 5. The ability of the materials to interest the learners and the teachers. 6. The ability of the materials to motivate the learners. 7. The value of the materials in terms of short-term learning (important, for example, for performance on tests and examinations. 8. The value of the materials in terms of long-term learning (of both language and communication skills) 9. The learners’ perceptions of the value of the materials. 10. The teachers’ perceptions of the value of the materials. 11. The assistance given to the teachers in terms of preparation, delivery, and assessment. 12. The flexibility of the materials. 13. The contribution made by the materials to teacher development. 14. The match with administrative requirements. Principles in Materials Evaluation Here are some theories which are articulated from teachers’ practice: 1. Language learners succeed best if learning is positive, relaxed, and enjoyable experience. 2. Language teachers tend to teach most successfully if they enjoy their role and if they can gain some enjoyment themselves from the materials they are using. 3. Learning materials lose credibility for learners if they suspect that the teacher does not value them. 4. Each learner is different from all the others in a class in terms of his/her personality, motivation, attitudes, aptitude, prior experience, interest, needs, wants, and preferred learning style. 5. Each learner varies from day to day in terms of motivation, attitude, mood, perceived needs and wants, enthusiasm, d energy. 6. There are superficial cultural differences between learners from different countries (and these differences need to be respected and catered for) but there are also strong universal determinants of successful language teaching and learning.

7. Successful language learning in a classroom (especially in large classes) depends on the generation and maintenance of high levels of energy. 8. The teacher is responsible for the initial generation of energy in a lesson; good materials can then maintain and even increase that energy. 9. Learners only learn what they really need or want to learn. 10. Learners often say that what they want is focused language practice but they often seem to gain more enjoyment and learning from activities which stimulate them to use the target language to say something they really want to say. 11. Learners think, say and learn more if they are given an experience or text to respond to than if they are just asked for their views, opinions and interests. 12. The most important thing that learning materials have to do is to help the learner to connect the learning experience in the classroom to their own life outside the course. 13. The most important result that learning materials can achieve is to engage the emotions of learners. 14. Laughter, joy, excitement, sorrow and anger can promote learning. Neutrality, numbness, and nullity cannot.

Qualities Each Unit of Material Should reflect)





 

1. 2. 3. 4. 5. 6. 



Many of the checklists and lists of criteria suggested in these publications provide a useful starting point for anybody conducting an evaluation, but some of them are impressionistic and biased. However, we must start on our evaluation by acknowledging that, to some extent, results are still inevitable subjective. This is because, any pre-use evaluation in subjective, both in this selection of criteria and in the judgments made by the evaluators. A useful exercise for anybody writing or evaluating language teaching materials would be to evaluate the checklists and criteria lists from a sample of the publications against the following criteria: Is the list based on a coherent set of principles of language learning? Are all the criteria actually evaluation criteria? Are the criteria sufficient to help the evaluator to reach useful conclusions? 4. Are the criteria organized systematically? 5. Are the criteria sufficiently neutral to allow evaluators with different ideologies to make use of them? 6. In the list sufficiently flexible to allow it be made use of by different evaluators in different circumstances? Both personality and of students and teachers, is that it is extremely useful to develop a set of formal criteria for use on a particular evaluation and then to use that set as a basis for developing subsequent contextspecific sets. The following are one way of developing a set of criteria:

1. Brainstorm a list of universal criteria  



Universal criteria are those which would apply to any language learning materials anywhere for any learners. They derive from principles of language learning and the results of classroom observation and provide the fundamental basis for any materials evaluation. The following are the examples of universal criteria would be: a. Do the materials provide useful opportunities for the learners to think for themselves? b. Are the target learners likely to be able to follow the instructions?

2. Subdivide some of the criteria If the evaluation is going to be used as a basis for revision or adaptation of the materials, or if it is going to be a formal evaluation and is going to inform important decisions, it is useful to subdivide some of the criteria into more specific questions.  For example; Are the instructions:   Succinct?   Sufficient?  Such a subdivision can help to pinpoint specific aspects of the materials which could gain from revision or adaptation. 3. Monitor and revise the list of universal criteria 

 



Monitor the list and rewrite it according to the following criteria: a. Is each question an evaluation question? If a question is an analysis question (e.g. ‘Does each unit include a test?’) then you can only give the answer a 1 or a 5 on the 5-point scale which is recommended later in this suggested procedure. However, if it is an evaluation question (e.g. ‘To what extent are the tests likely to provide useful learning experiences?’) then it can be graded at any point on the scale.

b. Does each question only ask one question?  Many criteria in published lists ask two or more questions and therefore cannot be used in any numerical grading of the materials.  This question could be usefully rewritten as:  Is the book likely to be attractive to your students?  Is it suitable for the age of your students? c. Is each question answerable?  This might seem an obvious question but in many published lists of criteria some questions are so large and so vague that they cannot usefully be answered.  For example:  Is it culturally acceptable?’ (Grant, 1987, p. 122).  Does it achieve an acceptable balance between knowledge about the language and practice in using the language? d. Is each question free of dogma?  The questions should reflect the evaluators’ principles of language learning but should not impose a rigid methodology as a requirement of the materials.  If they do, the materials could be dismissed without a proper appreciation of their potential value.  The following examples make assumptions about the pedagogical procedures of coursebooks which not all coursebooks actually follow:   Are the various stages in a teaching unit (what you would probably call presentation, practice and production) adequately developed?’ (Mariani,1983, p. 29).   Do the sentences gradually increase in complexity to suit the growing reading ability of the students? (Daoud and Celce-Murcia, 1979, p. 304) e. Is each question reliable in the sense that other evaluators would interpret it in the same way?  Some terms and concepts which are commonly used in applied linguistics are amenable to differing interpretations and are best avoided or glossed when attempting to measure the effects of materials.  For example, each of the following questions could be interpreted in a number of ways:   Are the materials sufficiently authentic?   Is there an acceptable balance of skills?  There are also a number of ways in which each question could be rewritten to make it more reliable and useful. 4. Categorize the list  



It is very useful to rearrange the random list of universal criteria into categories which facilitate focus and enable generalizations to be made. An extra advantage of doing this is that you often think of other criteria related to the category as you are doing the categorization exercise. Possible categories for universal criteria would be: o  Learning Principles o  Cultural Perspective

5. Develop content-specific criteria  These are criteria which relate to the topics and/or teaching points of the materials being evaluated.

‘Thus there would be a set of topic related criteria which would be relevant to the evaluation of a business English textbook but not to a general English coursebook; and there would be a set of criteria relevant to a reading skills book which would not be relevant to the evaluation of a grammar practice book and vice versa’ (Tomlinson, 1999, p. 11).  Examples of content-specific criteria would be: o  Do the examples of business texts (e.g. letters, invoices, etc.) replicate features of real- life business practice? o  Do the reading texts represent a wide and typical sample of genres? 6. Develop age-specific criteria  These are criteria which relate to the age of the target learners. Thus there would be criteria which are only suitable for 5-year-olds, for 10-year-olds, for teenagers, for young adults and for mature adults.  It would relate to cognitive and affective development, to previous experience, to interests and to wants and needs.  Examples of age-specific criteria would be: o  Are there short, varied activities which are likely to match the attention span of the learners? o  Is the content likely to provide an achievable challenge in relation to the maturity level of the learners? 7. Develop local criteria 

 



These are criteria which relate to the actual or potential environment of use. They are questions which are not concerned with establishing the value of the materials per se but rather with measuring the value of the materials for particular learners in particular circumstances. Typical features of the environment which would determine this set of materials are: o  The type (s) of institution (s); o  The resources of the institution (s);

8. Develop other criteria 

Other criteria which it might be appropriate to develop could include teacherspecific, administrator-specific, gender-specific, culture-specific or L1- specific criteria and, especially in the case of a review for a journal, criteria assessing the match between the materials and the claims made by the publishers for them.

9. Trial the criteria 

It is important to trial the criteria (even prior to a small, fairly informal evaluation) to ensure that the criteria are sufficient, answerable, reliable and useful. Revisions can then be made before the actual evaluation begins.

10. Conducting the evaluation 



From experience the writer has found the most effective way of conducting an evaluation is to: o  Make sure that there is more than one evaluator; discuss the criteria to make sure there is equivalence of interpretation; Answer the criteria independently and in isolation from the other evaluator(s)  There is no coursebook or materials to be perfect. Tomlinson (2003c: 15) said that it is because the needs, objectives, backgrounds and preferred styles of the learners differ from context to context. We need some models for teachers that coverage the criteria of materials. • In evaluating process, it can be examined by two evaluation: 1. External evaluation 2. Internal evaluation The External Evaluation The aim is examining the organization of the materials as stated explicitly by the author/publisher by looking at the description, or the claims made on the cover of the teacher‟s/students‟ book and the introduction also table of contents that should enable the evaluator to assess. The Example  an integrated skills series which is designed to offer flexibility with different teaching and learning styles. Fun for learners to use and easy for teachers to adapt.  Fully integrated grammar, skills, and lexical syllabuses provide a balanced learning experience. • Engaging topics motivate students and offer greater personalization.  A wide range of approaches exploits different learning styles.  Clearly, structured grammar presentations are reinforced with extensive practice.  Contextualized vocabulary focuses on authentic real-world language.  A variety of listening and speaking activities develop learning fluency  Learner training throughout the Student‟s Book and Workbook maximizes skills development. The example means that textbook for intermediate level students with different learning styles and different levels of motivation. This textbook is designed for flexible use and offered an integrated learning experience covering grammar, lexis, and skills. The italicized words are certain terms and key concepts. • Tasks and activities are designed to have a real communicative purpose rather than simply being an excuse to practice specific features. • We have placed a special emphasis on representing an accurate multicultural view of English as it is spoken today. Many courses still represent the English speaking world as

being the largest UK- and US-based. Considering the fact that there are now more nonnative English speakers than native, we have also included a variety of accents from a wide range of countries and cultures. • Throughout the Student‟s Book, learner autonomy is promoted via clear cross referencing to features in the Workbook and elsewhere. Here students can find all the help and extra practice they need. From the example of description and introduction we can conclude several point as follows: • The intended audience. We need to ascertain who the materials are targeted. • The proficiency level. We need to decide the materials claim to a particular level, such as beginner, intermediate, or lower. • We need to decide the context in which the materials are to be used, such as general learners or ESP and what degree of specialist subject knowledge is assumed in the materials. • We need to decide how the language has been presented and organized into teachable units/lessons. • We need to see the author‟s views on language and methodology and the relationship between the language, the learning process, and the learner.

The Other Factors • Deciding the materials be used are the main core course or supplementary. • Examining is a teacher‟s book in print and locally available. It is also worth considering whether it is sufficiently clear for non-native speaker teachers to use. • Examining is there any vocabulary list/index in the textbook. • The textbook should consist of the organization of the materials, give information about vocabulary study, skills to be covered, additional interactive digital material and so on, an indication as to how much class time the author thinks should be devoted to a particular unit. Those factors can be seen in the table of contents. • Examining is the visual material in the book, such as photographs, charts, and diagrams integrate into the text. • Examining are the layout and presentation clear or cluttered. Some textbooks are researched and written well but are so cluttered with information on every page that teachers/learners find them practically unusable. • Examining is the material too culturally biased or specific. • Examining are the materials represent minority groups and/or women in a negative way and present a „balanced‟ picture of a particular country/society. • Examining the cost of the inclusion of digital materials, such as CD, DVD, interactive games, quizzes and downloadable materials from the web and the essentials of digital materials to ensure language acquisition and development. • Examining the use for the learners of the inclusion of tests in the teaching materials, such as diagnostic, progress, and achievement.

The Internal Evaluation The aim is to analyze the extent to which the aforementioned factors in the external evaluation stage match up with the internal consistency and organization of the materials as stated by the author/publisher. The Factors of Internal Evaluation • The presentation of the skills in the materials. • The grading and sequencing of the materials. • There is the way of appropriate text beyond the sentence in reading/discourse skills. • The recordings are „authentic‟ or artificial in the listening skills. • The speaking materials should incorporate what we know about the nature of real interaction or offer artificial, dialogues. • The relationship between tests and exercises to learner needs and what is taught by the course material. • The material should suitable for different learning styles, independent learning, and learner autonomy. • The materials should motivate both students and teachers. Some materials may seem attractive to the teacher but would not be very motivating for the learners. • Rubdy (2003: 45) proposes three broad categories that are essential for evaluation: 1. The learners‟ needs, goals, and pedagogical requirements. 2. The teacher‟s skills, abilities, theories, and beliefs. 3. The thinking underlying the materials writer‟s presentation of the content and approach to teaching and learning respectively. The Overall Evaluation It is an overall assessment of the suitability of the materials. The factors to make an overall evaluation as to the suitability of the materials as follows: • The usability factor . • The generalizability factor. • The adaptability factor. • The flexibility factor. FACTORS AFFECTING MATERIALS PREPARATION 1. The curriculum, syllabus, and the learning competencies 2. Learners’ learning styles, aptitudes, proficiency 3. Pedagogical principles held by the teachers 4. Societal demands CONTEXT LEARNERS EDUCATIONAL SETTING IMPLEMENTATIONOF GOALS SYLLABUS CONSTRUCTION MATERIALS,CLASSROOM METHODS (lessons, tests) McDonough and Shaw (2000) present the following framework for materials and methods.The framework reveals that materials and methods cannot be seen in isolation but are embedded within a broader professional context.

CONTEXTUAL FACTORS 1. Learner factors-age, interests, level of proficiency in English, aptitude, mother tongue, academic and educational level, attitudes in learning, motivation, reasons for learning, preferred learning styles and personality 2. Setting-role of English in the country; role of English in the school; management and administration; resources available; support personnel; the number of pupils; time available for the program; physical environment; the socio-cultural environment; types of tests to be used; and procedures for monitoring and evaluating. A syllabus is an expression of opinion on the nature of language and learning; it acts as a guide for both teacher and learner by providing some goals to be attained. Hutchinson andWaters (1987:80) define syllabus as ‘at its simplest level a syllabus can be described as a statement of what is to be learnt. It reflects language and linguistic performance.’ This is a rather traditional interpretation of syllabus focusing as it does on outcomes rather than on process. However, a syllabus can also be seen as a “summary of the content to which learners will be exposed” (Yalden.1987:87). It is seen as an approximation of what will be taught and that it cannot accurately predict what will be learnt. Syllabus is the overall organizing principle for what is to be taught and learned. It is the way in which content is organized and broken down into a set of teachable and learnable units, and will include considerations on pacing, sequencing and grading items’ methods of presentation and practice, etc. Syllabus inventory is a list of the content to be covered in the language program much like a content outline.

  

Richards and Rodgers (1986) presents a useful framework for the comparison of the language teaching methods which illustrates the place of syllabus in the program planning.The Model has three levels: approach, design and procedure. Approach refers to the views and beliefs or theories of language and language learning on which planning is based. Design converts the principles in the first level (approach) into more practical aspects of syllabus and instructional materials. Procedure refers to the techniques and management of the classroom itself.

TYPES OF SYLLABUS (Reilley) Although six different types of language teaching syllabi are treated here as though each occurred “purely,” in practice, these types rarely occur independently of each other. Almost all actual language –teaching syllabi are combination of two or more of the types.The characteristics, differences, strengths, and weaknesses of individual syllabi are defined as follows: 1. STRUCTURAL (formal) SYLLABUSThe content of language teaching is a collection of the forms and structures, usually grammatical, of the language being taught.

 

Examples include nouns, verbs, adjectives, statements, questions, subordinate clauses, and so on. Issue/Criticism: One problem facing the syllabus designer pursuing a grammatical order to sequencing input is that the ties connecting the structural items maybe rather feeble. A more fundamental criticism is that the grammatical syllabus focuses on only one aspect of language: grammar; however in truth there exist many more aspects to be considered in language. Recent corpus based research suggest there is a divergence between the grammar of the spoken and of the written language, raising implications for the grading of content in grammar-based syllabuses.

2. A NOTIONAL/FUNCTIONAL SYLLABUS   

The content of the language teaching is a collection of the functions that are performed when language is used, or of the notions that a language is used to express. Examples of the functions includes: informing, agreeing, apologizing, requesting; examples of notions includes age, size, color, comparison, time, and so on.  Issue/Criticism: In order to establish objectives, the needs of the learners will have to be analyzed by the various types of communication in which the learner has to confront. Consequently, needs analysis has an association with notional-functional syllabuses. Although needs analysis implies a focus on the learner, critics of this approach suggest that a new list has replaced the old one.Where once structural/situational items were used, a new list consisting of notions and functions has become the main focus in a syllabus. White (1988:77) claims that “language functions do not usually occur in isolation” and there are also difficulties in selecting and grading function and form. Clearly, the task of deciding whether a given function (i.e. persuading), is easier or more difficult than another (i.e. approving), makes the task harder to approach.

3. SITUATIONAL SYLLABUS  The content of the language teaching is a collection of real or imaginary situations in which language occurs or is used. A situation usually involves several participants who are engaged in some activity in a specific meeting.  The language occurring in the situation involves a number of functions, combined into a plausible segment of discourse.  The primary purpose of a situational language-teaching syllabus is to teach the language that occurs in the specific situations.  Examples of the situations include: seeing the dentist, complaining to the landlord, buying a book, meeting a new student, and so on.  

4. A SKILL-BASED SYLLABUS The content of the language teaching is a collection of specific abilities that may play a part using language.  Skills are things that people must be able to do to be competent in a language, relatively independent of the situation or setting in which the language use can occur.While the situational syllabi group functions together into specific settings of the language use, skill-based syllabi group linguistic competencies (pronunciations, vocabulary, grammar, and discourse) together into generalized types of behavior, such as listening to spoken

language for main idea, writing well-formed paragraphs, giving effective oral presentations, and so on.  The primary purpose of the skill-based instruction is to learn specific language skills. A possible secondary purpose is to develop more general competence in the language, learning only incidentally any information that may be available while applying the language skills. 5. A TASK-BASED SYLLABUS    



    

The content of the teaching is a series of complex and purposeful tasks that the student wants or need to perform with the language they are learning. The tasks are defined as activities with a purpose other than language learning, but, as in the content-based syllabus, the performance of the tasks is approached in a way intended to develop second language ability. Tasks integrate language (and other) skills in specific settings of the language. Task-based teaching differs from situation-based teaching in that while situational teaching has the goal of teaching the specific language content that occurs in the situation (pre- defined products), task-based teaching has the goal of teaching students to draw on resources to complete some piece of work (a process).The students draw on a variety of language forms, functions, and skills often in an individual and unpredictable way, in completing the tasks. Tasks can be used for language learning are, generally, tasks that the learners actually have to perform in real life. Examples include: Applying for a job, talking with a social worker, getting housing information over the telephone, and so on. 6. A CONTENT-BASED SYLLABUS The primary purpose of the instruction is to teach some content or information using the language that the students are also learning. The students are simultaneously language students and students of whatever content is being taught. The subject matter is primary, and the language learning occurs incidentally to the content learning. The content teaching is not organized around the language teaching, but vice-versa. Content-based language teaching is concerned with information, while task-based language teaching is concerned with communicative and cognitive processes. An example of content-based language teaching is a science taught in the language the students need or want to learn, possibly with linguistic adjustment to make science more comprehensible.

SYLLABUS DESIGNS Multi-syllabus Lexical Process

Many would have a primary and secondary organizing principle like: At the bank: question forms At a garage: imperatives At a hotel: present perfect FACTORSTO CONSIDER IN WRITING IMs (ORNSTEIN) UNDERSTANDING It requires matching the materials to the learners’ abilities and prior knowledge. If students do not understand the materials, frustration sets in, making learning more difficult. The teacher/writer must know whether the materials are suited to the level of the students and whether they will understand those.Thus, the teacher/writer must provide for background lessons and check-up activities and exercises to assess students’ understanding.This is especially important for younger and slower students and when introducing new concepts. STRUCTURING/ CLARIFYING It involves organizing the material so that it is clear to the students. It is especially important when new subject matter is introduced, and when it is being linked to the previous lessons.      

Directions, objectives, and main ideads are stated clearly. Internal and final summaries cover the content. Transition between main ideas is smooth and well integrated. Writing is not vague. Sufficient examples are provided. New terms are defined. Adequate practice and review assignments reinforce new learning.

SEQUENCING It refers to the arrangement of the materials to provide for continuous and cumulative learning where complex concepts are taken only after prerequisite skills and concepts have been mastered.There are four basic ways of sequencing a material:  Simple to complex  Parts to whole  Whole to parts  Chronological arrangements BALANCING Balancing materials require establishing vertical and horizontal balance or relationships. Vertical relationships refer to a building of content and experiences in the lesson, unit and course level. Fourth grade language concepts build on third grade concepts; the second unit plan builds on the first, etc. Horizontal relationships establish a multidisciplinary and unified view of different subjects; for example the content of the social studies course is related to English and Science.

EXPLAINING It refers to the way headings, terms, illustrations, and summary exercises are integrated with the content. Does the example illustrate major concepts? Are the major ideas identified in the chapter objectives and overview? Do the headings outline a logical development of content? Do the materials show relationships among topics, events, facts to present an in-depth view of major concepts? The students s hould be able to discover important concepts and information and relate new knowledge on their own through the materials. PACING It refers to how much and how quickly the lessons in the textbooks are presented. The volume or length of the materials should not overwhelm students, but there must be enough to have an effect. As students get older, the amount of materials can increase, the presentation can be longer and more complex and the breadth and depth can be expanded. REVIEWING It refers to the extent to which the material allows students to link new ideas to old concepts in the form of a review. High- achieving and older students can tolerate more rapid pacing than low-achieving and younger students, thus less proficient learners would need more review or linking than the more proficient ones. ELABORATING Elaborating ensures that students learn better through a variety of ways. The idea is to provide in the textbook opportunities for students to transform information to one form to another, and to apply new information to new knowledge – by using various techniques such as comparing and contrasting, drawing inferences, paraphrasing, summarizing and predicting. A series of elaboration strategies help students learn new materials. The author must provide students with a broad list of questions (of comparing and contrasting, drawing, analogies, etc.) TRANSFER OF LEARNING Transfer of learning may be done in a number of ways. Transfer of learning maybe conceptrelated, inquiry-related, learner or utilization-related. The first two organizers seem to work best with intrinsically motivated students and the second two best with students who needs to be extrinsically motivated. Since most students need some extrinsic motivation, learnerrelated and utilization-related materials will be more effective with majority of students.  Concept-related, drawing heavily on structure of knowledge, the concepts, principles, or the theories of the subject.  Inquiry-related, derived from critical thinking skills and procedures employed by learning theorists or scholars in the field.  Learner-related, related to the needs, interest or experiences of the students.  Utilization-related-show how people can use or proceed with them in real life situations.