Examining the Readability Level of Reading Texts in English Textbook for Indonesian Senior High School

Assuming that textbooks authors have conducted the readability measure, many of its users often take this key feature of a good textbook for granted. Nonetheless, considering the unique context of each classroom and the generic nature of textbooks written for the public, teachers should double-check the appropriateness of the book they use, including its readability. Although tech-assisted readability measurement is developing in many parts of the world, in Indonesia, this area is still under-researched. For this reason, the present study attempts to examine the readability level of an English textbook using Coh-Metrix. Content analysis is employed since the object being analyzed is a textbook. This study also expands previous studies on readability measures by building dialogue between the result of Coh-Metrix measurement and the students’ perceived readability. To obtain the data, an automated Coh-Metrix readability measurement was conducted via cohmetrix.com. Also, to get more profound analysis, a questionnaire on students’ perceived readability was distributed to 35 tenth graders. Then the collected data were analyzed by using interactive model of Miles and Huberman. The finding of this study revealed that regardless of the slight difference between the perceived readability level and the Coh-Metrix-generated readability level, most of the texts in the book are mostly below the students’ level. Arguably, the textbook is relatively potential for language acquisition because it provides comprehensible input.


INTRODUCTION
Textbooks are one of the most important things in education because they act as teaching materials or instructional media during teaching and learning activities. Also, the students' ability to follow the textbooks is crucial because it is related to the success of learning objectives. Thus, the readability of the textbooks should be reviewed and analyzed from various perspective. It is commonly believed that the textbook is the core of every circumstance in the ELT context (Hutchinson & Torres, 1994;Sheldon, 1988). The textbook will act as a point of reference for teachers to track their success in teaching, as well as helping them to offer a concentration on teaching process (Tomlinson, 2008). Similarly, Hutchinson & Torres (1994) claimed that textbooks could provide a fundamental basis for delivering a lesson. Besides, the textbook may be used by the learners as a resource to study what they have learned previously, at the same time, they become familiar with the new things that will be taught shortly (Wong, 2011). Following the students' aid, the best way to achieve objectives and goals is to consider the textbook as a source for student needs (Cunningsworth, 1995). Furthermore, Tomlinson (1998) characterized a good material as fulfilling 16 criteria; among others, a good material should help students to feel comfortable, develop their confidence, and learners will perceive it as relevant and useful. These criteria apply to textbooks as well.
The studies of readability have been conducted by the experts and librarians who have attempted to discover a way of "putting the right book into the hands of the right reader" (e.g., Pitler & Nenkova, 2008;Crossley, Allen & McNamara;Amalia, 2016). Because education is generally about connecting a human to a book, the book should be suitable for the readers in every level of education (Dale & Chall, 1949). Besides, some of the private publishers have produced and published an English textbook for schools. Assuming that the publisher assures the overall quality of the book including its readability, most teachers take the books for granted.
However, it is crucial to examine the readability level of those books closely.
A textbook is a widely used resource in language teaching and learning process (Brown, 2000). Sheldon (1988) defines a textbook as a published manuscript, mainly developed to help language learners developing language and communication skills. They tend to appear in series according to a certain level, and it is designed with bright coverings, colored visuals, and pictures that are incredibly eye-catching (Masuhara & Tomlinson, 2008). Similarly, the textbook is a manuscript written by professionals and experts in a particular field of study, and the materials used in it are typically carefully checked before publication in pilot studies in real teaching situations (Cunningsworth & Tomlinson, 1984). In brief, a textbook is the most common material support for language learning that is specially designed to help language learners to improve their skills and it is carefully tested in real teaching situations before publication. Furthermore, textbooks are significant because they hold several roles in ELT as identified by Cunningsworth (1995) that textbook can act as a resource for spoken or written presentation material, the medium of learning and communicative interaction practices, point of reference, syllabus, self-directed learning tools or self-access practice, and less-experienced teachers' assistance. Sheldon (1988) also explained three reasons support the widespread of textbooks around the world. First, textbooks are essential in ELT contexts because of the teachers' difficulties related to the development of their teaching material.
Furthermore, textbooks reduce the preparation time with ready-made instructional texts and learning assignments. Lastly, textbooks can act both as a syllabus and as a guide for classroom development, a measure that allows the external stakeholders to assess the teaching and learning process.
Readability is one of the important aspects that should be considered in designing and evaluating a textbook because most of its contents are in the form of written text as Dubay (2004) said that readability is something that influences the difficulties of comprehension passages. In the past, several readability calculators for evaluating the readability of a reading text were developed, such as the Dale-Chall formula, SMOG Grading, Fry Graph formula, Flesch Reading Ease formula, and Flesch-Kincaid Grade Level. In this study, however, the researcher uses the concepts of the Flesch Reading Ease formula (Flesch, 1948) as the guidance to measure the readability level of the reading texts in English language textbooks because many researchers have widely used them. The formulas will be automatically calculated by the Coh-Metrix web tool (Graesser et al., 2004).
The Flesch Reading Ease scale is from 0 to 100, with a higher score indicates more understandable the reading text. Because this thesis will focus on the 10 thgrade students, the score between 50 and 60 is considered relatively appropriate and standard because the approximate grade level is from 10th to 12th.
Furthermore, in Flesch's book entitled The Art of Readable Writing (Flesch, 1949), he designed a table that provides readability score index. It is helpful to measure the readability level of a reading text and becomes guidance for all researchers who measure the readability level of a reading text by using the Flesch formula. The table readability score index is presented in table 1.  Crossley, Greenfield, &Mcnamara (2008) andGupta (2014). In the study by Crossley, Greenfield, & Mcnamara (2008), cohesion and text problems were evaluated at different stages of language discourse and theoretical evaluation, while Gupta (2014), examined the readability of English textbooks implemented in the English-medium schools utilizing Coh-Metrix. Moreover, conventional readability formulas, for example, Flesch Reading Ease (Flesch, 1949) and Flesch-Kincaid Reading Ease (Kincaid et al., 1975), calculated the difficulty of the written text based on the length of word and sentence, while Coh-Metrix L2 Readability is based on cohesion relations, world knowledge, and features of language and communication.
Although tech-assisted readability measurement is developing in many parts of the world, in Indonesia, this area is still under-researched. Three earlier studies by (Hidayat, 2016), Tasaufy (2017), and Miftaahurrahmi et al. (2017) that focused on English textbooks used by Senior High School were still conducted using manual readability method. This study, on the other hand, attempts to highlight the English textbook that is published by one of private publishers in Indonesia for Senior High School by using Coh-Metrix. This study also expands previous studies on readability measures by building dialogue between readability resulted from the Coh-Metrix and the students' perceived readability.

RESEARCH METHODOLOGY
This study is a content analysis since it aims to evaluate an English textbook According to Ary, Jacobs, and Sorensen (2010), content analysis is a research methodology applied to printed or graphical resources to define the specific features of the material. Moreover, to get more profound analysis, 35 students who were recruited as participants were students of a Senior High School in Karanganyar Regency. They were in the tenth grade and majoring in science. The researcher chooses them because they are the user of a textbook. The data from the students are the difficulty of each text related to the result of Coh-Metrix, which text is the easiest and most difficult, and what aspects that make the text is easy or difficult.
They were used to compare if the result of Coh-Metrix and the user of the textbook are the same or not.
This study attempts to examine the readability level of an English textbook automatically by using Coh-Metrix and also expands the previous studies on readability measures by building dialogue between the result of Coh-Metrix measurement and the students' perceived readability. To make general research problems more specific and easily handled, this research attempts to answer the following questions: 1. What is the readability level of reading texts in the English textbook entitled English on Target for grade X published by a private publisher in Indonesia? 2. What is the students' perception of the readability of reading texts in the English textbook English on Target for grade X published by a private publisher in Indonesia?
There were two kinds of data in this study: qualitative and quantitative. The raw data were texts (words) and the numerical data were generated from the readability measurements. Additionally, students' responses in the questionnaire were words that were later quantified to draw tendency. While the data were mostly quantitative, this study also aims to get a more in-depth understanding. Concisely, the researcher wants to know and measure the readability of the textbook by studying this document. Since it analyzed the existing book and the students' perspective as a data source, the result of this research was a descriptive analysis of the readability score and the result of the questionnaire.
The researcher used an interactive model proposed by Miles and Huberman (1994). It consists of three steps that were explained below:

Data Condensation
Data condensation was the process of selection, simplification, and transformation of the data to the field notes. In this research, data reduction was conducted by using purposive sampling, which is selecting the reading texts that are suitable with the criteria of Coh-Metrix inputs. The researcher selected the texts that were in the form of monologues and having more than 200 words and below 15.000 words as limited by Coh-Metrix.

Data Display
Data display is a set of information that has been classified and organized based on data reduction, which leads to the conclusion. In this article, the data display was conducted by displaying the readability level of each passage and then adding the data from the students to know their perception on the readability level.

Drawing Conclusion
The last step was drawing the conclusion. It is an analytic tactic for generating meaning from data and for testing or confirming findings. In this study, the researcher explained the result of Coh-Metrix and then checking the confirmation from the result of the questionnaire. Additionally, the questionnaire was designed based on the Coh-Metrix result. In conclusion, collecting the data from the students was aimed to deepen judgment by comparing the result of Coh-Metrix and the result of the questionnaire so that the judgment is not one-sided.
The trustworthiness of this study is convinced by the data and method triangulation. This study used a different source of data, textbook and the students, and method, document analysis and questionnaire. Besides, validity and reliability of the data is one of the important aspects in a study. Therefore, this study employs SPSS version 25 to make the data from the questionnaire valid and reliable. For the validity, the researcher uses Pearson correlation and for the reliability, the researcher uses Cronbach's Alpha.

Text Selection
A total of 67 texts are available in the textbook, 1 dialogue, and 66 monologues. The texts taken for this study are in terms of monologue. The number of texts selected for analysis is 19. The selection was based on the length compatibility of Coh-Metrix. It can only provide an analysis of the text, which has more than 200 words but less than 15.000 words. To shorten and simplify the analysis, the first to the last text is sorted and marked using a code, for example text 1 becomes T1, text 2 becomes T2 and so on.

Students' Perception and Its Relation to the Coh-Metrix Result
The findings related to students' perception show that there was a gap between the result of Coh-Metrix readability measures and the students' perception of the same texts' readability. Also, most of the students think that having sufficient vocabulary helps their performance in reading. The findings of another perspective that comes from the students who is the user of the textbook is used to deepen the analysis. The data were obtained by using an online questionnaire. For the try out questionnaire, it was distributed to one class of tenth-grade students, and the number of students is 34. Moreover, the questionnaire is divided into two parts. The first is the difficulty scale of sampling texts, and the second is the reason why they think the texts are easy or difficult. After the data is obtained, the researcher checks the reliability of the data by using Cronbach's Alpha in SPSS version 25, and the result is 0.781. According to Cortina (1993), Cronbach's Alpha can be rather high and acceptable by the standards of many (greater than 0.70). Based on that statement, it means that the questionnaire is reliable. Additionally, all of the items in the questionnaire are valid since the Pearson correlation score of each item is higher than r for product-moment, with N=34, and the significance level is 5%. The SPSS result of the Pearson correlation is attached in table 5. The researcher has two ways to measure the readability level. The first is by using Coh-Metrix. And the second is using the questionnaire. After obtaining the result from Coh-Metrix, 19 texts were classified into six levels.
1. Very easy. Two texts classified into this level. The texts are on the page 105-106 (T17) and 108 (T19). It is estimated for 2 nd -grade students. According to those classifications, it can be concluded that from 19 texts, there was only one text (5.26%) that suitable for the level of students; that is the text on page 45-46 (T1). 14 (73.68%) texts were below the students' grade level. And 4 (21.05%) texts were above the students' grade level. On average, the texts on the textbook were at the 'standard' level, specifically in 7 th grade. It means that there was only 5.26% suitable text, and 94.74% is not suitable for 10 th -grade students.
These findings are similar to the ones found by Owu-Ewie (2015) in his study on English textbook readability in Ghana. He found that many of the texts were inappropriate with the level of the students, which was above their level. Another case was also found that three textbooks being analyzed were generally between 'fairly difficult' and 'difficult' level. This may have contributed to the low readability of the textbooks for diploma students (Gyasi & Slippe, 2019). Moreover, the study conducted by Tasaufy (2017), have also found similar cases that most of the texts are not suitable for the level of the students. From nine texts, there were only three texts that are suitable for the grade level of the students. Hidayat (2016)also found similar cases. Based on his study findings, the closest result to the readability level was only one text. The text has a fairly difficult level. In this case, the descriptive text entitled "Bullying: A Cancer that Must be Eradicated" predicted suitable for the tenth to twelve grade students. Last, the study conducted by Miftaahurrahmi et al.
(2017) discovered the same result. They found that from the ten samples of texts, there is only one text that is suitable for the grade level of the students.
Another point of discussion is built based on the comparison of the three readability measurements employed in this study: Flesch Reading Ease, Flesch-Kincaid Grade Level, and Coh-Metrix L2 Readability.  which, based on Coh-Metrix measures, were categorized below the students' level were perceived as easy and difficult by the students. Similarly, the texts which were categorized as suitable by Coh-Metrix were perceived as fairly easy and fairly difficult also by the students. Meanwhile, the texts which were categorized above the students' level by Coh-Metrix, were perceived as fairly easy by the students.
Overall, most of the texts were perceived as fairly easy by the students.
To explain the gap, this study dug into the reason why the students think that the texts were difficult or easy to understand. Most of the students (79.4%) feel that lack of vocabulary is the biggest problem in comprehending a text. On the contrary, most of them think that having sufficient vocabulary helps their performance in reading. It is in line with the statement of Day (1994) who said that it is clear that as the number of unknown lexical items in a reading passage increases, the more difficult it is for students to read it with comprehension. Also, Sidek & Rahim (2015) provide evidence that a reader's level of vocabulary knowledge is one of the elements that play an impacting role in determining reading comprehension performance in the EFL context. Moreover, Nation & Coady (1988) state that vocabulary difficulty is estimated in various ways; the most usual is word frequency and/or familiarity and word length. That is, sentences are more readable if they contain words that are of high frequency in occurrence and that are shorter rather than longer.
Overall, with 14 texts out of 19 texts that categorized below 10 th -grade students, this book is too easy for them based on the Coh-Metrix result. Moreover, it is in line with the students' perceived readability that most of them feel the reading texts were easy to read; the result of the questionnaire proves it. From 9 sampling texts, only two texts that the students feel fairly difficult. In conclusion, this book is below the standard of readable texts for 10 th -grade students, both by the result of the Coh-Metrix and questionnaire. Possibly, the textbook is relatively potential for language acquisition because it provides comprehensible input as Krashen (1995) explained on his theory. Likewise, for students with lower intermediate English proficiency, reading lower-level simplified texts was more helpful to the development of fluency as compared to reading higher-level simplified texts. It is based on Beglar and Hunt (2014) research that explored the effects of text type and text level in ER program on the reading fluency development on a Japanese college over one academic year.

CONCLUSION AND SUGGESTION
The findings show that reading texts in this book are below the standard of readable texts or too easy for 10th-grade students, both by the result of the Coh-Metrix and questionnaire. Additionally, there are some levels related to the classification of the reading texts. All of 19 texts were classified into very easy, easy, fairly easy, standard, fairly difficult, and difficult. From 19 texts, there was only one text that suitable for the level of students. 14 texts were below the students' grade level. And 4 texts were above the students' grade level. In short, the grade level based on the result of Coh-Metrix was mostly below the suitable grade level for tenth-grade students.
It is in line with the students' perception that most of them feel that the overall reading texts are easy. Besides, most of the students feel that having sufficient vocabulary helps their performance in reading and vice versa.
In conclusion, this study confirms that the reading texts in the EFL textbook entitled English on Target for tenth-grade students published by a private publisher in Indonesia are mostly below the students' level, meaning that it is too easy for them. Similarly, the students themselves feel that most of the reading texts are easy for them.
By knowing this, the teachers are expected to plan and determine the appropriate reading materials for the students. Textbook authors should also be able to organize and present the level of the reading texts from the easiest one to the difficult one. Thus, the students enjoy reading and receive every level of reading text as easy, and, finally, language acquisition works well.