DEVELOPING PISA-LIKE PROBLEMS TO ASSESS STUDENTS’ MATHEMATICAL COMMUNICATION AND PROBLEM SOLVING ABILITIES

This research aims to produce PISA-like problems that are valid, practical, and effective for assessing students' mathematical communication and problem solving abilities. This research type is formative evaluation development by Tessmer, including expert review, one-to-one, small group, and field test. The subject of this research is 60 students of Year 9 at SMP Negeri 19 Percontohan Banda Aceh, Indonesia. The results of this study yielded 15 PISA-like problems that are: (a) valid because experts state that the set of questions developed is based on a strong theoretical basis, almost all problems have valid criteria, and the set of problems has high reliability; (b) practical because teachers agree that the problems can assess students' mathematical communication and problem solving abilities; (c) effective because mathematical communication and problem solving abilities of students meet PISA levels and students give positive responses to PISA-like problems. The implication of this research is that mathematics teachers can use this problem to familiarize students in answering PISA-like problems, primarily to assess mathematical communication and problem solving abilities.


INTRODUCTION
athematical communication and problem solving abilities are two of the five mathematical competency standards that are the goals of mathematics learning according to the National Council of Teachers of Mathematics (NCTM, 2000). Mathematical communication is the ability to write, read, listen, interpret, and evaluate ideas, symbols, terms, and mathematical information (Baroody, 1993) and use mathematical language to express mathematical ideas and arguments accurately, concisely, and logically (Kaur & Toh, 2012). Problem solving is a competency in applying concepts and skills in understanding, choosing solving strategies, solving problems (Zakaria, 2007), and solving non-routine mathematics problems (NCTM, 2000). According to De Lange (2006), abilities that will improve mathematical literacy skills include, among others, mathematical communication ability that expresses ideas in mathematical components in various ways. Mathematical communication is needed both orally and in writing. In addition, problem solving ability are also needed to solve various kinds of mathematical problems in different ways.
One of the international assessments that aim to determine students' mathematical communication and problem solving ability is PISA (Programme for International Student Assessment). PISA was developed by the Organization for Economic Corporation and Development (OECD). PISA aims to measure student literacy (maximum age 15 years) in reading, mathematics, and science (OECD, 2013). The aspects measured in PISA in the field of mathematics consist of three components relating to: (a) mathematics content, M (b) the process that students undertake when observing a symptom, linking these symptoms with mathematics, then solving the problem being observed, and (c) the situation or the context used in mathematical problems (Hayat & Yusuf, 2011).
Indonesia has participated since PISA was first implemented, namely in 2000. Wijaya (2012) states that although Indonesia has been involved from the beginning in organizing PISA, the results achieved by Indonesian students in PISA are far from satisfying. Based on Permendikbud No. 67 of 2016, external challenges faced by the Indonesian people regarding the development of education at the international level whose aim is to improve the rank of Indonesian students at the international level, namely TIMSS and PISA (Kemdikbud, 2016). According to Stacey (2011), almost 70% of Indonesian students are only able to complete PISA questions in 2009 up to level 2 for all topics, while many students from developed countries reach level 6.
Many factors certainly cause low PISA results. One of the contributing factors is that students are generally poorly trained in solving problems with characteristics such as PISA questions (Jurnaidi & Zulkardi, 2013;Wardani, 2011). This is evident from the results of the Mathematics Literation Contest (Kontes Literasi Matematika), a PISA-like problems contest in Indonesia, which shows that the ability of students to solve given problems is still low. Some students only pay attention to the pictures on the question without knowing what to do (Jurnaidi & Zulkardi, 2013). The low ability of students to solve PISA questions is also caused by students not getting used to solve these problems. This is because the PISA questions that teachers have are very limited. Therefore, efforts are needed to complete PISA-like problems in the Indonesian context.
Based on the explanation above, the research question in this study was how is the validity, practicality, and effectiveness of PISA-like problems to assess students' mathematical communication and problem solving abilities?

METHODS
The type of research used is development research (Tessmer, 1993). The criteria for developing PISA-like problems in this study are valid, practical, and have potential effects. This research was conducted in two stages, namely: 1) the preliminary stage, which includes preparation and designation, and; 2) the formative evaluation stage, which includes self-evaluation, expert reviews, and one-to-one, small groups and field tests.
The participants in this study were Year 9 students of one secondary school in Banda Aceh, Indonesia, which is SMPN 19 Percontohan. In the pilot phase, one-to-one students involved three people, namely two students from class IX-1 and one student from IX-4, the small group stage of students involved as many as six people, namely 2 students of class IX-1 and 4 class IX-4 students and field tests (field trials) selected test subjects IX-2 and IX-3 classes as many as 60 students. In each of these stages, the test subjects are different students.
The instruments in this study were validation sheets, practicality sheet, student responses, and mathematical communication and problem solving tests. Indicators of mathematical communication ability according to NCTM (2000) are: (a) communicating mathematical thoughts clearly; (b) use mathematical language to express ideas appropriately; (c) organize mathematical thinking through communication; and (d) analyze and evaluate mathematical thinking. Indicators of problem solving ability according to NCTM (2000) are: (a) applying and adapting various appropriate strategies to solve problems, (b) solving problems that arise in mathematics and other contexts, (c) forming new mathematical knowledge through problem solving, and (d) monitor and reflect on the mathematical problem solving process.
The data analysis technique was a descriptive analysis based on criteria of validity, practicality, and effectiveness effect. A set of PISA-like problems meets valid criteria if it meets the indicators modified from Nieveen (1999), namely at least three out of five experts state that the set of questions developed is based on a strong theoretical basis. Five experts gave assessments in this study representing the following provisions: (a) had researched on the development of PISA-like problem, (b) had attended a PISA question development workshop held by Puspendik, (c) lecturers who taught evaluation courses and understood PISA studies, and (d) had been a jury on the Mathematics Literation Contest held by department of mathematics education of Universitas Syiah Kuala. In addition, this study also tested the validity and reliability empirically. The calculation of the validity of the items is done by determining the productmoment correlation of Karl Person, while the Cronbach-Alpha formula uses the reliability of the questions. The analysis was carried out with the help of SPSS-22. According to Guilford (1956), an instrument has valid criteria if rcount ≥ rtable. Moreover, the instrument has high reliability if r11≥ 0,7.
A set of PISA-like problems meets practical criteria if they meet the indicators modified from Nieveen (1999), namely: (a) at least three of the five teachers consider that the developed questions are practical and (b) the level of use of the developed questions is included in the appropriate and very appropriate category. Teachers who assess the practicality of the questions are senior teachers who are experienced in developing and solving PISA questions.
A set of PISA-like problems meets effective criteria if they meet the indicators modified from Nieveen (1999), namely the questions have a potential effect which can be seen from (a) mathematical communication and problem solving abilities of students have the potential to meet level 1 to level 6, and (b) most students respond that they are interested and serious and wants the teacher to ask questions that are similar to the questions given by the researcher. According to Geskey ( Zulkardi, 2002), potential effects can be seen from the level of effectiveness of the development, including reactions given by subjects (participants or students) and the use of new knowledge and skills by the students.

RESULTS AND DISCUSSION
The following description explains the results of developing questions for each stage.

Preliminary Stage
Activities carried out at this stage are self-evaluation, including analyzing curriculum, formulating indicators of mathematical communication and problem solving abilities, analyzing PISA questions, and designing questions.
The designing stage consists of 8 problems to assess students' mathematical communication ability and 7 problems to assess students' problem solving ability, totally 15 items. The results of this designation are called prototype 1, as shown in table 1.

Prototyping Stage Expert Review Stage
At this stage, a set of PISA-like problems as prototype 1 was consulted with five experts. They gave comments or suggestions regarding the content, construct, and language, as shown in table 2. Language ▪ In question number 12 the question order so that students can solve the question is not yet clear ▪ In question number 8 the word "train" is changed to "motorcycle" ▪ In question number 9, the rupiah currency symbol is adjusted to the EYD ▪ Problem number 15 should be revised because many students do not know about calculations in Aceh culture.
Theory ▪ For PISA-like problems, there are some problem solving types that are still too easy, for example, problem number 7 is too easy for students to do, so the problem is not quite right to say problem solving ▪ In problem number 1, note the problem solving indicator again.

One-to-One Stage
A set of the PISA-like problems as prototype 1 was tested one-to-one on three students with different abilities to read the problems. Student 1 with high ability, student 2 with moderate ability, and student 3 with low ability. The three students were asked to work on 15 items, then each student was asked to provide comments and suggestions for the questions they had worked on. The comments and suggestions from students in the one-to-one stage can be seen in table 3. ▪ Don't understand the meaning of the phrase "waiting time cost adjusted" ▪ Can Ho-Bike transport ride by 2 people (pull three) 11 ▪ How do you mean the spacing of 9m x 9m and 9.2m x 9.2m ▪ It is better to write the spacing of 9m and 9.2m ▪ It is better to mention the shape of whether the isosceles triangle or the equilateral triangle or any triangle 12 ▪ The graph is unclear, meaning 0, 0.5, 1 etc. 13 ▪ The sentence on the information "7 matches left" makes me confused, whether each club has 7 matches left or 7 matches for all clubs ▪ Explain the information available 14 ▪ Many numbers are included that are not used to answer the question ▪ Deception was reduced Furthermore, based on comments or suggestions from expert reviews and one-to-one, revisions are made. The results of these revisions are called prototype 2.

Small-Group Stage
PISA-like problems as prototype 2 are tested on the small group of six students with different abilities and students who are different from the students in one-to-one stage. The researcher asked the students to answer the questions. The trial phase was carried out in 60 minutes, involving 60 students, and the questions were divided into two sets, namely set 1 of eight questions and set 2 of seven questions. During the implementation, the researcher supervises and observes students to see the difficulties that occur so as to give an indication of whether prototype 2 needs to be repaired or not. The six students were asked to provide comments/suggestions on PISA-like problems that had been developed, then the researchers revised prototype 2 based on the results of the small group stage trial based on comments/suggestions from students. In addition, the researchers also analyzed the items in prototype 2 to test the validity and reliability of the questions. The validity test of the items and the reliability of the questions was carried out on students of class IX-4 SMP Negeri 19 Percontohan Banda Aceh totaling 30 students. The item validity calculation is done by determining the product-moment correlation of Karl Pearson while the reliability of the problem is used the Cronbach-Alpha formula, as shown in table 4. Based on table 4, it was found that 1 out of 15 questions was invalid, namely question number 6 because rxy < rtable or 0,344 < 0,3610. However, question number 6 is still used because it is related to number 5. The reliability coefficient of the question is 0.819, meaning that PISA-like problems have high reliability.
Then the PISA-like problems are given to teachers to assess the practicality of problems. Table 5 presented the results of the assessment of five teachers about the practicality of the questions. Based on table 5, in general, the teachers chose to agree (A) and strongly agree (SA) for each item provided. All the teachers voted strongly agree that the display of the questions developed was clear and easy to understand by the teacher. Four teachers chose to strongly agree that the instructions for working on the questions were easy to understand, and one teacher chose to agree. Thus it can be concluded that PISA-like problems meet the practical criteria.

Field Test Stage
After obtaining a valid and practical prototype 3, the prototype was then trialed at the field test stage in class IX-2 and class IX-3 of SMP Negeri 19 Percontohan Banda Aceh with 60 students. The questions consist of four contents: quantity, change and relationship, space and shape, uncertainty and data. The questions consist of 8 problems of problem solving ability and 7 problems of mathematical communication ability.
The field test phase aims to determine students' ability to solve mathematical communication and problem solving problems and students' response. The ability of students to solve these problems is seen according to the scores of students solving PISA-like problems. The score is converted into a PISA level score, as shown in table 6. Based on table 6, it was found that in terms of mathematical communication ability, most of students were below level 1 (41 out of 60 students) and in term of problem solving ability most of students are below level 1 (30 out of 60 students). Scores obtained by students at this level are less than 358. However, some students are able to reach a higher level, even level 5 and 6. Many students who reach level 5 and 6 in problem solving problems are more than mathematical communication problems. Figure 1 shows mathematical communication problem (number 10), occupational context and change, and relationship content. This problem requires students to be able to understand the purpose of the problem by applying an understanding of straight-line equations through a problem. Based on the results of data analysis, it was found that only 4% of students were able to answer correctly, 40% were able to answer, but the answer was not correct. Students generally know the initial steps in solving the problem, namely by determining the gradient then substituting into the equation of the line through a point and can determine the value of constants. However, students have difficulty in represent the problem in a mathematical sentence. This shows that students are less accustomed to solving mathematical problems that require communication ability, where students must use mathematical language to express ideas appropriately. The students' answers to problem number 10 are shown in figure 2.

Figure 2. Student Answers at the Field Test Stage for Question Number 10
Based on students' written answers, it was concluded that students met level 1 to level 6. Furthermore, based on the response questionnaire given to students, the percentage of students' responses was obtained, as shown in table 7. Every time students solve mathematics problems, students want to answer with the same solving process as the questions given by the researcher

55%
The results of the student response questionnaire in table 7 show that there are some students who are happy in solving PISA-like problems because they say that these questions are not only challenging but also stimulate ways of thinking, but also add new knowledge about their environment. However, there were also some students who were not happy because the PISA-like problems that the researcher gave were difficult to understand and difficult to solve. Overall, the questions developed can stimulate students' thinking and explore students' potential to solve mathematics problems.
The explanation above shows that the development of PISA-like problems to assess students' mathematical communication and problem solving abilities achieved valid, practical, and effective criteria. Validation is obtained from the experts' evaluation, which almost all experts state both in terms of content, construct, and language are valid so that the set of questions developed is based on a strong theoretical basis. Then, the empirical validity that 14 out of 15 questions was valid and the invalid question is still used because it is related to another number, and the reliability coefficient of the questions is high.
Practical results are obtained from teachers who agree that the results of the development of PISA are to assess students' mathematical communication and problem solving abilities. The effectiveness of the questions is obtained because 1) mathematical communication and problem solving abilities of students meet level 1 to level 6, and 2) most students respond that they are interested and serious and have the desire to solve PISA-like problems. This is as stated by Amalia, Rusdi, and Kamid (2021) that the seriousness and response of students to PISA-like problems as well as the potential for mathematical abilities that may be raised by students during the problem solving process are potential effects of the problem.
The results of this study are in line with previous studies. Amalia, Rusdi, and Kamid (2021) have produced valid HOTS PISA-like mathematics problems through validator assessment results, practical where students can use the set of questions well and are effective because they can measure students' mathematical literacy skills. Putra, Zulkardi, and Hartono's research (2016a) does measure not only theoretically valid but also valid criteria, namely based on item analysis, while practically all students can understand the questions well, and are effective as seen from the questions developed has a potential effect on students' mathematical literacy skills. In addition, potential effects can also be seen from questions involving various basic mathematical abilities in the completion process (Charmila, Zulkardi, & Darmawijoyo, 2016).
Based on table 2, it can be concluded that overall the average ability of students to solve PISA-like problems to assess mathematical communication and problem solving abilities can only reach level 2. The results of this study are in line with research by Stacey (2011), stating that almost 70% of Indonesian students are only able to complete PISA questions up to level 2 for all topics and Widjaja (2011) stating that nearly 80% of Indonesian students are only able to answer problems given at level 2, almost no Indonesian students can solve problems for level 6. This shows that students are only able to solve problems that use basic algorithms, use formulas, carry out simple procedures. In other words, students can only interpret problems and solve them with formulas. However, students are still weak in solving problems using mathematical communication and problem solving abilities. Wardani (2011) states that many weaknesses in Indonesian students' mathematical abilities were revealed in PISA learning outcomes. In general, the weaknesses of Indonesian students are not accustomed to reading while thinking and working so they can understand essential information and determine strategies for solving problems.
The findings of this study include students having difficulty in making mathematical sentences of a problem. This is due, among other things, to the daily learning process students are not accustomed to solving problems by making mathematical sentences from a problem. According to Jurnaidi and Zulkardi (2013), one of the contributing factors is that students are generally poorly trained in solving problems with characteristics such as questions on PISA.

CONCLUSION
The development of PISA-like problems to assess students' mathematical communication and problem solving abilities has achieved valid, practical, and effective criteria. Validation is obtained from the results of the experts' evaluation which three out of five experts state that the set of questions developed is based on a strong theoretical basis, the validity test of the items that almost all problem is valid, and the reliability of the questions is high. Practical results are obtained from teachers who agree that the results of the development of PISA-like problems can assess students' mathematical communication and problem solving abilities. The effectiveness of the questions is obtained because 1) mathematical communication and problem solving abilities of students meet level 1 to level 6, and 2) students give positive responses (interested, serious, have the desire to solve) to PISA-like problems.
Based on the results of the research and the conclusions that the researchers have put forward, the researchers suggest that students and mathematics teachers can use PISA-like problems to explore mathematical abilities, especially mathematical communication and problem solving abilities. For other researchers, it is hoped that this series of questions can be used as input to further study the development of PISA-like problems in future research.