PISA Tasks: Evaluation of the Foundation’s Portfolio

With the initiative and support of the Trump Foundation, leading developers in universities and education organizations are preparing an arsenal of new learning material for middle school students. The new material targets high ability students in mathematics who study in excellence classes, and its content is aligned with the advanced 5-6 proficiency levels of the PISA mathematics framework. So far, 19 development programs have been approved by the foundation, totaling 350 high order mathematics tasks. Each program developed its learning tasks in a different context and they are now deployed in professional development for teachers.

Since the nature of PISA tasks is new to the Israeli education community, a threshold question facing each developer is if and how they succeed in addressing key concepts and skills that are measured by PISA. Therefore, representative sample tasks from each project are routinely sent by the foundation team for review by Zbigniew Marciniak. Marciniak, a mathematician, is an international expert in this area and he serves as co-chair of PISA’s Mathematics Expert Group. He provides professional advice to our partners, including feedback and comments for improvement.

Marciniak is now proposing to conduct a comprehensive research study to provide an overall assessment. It will be similar to a study he carried out in Poland, together with Agnieszka Sulowska, master teacher and leader of the Polish mathematics expert group in the PISA test. The study in Poland conducted a performance analysis of Polish students based on their PISA results, and formed the basis for a national reform of the curriculum. Eventually it led to an impressive increase in Poland’s performance on PISA. Since the PISA research performance data of all countries is public domain, Marciniak and Sulowska could conduct such a similar analysis concerning Israel.

Their plan is to conduct an item-by-item analysis of Israeli students’ performance in relation to the average international performance. Each item will be briefly characterized and the distribution of students’ performance on the item will be presented in accordance with PISA’s six levels of mathematical proficiency. This will be followed by a conceptual analysis of each item based on the skills and knowledge it addressed, and unique challenges indicated by students’ performance.

The observations and insights will be combined into an integrative report that will include general conclusions and recommendations. In addition, Marciniak and Sulowska will review 2-3 sample tasks of each of the 19 programs in the foundation’s portfolio to determine how they correspond to concepts and proficiency levels set by the PISA framework. This evaluation will aim to explore if and how the portfolio addresses the challenges that were highlighted in the performance analysis, and identify gaps that need to be dealt with. A final report will be translated into Hebrew and shared with the professional community in Israel.