Interventions, Practices and Contextual Factors Linked to Indonesian Students’ Literacy and Numeracy Outcomes: A Systematic Review

Interventions, Practices and 

Contextual Factors Linked to 

Indonesian Students’ Literacy and 

Numeracy Outcomes: 

A Systematic Review 

August 2017

INOVASI - Innovation for Indonesia’s School Children 

Ratu Plaza Office Tower 19th Floor, 

Jl. Jend. Sudirman Kav 9, Jakarta Pusat, 10270 

Indonesia 

Tel : (+6221) 720 6616 ext. 304 

Fax : (+6221) 720 6616 

http://www.inovasi.or.id 

Published in July 2017 

Cover photo courtesy by Palladium 

Pemerintah Australia menjalin kemitraan dengan Pemerintah Indonesia melalui INOVASI (Inovasi untuk 

Anak Sekolah Indonesia), program pendidikan yang bertujuan untuk menemukan berbagai cara untuk 

meningkatkan kemampuan membaca dan berhitung siswa Indonesia. INOVASI dimulai pada bulan Januari 

2016 dan akan berlangsung hingga tahun 2019. Program ini merupakan program tingkat daerah yang didanai 

oleh Pemerintah Australia (AUD 49juta) yang bermitra dengan Kementerian Pendidikan dan Kebudayaan 

RI. Fokus INOVASI adalah memahami dan mengatasi tantangan pembelajaran di kelas, khususnya yang 

berkaitan dengan kemampuan membaca dan berhitung. Nusa Tenggara Barat (NTB) adalah provinsi mitra 

pertama INOVASI dan mulai tahun 2017 akan ada dua provinsi mitra baru. 

INOVASI dikelola oleh Palladium atas nama Pemerintah Australia 

info@inovasi.or.id www.inovasi.or.id www.facebook.com/InovasiPendidikanAIP

Interventions, Practices and Contextual 

Factors Linked to Indonesian Students’ 

Literacy and Numeracy Outcomes: 

A Systematic Review 

Rarasati, Niken; Nugroho, Dita; Suryadarma, Daniel; 

Kurniawati, Sandra; Akrom, Muhammad; Artha, Rima P.; 

Wanita, Cici T.; Power, Lorna 

INOVASI (Innovation for Indonesia’s School Children) 

Australia-Indonesia Partnership 

Fujianita, Sisca; Sabon, Simon S.; Biantoro, Sugih; Fajarini, 

Catur D.; Ulumuddin, Ihya 

Center for Policy Research on Education and Culture (Puslitjakdikbud) 

Indonesian Ministry of Education and Culture

ii

Contents 

Tables & Figures 

v 

Abstract 6 

Introduction 7 

Focus of the Review 9 

Basic Education in Indonesia 10 

Learning Factors: Interventions, Practices, and Contextual Factors 10 

Learning Outcomes: Literacy and Numeracy 11 

Review questions 12 

Method 13 

Study Selection Criteria 13 

Search 14 

Abstract and Full Text Screening 15 

In-depth review 15 

Assessing quality of studies 16 

Meta-analysis 19 

Discussion 28 

What works to improve students’ literacy and numeracy outcomes? 28 

Intervention, Practices, and Contextual Factors 30 

References 34 

iii

iv

Tables & Figures 

Figure 1 Review Concepts 9 

Figure 2 Systematic map; searching, screening, and quality appraisal 18 

Figure 3 Summary of interaction among interventions, practices, and contextual factors 31 

Table 1 What this systematic review is (not) about 14 

Table 2 Cohen’s Effect Size Index 19 

Table 3 High quality findings 20 

Table 4 Overall effect size for numeracy from high quality studies 21 

Table 5 Overall effect size for literacy from high quality studies 21 

Table 6 Student effect to numeracy outcome 23 

Table 7 Student effect to literacy outome 23 

Table 8 Teacher effect to numeracy outcomes 24 

Table 9 Teacher effect to literacy 24 

Table 10 Teaching effect to numeracy 24 

Table 11 Teaching effect to literacy outcomes 25 

Table 12 School effect on numeracy outcomes 25 

Table 13 School effect on literacy outcomes 26 

Table 14 Home effect to numeracy outcomes 26 

Table 15 Home effect to literacy outcomes 26 

Table 16 Other factors linked to numeracy outcomes 27 

Table 17 Other factors linked to literacy outcomes 27 

v

Abstract 

Despite the large number of reforms that have been implemented in Indonesia to improve education 

quality, there is not yet a systematic review that maps how these reforms and programs affect the 

quality of learning in Indonesia. This systematic review aims to understand the different studies 

have been undertaken in Indonesia so far, and how they link to student literacy and numeracy 

outcomes. The review team screened 2,693 potentially relevant studies that were obtained from a 

systematic search. The screening process selected 120 relevant studies. Based on the availability 

of data provided in the paper as well as the quality of the individual studies, only 25 individual 

studies (reporting 75 findings) are suitable for a meta-analysis. Several factors have been mapped 

based on the magnitude and direction of the effect to the student literacy or numeracy performance. 

Comparing pooled effect sizes between types of interventions, we can see that teaching and school 

factors have the highest pooled effect on both literacy and numeracy learning outcomes. Amongst 

school factors, principal preparedness to implement school-based management contributed the most 

to the numeracy learning outcomes. There is strong evidence that using teaching strategies based 

on students’ needs and difficulties can improve learning outcomes. This review also suggest that 

teachers should have sufficient comprehension of the subject matter, pedagogical knowledge and 

adequate social skills to understand the problems in the classroom, and to choose and implement 

the most appropriate teaching strategy. Based on the quality assessment employed for all screened 

individual studies, the current review also strongly suggests that more rigorous studies on learning 

quality needs to be undertaken in Indonesia. 

Keywords: learning outcomes, Indonesia, literacy, numeracy, systematic review, meta-analysis 

6 

Interventions, Practices and Contextual Factors Linked to Indonesian Students’ Literacy and Numeracy Outcomes:

Introduction 

International tests of literacy and numeracy consistently place Indonesia among the poorest 

performers. While Indonesia was one of only a few countries showing improvements in its scores on 

the Programme for International Student Assessment (PISA) between 2000 and 2012 (OECD, 2014), 

it is still one of the lowest performing countries overall, and also compared to other participating 

countries in its region. Performance on such international learning assessments has been linked 

not just to the outcomes of individual students as adults (Breakspear, 2012) but also to countries’ 

economic growth (Hanushek & Woessmann, 2009). For these reasons, improving student learning 

outcomes have increasingly become a priority in Indonesia. 

The Indonesian government has significantly increased its public spending on education. Public 

education expenditure more than doubled in real terms between 2001 and 2009 (Chang et al., 2014). 

More attention is now being paid to whether this dramatic increase has had a positive effect on the 

outcomes and if it has not, what has gone wrong with this programme (van Trotsenburg, Chaves, 

Costin, Patinos, & Ragatz, 2015). At the same time, a number of education reforms, programs and 

interventions of varying scope have been implemented over the last two decades (OECD, 2015). 

While some of these have been found to have influenced educational attainment, the existing evidence 

shows very little or no returns on students’ learning outcomes (de Ree, Muralidharan, Pradhan, & 

Rogers, 2015; OECD, 2015). In response to this situation, the Indonesian Ministry of Education 

and Culture (MoEC) set the 2015-19 strategic plan to focus on achieving a better quality of learning 

(Kementrian Pendidikan dan Kebudayaan, 2015). MoEC’s Policy Research Centre (Puslitjakdikbud) 

plays a substantial role by providing comprehensive research evidence for policy making. 

Policymakers and practitioners face a constant barrage of news and stories on new strategies, 

programs and products that can improve student learning. They often do not have the time to assess 

the evidence that underlies such claims or which intervention or strategy would actually help them 

improve student learning in their particular context. One of the greatest problems faced by schools 

is not resistance to innovation, but the “fragmentation, overload, and incoherence resulting from the 

uncritical and uncoordinated acceptance of too many different innovations” (Fullan, 1991, p.97). 

This systematic review aims to provide a picture of the body of research on what factors contribute 

to the improvement of student learning outcomes in Indonesia. The findings will contribute to future 

research, interventions, and to provide an evidence base to assess whether particular types of 

interventions are likely to improve literacy and numeracy outcomes. It will also be used as a basis of 

engagement with education stakeholders; a starting point for discussions around the interventions, 

practices and factors that are most relevant to student learning. 

Thousands of meta-analysis have been conducted in the field of education. Most prominently, John 

Hattie has published a synthesis of over 800 of these studies – a meta-meta-study – that focus 

on student achievement (Hattie, 2009). The findings have highlighted the importance of student 

self-expectations, the provision of formative assessments and feedback, classroom discussion, 

and teacher-student relationships, as well as specific approaches such as response to intervention, 

7 

A Systematic Review

eciprocal teaching and spaced practice. As most of the underlying studies were carried out in 

developed and English-speaking countries, therefore, Hattie has cautioned against generalizing his 

findings in other contexts. 

More recently, academics and development partner researchers have undertaken systematic 

reviews (e.g. Asim, Chase, Dar, & Schmillen, 2015; Krishnaratne, White, & Carpenter, 2013; Masino 

& Niño-Zarazúa, 2015; Mcewan et al., 2013)as compared with \” business-as-usual \” in the same 

settings. The treatments included instructional interventions, health interventions, and incentivebased 

interventions. On average, monetary grants and deworming had effects that were close to 

zero and statistically insignificant. Nutritional treatments, treatments that provided information to 

parents or students, and treatments that improved school management and supervision had small 

mean effect sizes (0.04-0.06 on learning outcomes in developing countries, and in 2015 the World 

Bank published an analysis of six of such reviews. They found that three categories of programs 

were recommended with consistency: pedagogical interventions that tailor teaching to student 

skills (including those that use computer-assisted learning), repeated teacher training interventions 

that are usually linked to other pedagogical interventions, and in certain contexts, strengthening 

accountability through contracts or performance incentives (Evans & Popova, 2015). 

The World Bank analysis and underlying reviews largely considered studies that are experimental 

or quasi-experimental in nature. This means that the studies they considered either allocated 

interventions randomly or utilised another study design or statistical method to approximate a random 

allocation of a specific intervention. Meanwhile, the Hattie meta-review also incorporated studies that 

were observational in nature, looking at the practices and factors that influence learning outcomes 

more broadly rather than just the introduction of specific interventions. 

Few Indonesian studies were included in the above reviews. The 11 primary studies within the reviews 

included in the World Bank’s report that were conducted in Indonesia examined the effects of access 

improvement, nutritional supplementation, school governance and resources, including teacher 

characteristics. A review that includes a broader range of study designs – including observational 

studies- and specifically targets Indonesian context will be able to complement and compare those 

findings with the effects of classroom teaching and school practices. In addition to that, this review 

also included studies undertaken in specific contexts, such as low SES (Socio-Economic Status) 

community, children with low exposure to the Indonesian language, as well as specific ethnic groups 

in Indonesia. A narrative review as the supplementary analysis to the meta-analysis will explain how 

each intervention interact with the context. 

8 


Focus of the Review 

The population of interest for this review covers students in the Indonesian formal education system 

at the primary and junior secondary levels. The review covers public and private schools that are 

overseen by MoEC, as well as public and private madrasahs (Islamic school) that are overseen by 

the Ministry of Religious Affairs. This sub-section discusses the definitions and any possible issues 

around the three key concepts for this review: learning outcomes, basic education and learning 

factors (illustrated in Figure 1 below). The study focuses on the overlap between these three areas, 

marked X in Figure 1. 

Figure 1: Review Concepts 

• Interventions 

• Current teachinglearning 

practices 

• Context 

• Literacy 

• Numeracy 

• Primary (SD/MI) 

• Junior Secondary (SMP/MTs) 

9 


BASIC EDUCATION IN INDONESIA 

Indonesia has a 6-3-3 school-based education system with six years of primary, three years of 

junior secondary, and three years of senior secondary (general or vocational) school. Children enter 

primary school at the age of 6-7 and complete high school at around the age of 18-19. In line with 

INOVASI’s support of the GOI’s 9-year compulsory basic education program, this review will focus 

only on studies of primary and junior secondary students. 

Two Ministries oversee formal schools in Indonesia. While most schools fall under the purview of 

MoEC, the Ministry of Religious Affairs is responsible for the Islamic school sector. The schools in 

this latter sector fall mainly into two categories: madrasah, which forms part of the formal education 

system, and the more traditional pesantren, which do not. Alongside delivering most of the general 

curricular content set by MoEC, close to a third of the curriculum in madrasah institutions covers 

religious content set by MoRA. The madrasah system enrols approximately 20% of primary and 

junior secondary students within the formal education system (MoRA, 2013). Both the general and 

madrasah schools sector have public and private schools. However, around 90% of madrasah are 

private compared to only around 15% of schools in the general sector (MoEC, 2013; MoRA, 2013). 

This review will include studies involving students in both private and public institutions in: 

• Primary schools, both SD (Sekolah Dasar, or general primary school) and MI (Madrasah 

Ibtidaiyah, the madrasah equivalent) 

• Junior secondary schools, both SMP (Sekolah Menengah Pertama in the general system) and 

MTs (Madrasah Tsanawiyah, the madrasah equivalent). 

LEARNING FACTORS: INTERVENTIONS, PRACTICES, AND 

CONTEXTUAL FACTORS 

As previously described, the recent meta-analyses and reviews on learning outcomes in developing 

countries largely focused on interventions. Those reviews mostly restricted the meta-analysis to 

studies of experimental (or randomised controlled trials) or quasi-experimental design. Besides 

focusing on Indonesia only, this review seeks to incorporate other types of study designs and along 

with them other types of correlates of student learning outcomes. 

More specifically, the three aspects on which this review is interested in for examining their relationship 

with learning outcomes include interventions, practices and contextual factors. These are defined as 

follows: 

• Interventions: these are programs, projects or approaches that have been purposely designed 

and introduced (or removed). Their scale, scope and specifications may vary. They may range 

from a large-scale government-led reform such as teacher certification, to a new pedagogical 

approach to teaching mathematics piloted in a group of schools. The studies that examine 

interventions are often evaluation studies, which may include experimental, quasi-experimental 

and non-experimental designs with and without controls. 

10 


• Practices: these refer to classroom, school- and community-level practices that are undertaken 

by the relevant actors. Practices may be influenced by interventions. Unlike interventions, 

however, such practices are not necessarily a direct result of an explicit policy. They also may 

not have a clear starting or end point. Relevant studies of teaching-learning practices are often 

non-experimental in design, capturing variations in observed practices and their correlation with 

student learning outcomes. For example, a study may collect and link data on different types of 

school management practices and their relationship with student learning. In another example, 

a household survey may be used to examine the amount of time parents spend helping their 

children with homework with the children’s school outcomes. 

• Contextual factors: these are factors that may directly or indirectly affect student learning but 

cannot be easily influenced or changed by actors in question. Many studies, for example, 

account for the influence of students’ socio-economic background on their learning outcomes 

by asking about their parents’ education levels or resources they have at home. This review 

will also collect estimates of contextual factor that correlates with learning outcomes. These 

can be used as a benchmark for comparing the size of effects of interventions 

and practices. 

LEARNING OUTCOMES: LITERACY AND NUMERACY 

Literacy and numeracy include the functional “three-Rs” skills of reading, writing and arithmetic 

(in Indonesian this is popularly known as membaca, menulis dan berhitung or calistung). Both 

concepts, however, also extend to broader cognitive abilities: to do abstract and logical reasoning, 

and interpreting a problem-solving process using these basic skills. 

The literacy and numeracy skills that this review focuses on are most closely, but not strictly, linked 

to the subjects of mathematics and languages in the curriculum. An intervention or practice at the 

school- or household-level, or delivered through another curriculum subject, may also influence 

literacy and numeracy outcomes. 

A final key concept to delineate this review’s area of focus is outcomes, or more specifically, 

measurable literacy and numeracy outcomes. In particular, it seeks studies that can clearly define 

and measure the learning outcomes that such factors influence. The review was only interested in 

the learning outcomes that is captured by a set of objective measures. The most common way for 

studies to do this is by using assessments of student learning outcomes in literacy and/or numeracy. 

11 


Review questions 

The aim of this review is to investigate the interventions, practices, and contextual factors that are 

linked to literacy and numeracy outcomes for basic education in Indonesia. More specifically, the 

review will seek to address the following two questions. 

1. What are the effects of different interventions, practices and contextual factors on student 

learning outcomes in Indonesia? 

2. How do these factors interact with each other in contributing to student learning outcomes? 

Taken together, in answering the above questions, the review will describe and compare the factors 

that can be harnessed to improve student learning outcomes in Indonesia, as well as provide insight 

into the state of research for this important area. 

12 


Method 

STUDY SELECTION CRITERIA 

One approach to defining studies relevant to a systematic review question is to address population, 

intervention, control or study design, and outcomes of interest or PICO (The Joanna Briggs Institute, 

2014). PICO are defined as follows: 

• Population: the review will focus on studies involving Indonesian students in basic education 

(that is, Grades 1 to 9) in the formal education system (which includes primary 

schools or SD/MI and junior secondary schools or SMP/MTs). 

• Interventions, practices and contextual factors: the review will include studies that examine 

the role of policy and program interventions, practices within the classroom, school and 

community, as well as home and community contextual factors that affect student learning 

outcomes. 

• Comparison: This review will include experiment studies that explicitly identify control or 

comparison groups, as well as studies that identify the effect of differences in particular 

observable variables on literacy and numeracy outcomes. 

• Outcomes: the review will include studies that link the above factors to measurable and clearly 

defined student learning outcomes in literacy or numeracy. In other words, the studies should 

assess literacy or numeracy learning outcomes in an objective manner. These outcomes 

were measured quantitatively, most commonly as change or differences on an assessment of 

literacy or numeracy skills. 

Given the specifications regarding control groups and outcomes above (and definitions discussed), this 

review will include studies that provide measured estimates of the effects of particular interventions, 

practices and factors on student learning outcomes. Therefore, this will exclude studies that employ 

a purely qualitative study design to describe effects. A complete list of what this systematic review is 

about can be seen in the table below. 

13 


Table 1. What this systematic review is (not) about 

Qualifier Not about About 

Population of study 

Studies that were conducted in country(s) 

other than Indonesia, and Indonesian 

sample were not included 

Studies that conducted in 

Indonesia or present specific 

analysis on Indonesian 

sample 

Level of schooling 

Type of schooling 

Learning outcomes 

Information provided 

Senior high school level (SMA or MA) and 

Vocational high school 

Non-formal education (e.g. religious 

schools or pesantren that do not follow 

madrasah or formal school curriculum). 

Studies did not measure learning 

outcomes, or measuring learning outcomes 

other than literacy and numeracy. Studies 

measured literacy outcomes other than 

Indonesian language. The learning 

outcomes was not measured in an 

objective way 

Narrative reviews, discussion or opinion 

papers, quantitative studies with incomplete 

statistical report. 

Primary (SD or MI) and junior 

secondary school (SMP or 

MTs) level or grade 1 to 9 

Formal school including 

madrasah, private and public 

schools 

Studies measured student 

numeracy and literacy 

(in Indonesian language) 

learning outcomes with a set 

of clear criteria 

Studies had to provide data 

that are possible to be metaanalysed 

Intervention, practices, and 

contextual factors 

Studies that did not investigate the link 

between factors to the learning outcomes. 

Studies had to investigate the 

link between factor(s) to the 

student learning outcomes. 

SEARCH 

The review team conducted a systematic search in August 2016. The team collected published 

papers, unpublished research reports, as well as ‘grey’ literatures from academic databases, 

Indonesian universities repositories (most of these are teacher training institutions), websites of 

international and local research institutes. The review team searched studies that are written in 

English and Indonesian. Several full texts of an unpublished research reports from universities 

were obtained directly from the universities. A total of 3,483 studies were collected, but we found 

many duplications 1 . The unique set of 2,693 individual studies proceeded to the screening phase. A 

complete list of sources searched as well as the search strings and number of studies found for each 

search is presented in the Appendix 2. 

1 

Same studies appeared in more than one database. 

14 


ABSTRACT AND FULL TEXT SCREENING 

The screening process was done in two stages. The first stage was title and abstract screening 

where the studies were included if the abstracts met all of the following criteria: 

• The study was done in Indonesia 

• The participants were elementary and/or junior secondary students 

• The study assessed students’ literacy and numeracy outcomes 

All studies were double screened using Covidence Systematic Review Software (Veritas Health 

Innovation, Ltd.). All disagreements were coded as conflicted studies and were assigned to a third 

reviewer. As a result of the abstract screening, 233 studies were selected to be carried forward to 

full-text screening. 

The double screening method was also employed in the full-text screening. In this stage, the reviewers 

looked more into the detail of the study full-texts and included them if the study: 

• examined the relationship between any factors and student’s literacy and numeracy learning 

outcomes in a quantitative manner, and 

• assessed students’ learning outcomes in an objective manner. 

This process resulted in 120 included studies to be extracted. 

IN-DEPTH REVIEW 

The in-depth review stage moved from a broad characterisation or mapping of the included studies, 

to making detailed descriptions and extracting the relevant evidence within the studies. In this 

stage, findings reported by each of the studies were double coded using EPPI Reviewer 4 (Thomas, 

Brunton, & Graziosi, 2010). In this stage, reviewers extracted statistical outcomes reported in the 

paper for further analysis. This process also included quality appraisals of each of the study as the 

basis for further cut-off of the low-quality studies. All extracted data were categorised into factors that 

contribute to students’ achievements as suggested by Hattie (2009), as follows: 

• Home, e.g. parents’ education, support from parents, language spoken at home, resources 

available at home, household wealth 

• School, e.g. student-teacher ratio, facilities or resources at school, school budget, school 

community size, language of instruction 

• Curricula, e.g. type of curricula, subject hours, use of standardised national or regional 

assessment 

• Teacher, e.g. qualifications, experience, gender, professional development undertaken, 

absence rate, job satisfaction 

• Teaching, e.g. lesson plan, provision of feedback, degree of interaction with students during 

lessons, collaboration with other teachers, strategy or approach 

• Student, e.g. gender, age, health condition, attitude, motivation, time spent on reading or 

sports or other activities, participation in tutorial classes 

15 


Given the lack of findings related to curricula, this category was excluded from further analysis. 

The final component for this stage was the quantitative synthesis of these collected evidence using 

DerSimonian & Laird’s (2007) random effect model of meta-analysis to answer the first question of 

the review. 

A limitation of a standalone meta-analysis tends to be that it is unable to explore the mechanisms 

behind the relationship between inputs and outcomes (Evans & Popova, 2015). To give a deeper and 

more comprehensive picture of what was behind the relationship between the factors and student 

learning outcomes, this review also extracted qualitative information for supplementary analysis. The 

information was coded based on the following questions: 

• Have disadvantaged groups been excluded from the study (such as students with learning 

difficulties, living in remote areas, coming from low SES family etc.)? 

• Did the study focus on disadvantaged groups such as special needs children, lower socioeconomic 

background, different ethnic backgrounds? What are the characteristics of the study 

population? Can the intervention be adapted to the population without disadvantages? 

ASSESSING QUALITY OF STUDIES 

As discussed above, the quality of each study was doubly assessed by the review team. The criteria 

for the assessment are adapted from the Cochrane Risk of Bias Assessment tools (Higgins & Green, 

2011). The quality assessment tools were modified into two sets of forms to accommodate the nature 

given by two distinct research methods; observational and experimental studies. 

For experimental studies, the reviewers assessed whether the study has a high, low, or unclear risk 

of bias. The different types of bias are listed below: 

• Selection bias: 

Assessing whether there is potential bias due to the sampling method employed in the study 

and whether the intervention allocations could have been foreseen before or during the 

enrolment. 

• Attrition bias: 

Detecting if there is any risk of bias due to the withdrawal during the experiment process or 

exclusions of observations from the analysis. 

• Reporting bias: 

Looking whether the author only reported selected outcomes and the absence of the 

unreported variable could lead to the higher risk of bias. 

• Omitted variable bias: 

The low risk study is the one that identified confounding factors and used strategies to deal with 

them, e.g. with the use of covariates 

• Measurement bias 

To minimise the bias given by the instrument that assessed literacy and numeracy outcomes, 

reliability and validity of instruments to measure student learning outcomes should be tested 

and described 

16 


• Performance and detection bias 

Detecting the risk of bias of the measured outcomes due to whether the outcomes assessors 

know which students who are assigned to treatment or control group. 

In addition to the adapted Cochrane risk of bias form above, we also assessed whether the studies 

used a particular calculation to justify the sample size needed or presented the error margins (e.g. 

standard error, standard deviation, confidence interval) alongside the estimates. 

For the observational studies, several dimensions such as concealment of allocation under selection 

bias, attrition bias, as well as performance and detection bias were included in the form. 

To summarize these quality criteria into an overall quality indicator for each of the study, the reviewers 

put weight on each dimension based on how strong the bias was for a particular dimension and by 

how far it would affect the accuracy of the estimation. Studies with high risk of bias due to selection 

process methods, omission of confounding variables in the analysis, or low validity and reliability of 

the outcomes’ measurements were considered to be low quality studies. If the study has a high risk 

of bias in at least one of the other criteria, that study was judged as a medium quality study. The high 

quality studies were those with low risk of bias in all dimensions assessed in the form. The overall 

judgements of the included studies is presented in the Appendix 5. 

In addition to the assessment based on the risk of bias assessment tools, we also assessed whether 

the statistical report presented in the data are sufficient to be meta-analysed. Findings from 120 

individual studies were collected. However only 71 studies that reported 150 findings could be 

meta-analysed. The rest do not report sufficient information. From the 71 studies, only nine of them 

considered as high quality studies, while about 45 studies are considered to be low quality. To reduce 

the risk of bias that this systematic review findings could have, the low quality studies were excluded 

from the review. The detail explanation about the data found in the low quality studies is explained 

in the result section. 

17 


Figure 2. Systematic map; searching, screening, and quality appraisal 

Studies identified through the 

systematic search process 

(n= 3,483) 

Records after duplicates removed 

and screened based on their titles 

and abstracts (n= 2693) 

Excluded from title and abstract 

screening (n= 2460) 

Selected from Abstract and Title 

Screening (n=233) 

No Full text found 

(n= 23) 

Screened based on full-text 

(n= 210) 

Excluded (n= 90) 

Quality Appraisal process (n= 120) 

Excluded due to the incompleteness 

of statistical report (could not be 

meta-analysed). (n= 49) 

Excluded due to low quality (n= 45) 

Studies Included 

(n= 26) 

18 


Results 

The result of the synthesis will be presented in two sections. This first section compares the magnitude 

of effects given by student, home, teacher, teaching, and school factors to the increase of literacy 

and numeracy outcomes as well as highlighted particular specific factors or interventions that has 

a remarkable effect in each category. We found that teaching is the area that contributed most to 

the learning outcomes. The second part explores the mechanism behind the positive or negative 

effects of the teaching category. For example, good teaching practice is dependent on the teachers’ 

pedagogical skill as well as subject proficiency. However, it is important to keep in mind that this 

review only considered practices and interventions that have been evaluated in Indonesia, which are 

not exactly representing the complete picture of Indonesian education. 

META-ANALYSIS 

Each of the findings from all included studies was converted into Cohen’s effect size (d), or 

standardised mean difference (see the basic formula below). 

d = X - X t c 

SD c 

iiiiiiandiiiiiiiare X t 

X c 

mean of post-test score in treatment and control group, respectively. The score 

difference is divided by standard deviation of the control group (Please see the detail formula used 

to convert various types of outcomes in Appendix 6). 

Standardising the outcomes into effect sizes enable us to have same standard to interpret findings 

from different studies (Lipsey & Wilson, 2001). We interpret the effect size based on Cohen’s Effect 

Size Index (Cohen, 1992). 

Table 2. Cohen’s Effect Size Index 

Effect Size 

Interpretation 

-0.19 to 0.19 No effect 

0.2 to 0.49 or (-0.2) to (-0.49) Small effect 

to 0.79 or (-0.5) to (-0.79) 

Medium effect 

> 0.8 or

Meta-analysis by the quality of the studies 

Based on the quality appraisal process, we only found 9 studies, reporting 21 findings, that were 

considered as high quality studies or having low risk of bias on all the dimensions assessed. The 

majority of the studies were considered to have low quality (45 studies reporting 74 findings) while 

16 studies reporting 54 findings were appraised as medium quality studies. By only looking at high 

quality studies, we found that number of findings per category is very limited as can be seen in the 

table below. 

Table 3. High quality findings 

Intervention 

Category 

N of 

Findings 

Source/ Independent Variable 

Direction of effect 

Home 1 

a. Ali et al (2011)/ Home background factor (parental 

education background and number of resources 

a. Positive 

School 8 

a. Ali et al (2011)/ Number of school resources 

b. Suryadarma et al (2006)/ proportion of permanent 

teacher 

c. Pradhan et al. (2011)/ collaboration school committee & 

village council – numeracy 

d. Pradhan et al. (2011)/ collaboration school committee & 

village council – literacy 

e. Pradhan et al. (2011)/ Democratic election of school 

committee 

f. Suryadarma et al (2006)/ School using Indonesian 

g. Suryadarma et al (2006)/ Student-teacher ratio 

h. Suryadarma et al (2006)/ school using language that 

different from mother tongue of the majority of students 

a. Positive 

b. Negative 

c. Positive 

d. Positive 

e. Positive 

f. Positive 

g. Negative 

h. Negative 

Student 5 

a. Suryadarma (2015)/ gender (girls > boys) 

b. Sudijono et al (2014)/ Self-efficacy 

c. Ali et al (2011)/ age 

d. Ali (2011)- students’ expectation on education 

e. Ali (2011)- Negative attitude toward school 

a. Positive 

b. Positive 

c. Negative 

d. Positive 

e. Negative 

Teacher 4 

a. De Ree et al (2015)/ teacher certification - numeracy 

b. De Ree et al (2015)/ teacher certification - literacy 

c. Trorsenburg et al (2015)/ Teachers’ subject and 

pedagogical competence -numeracy 

d. Trorsenburg et al (2015)/ Teachers’ subject and 

pedagogical competence – literacy 

a. No effect 

b. No effect 

c. Positive 

d. Positive 

Teaching 3 

a. Iqram (2014)/ Collborative and problem-based learning 

b. In’am (2013)/ Formula for understanding number 

sequence 

c. Sudijono (2014)/ Mind mapping 

a. Positive 

b. Positive 

c. Positive 

Total 21 9 studies 

20 


The outcomes of this quality assessment is consistent with a previous review conducted by Evans and 

Popova (2015) where only 11 studies from Indonesia are sufficiently rigorous to be included in their 

review. Table 4.2. and 4.3 below summarise the effect size for numeracy and literacy, respectively, 

from high quality studies. If all high quality findings are combined into one pooled effect size, we 

can found that there is a high overall effect to the numeracy (d=0.901) 2 but no effect to the literacy 

(d= 0.097) 3 learning outcomes (see Table 2). The forest plot of the effect sizes is presented in the 

appendix 7. 

Table 4. Overall effect size for numeracy from high quality studies 

Study Effect [95% Conf. Interval] % Weight 

Percentage of permanent teacher -1.762 -1.801 -1.722 8.37 

Negative attitude toward school -0.436 -0.438 -0.435 8.38 

Age -1.876 -12.013 8.261 0.19 

Collaborative & problem-based 0.743 0.543 0.942 8.24 

committee and village apparatus collaboration 0.334 0.317 0.35 8.38 

Competence 1.387 1.311 1.464 8.36 

expected education level 4.129 3.901 4.358 8.2 

Gender 0.708 0.698 0.719 8.38 

mind mapping 0.64 0.563 0.718 8.36 

Number of school resources 0.505 0.439 0.571 8.36 

self-efficacy 0.627 0.4 0.854 8.2 

Changing strategy based on students’ needs 1.888 1.662 2.115 8.2 

use Indonesian language of instructions 2.188 2.113 2.263 8.36 

Overall effect (dl) 1 0.901 0.451 1.351 100 

Table 5. Overall effect size for literacy from high quality studies 

Study Effect [95% Conf. Interval] % Weight 

Collaborative &problem-based 0.448 0.43 0.465 20 

Competence 0.345 0.319 0.371 20 

Democratic 1.05 1.019 1.081 19.99 

diff mother tongue -0.693 -0.702 -0.683 20 

Student : teacher ratio -0.666 -0.676 -0.657 20 

Overall effect (dl) 0.097 -0.452 0.645 100 

The above table shows how limited the information is that can be drawn from this meta-analysis. 

By only including the nine high qualified studies in this meta-analysis, this review would not be able 

to draw a comprehensive comparison of a wide range of factors that contribute to numeracy and 

literacy learning outcomes. 

2 

The overall effect was significant (at 95% confidence interval) with variance of 0.651 

3 

The overall effect was not significant (at 95% confidence interval) with variance of 0.391 

21 


To map how factors from all the included studies contribute to learning outcomes, we then need to 

consider all findings that linked to literacy and numeracy outcomes regardless of the quality of the 

studies. The mean of weighted effect size for factors that contribute to numeracy outcomes is 0.717 

with a variance of 0.152 and the overall heterogeneity (Q) value is 5600000 (df= 79, p= 0.00) which 

indicates a lot of variation in the study. Similarly, the pooled effect size for literacy outcomes is 0.563 

with a variance of 0.218 and overall heterogeneity value is 7800000 (df = 73, p= 0.00). The table and 

forest plots that illustrate the effect size can be found in the Appendix 8. 

From the low quality studies, there are several promising interventions that had a large effect size 

such as giving iron supplement to anaemic students (d= 2.97, v= 0.147), discovery learning teaching 

method (d= 3.91, v= 0.1818), using media boards as inspiration for students in writing lesson (d= 

3.51, v= 0.17), and students’ ability to regulate themselves in learning (d= 2.4, v= 0.34). However, it 

is too risky to include these studies due to their poor methodology. The studies above, for example, 

did not use a rigourous method in recruiting their participants. Most of the low quality studies also 

tested their interventions on very small samples, as low as 20 to 30 participants, and did not use a 

control group for comparison. 

If we summarise the weighted effect sizes of high and medium qualities altogether for numeracy 

outcomes, we can have a medium pooled effect size (d= 0.592) 4 . For numeracy outcomes, the 

pooled weighted effect size is small (d= 0.214) 5 . 

To find a balance between meeting the review purpose and presenting the results with minimum 

risk of bias, this review only includes studies with high and medium quality in the further analysis. 

In addition to that, several medium quality studies with large standard errors were considered as 

outliers and eliminated from the further analysis. The eliminated studies are Rosida (2016), Nurhayati 

& Wimbarti (2013), Rostika, Windayana, & Komariah (2016) and Habibah & Wimbarti (2003). 

Meta-analysis by type of intervention 

Student Effect 

In this category, we grouped independent variables related to the child or aspects that the child 

brings to their learning process. In this category, we only include factors that can suggest possible 

areas of improvement, for example, attitude, aspiration, or ability to think logically. The overall pooled 

effect size of these factors is 0.486 with a value of variance 0.18. However, there are several factors 

that have a large effect size (d> 0.8), such as students’ expectation on their education attainment (d= 

4.129, v= 0.12) and motivation (d= 0.827, v= 0.02). Table 4 below illustrates the effects of student 

factors on numeracy outcomes. 

4 

The variance is 0.151 and heterogeneity value is 5600000 (df= 48, p= 0.00) 

5 

The variance is 0.218 and heterogeneity value is 7700000 (df= 30, p= 0.00). 

22 


Table 6. Student effect to numeracy outcome 

Study Effect [95% Conf. Interval] % Weight Quality 

negative attitude toward school -0.436 -0.438 -0.435 8.57 High 

Negative attitude toward Indonesian lesson -0.264 -0.292 -0.236 8.56 medium 

Attendance 0.161 0.16 0.162 8.57 medium 

expected education level 4.129 3.901 4.358 7.97 High 

Girls > boys 0.708 0.698 0.719 8.57 High 

Time spent on learning at home 0.06 0.014 0.106 8.54 medium 

logical thinking 0.1 0.054 0.147 8.54 medium 

Motivation 0.827 0.791 0.862 8.55 medium 

self-directed 0.06 0.014 0.106 8.54 medium 

self-efficacy 0.627 0.4 0.854 7.97 High 

spatial ability 0.453 0.067 0.839 7.05 medium 

Perception on study obstacles -0.34 -0.369 -0.311 8.56 medium 

Overall effect (dl) 0.486 0.243 0.729 100 

The meta-analysis shows that these factors have no statistically significant effects on literacy 

outcomes (d= -0.255, v= 0,312). The table below illustrates that there is no individual factor that has 

an effect higher than 0.8. There is a medium negative effect for gender (d= -0.651, v= 0.00), while 

the other factors only contribute small or no effect. 

Table 7. Student effect to literacy outome 


Positive attitude toward Indonesian lesson -0.202 -0.229 -0.174 20 medium 

Negative attitude toward school -0.228 -0.256 -0.201 20 medium 

Boys < girls -0.651 -0.652 -0.65 20.01 medium 

Attendance 0.141 0.14 0.141 20.01 medium 

Perception on study obstacles -0.334 -0.362 -0.305 19.99 medium 

Overall effect (dl) -0.255 -0.744 0.235 100 

The findings for student effects are in line with John Hattie’s (2009) conclusions that developmental 

effects only contribute 0 to 0.15 standard deviation to the learning outcomes. 

Teacher Effect 

An overall medium effect (d= 0.582, v= 0.252) is contributed by the pooled teacher factors to the 

numeracy learning outcomes. The teacher category consists of measures such as their experience in 

teaching, employment status, payment scheme received and competence. Among the synthesized 

factors, the highest influence is demonstrated by teachers’ subject and pedagogical competence (d= 

1.39, v= 0.04), whereas the other factors only provide a small or no effect. This is summarised in the 

table below: 

23 


Table 8. Teacher effect to numeracy outcomes 


years in teaching 0.12 0.12 0.121 33.43 medium 

Competence 1.387 1.311 1.464 33.23 High 

Certification 0.243 0.192 0.294 33.34 medium 


A very small effect is observed in the meta-analysis for teacher factors of literacy outcomes (d= 

0.175, v= 0.002). Similar to numeracy outcomes, teachers’ subject and pedagogical competence 

provides by far the highest effect on literacy outcomes (d= 0.345, v= 0.01). However, according to 

Cohen’s (1992) barometer, this effect is still considered to be small. 

Table 9. Teacher effect to literacy 


Certification 0.121 0.08 0.162 32.71 medium 

Competence 0.345 0.319 0.371 33.41 High 

years in teaching 0.161 0.16 0.162 33.88 medium 


Teaching Effect 

This meta-analysis synthesizes the effects of various teaching strategies used in the classroom, such 

as use of formative assessment, collaborative learning, problem-based learning, or a combination 

between several types of strategies. A large effect of this intervention is shown in the meta-analysis 

both for numeracy (d= 1.319, v= 0.305) and literacy (v= 0.931, d= 0.448) 

Interventions that showed distinctly high effects for numeracy learning outcome are formulas for 

understanding numerical sequences (teaching formula) (d=1.89, v= 0.12) and hypno-teaching 

(d=5.376, v= 0.32). The summary can be seen in the table below. 

Table 10. Teaching effect to numeracy 


peer-tutoring 0.206 0.087 0.326 12.93 medium 

formative assessment 1.358 1.326 1.39 13.08 medium 

computation strategy 0.649 0.511 0.786 12.88 medium 

mind mapping 0.64 0.563 0.718 13.02 high 

information mapping 0.675 0.612 0.737 13.05 medium 

Hypno teaching 5.376 4.752 5.999 9.83 medium 

Collaborative & problem-based 0.743 0.543 0.942 12.66 high 

Using learning media 1.888 1.662 2.115 12.54 high 


24 


For literacy learning outcomes, interventions that showed a large effect are using serial pictures 

to help children compose a passage (d= 2.412, v= 0.12) and intensive reading to help students 

comprehend texts (d= 0.865, v= 0.15). Beside these two strategies, most of the synthesised strategies 

have a medium effect on literacy outcomes. The teaching strategies and their effects can be seen in 

the table below. 

Table 11. Teaching effect to literacy outcomes 


intensive reading 0.865 0.564 1.167 19.43 Medium 

talking stick game 0.452 0.337 0.567 20.3 Medium 

SQ3R (Survey, Question, Read, Recite, Review) 0.405 0.241 0.568 20.14 Medium 

Collage 0.547 0.415 0.678 20.25 Medium 

serial pictures 2.412 2.187 2.638 19.87 Medium 


School Effect 

School effects comprise factors that relate to school policy and resources. The random effect metaanalysis 

model for school factors to numeracy outcomes shows an insignificant or no effect of the 

overall factors. However, looking at the effect of individual studies, we found a large negative and 

positive effects, such as a large proportion of permanent teachers have a negative impact (d= 

-1.762, v= 0.02), whereas large positive effects were contributed by the use of Indonesian language 

in schools (2.188, v= 0.04) and principal preparedness to implement school-based management (d= 

2.762, v= 0.01). Individual school effects from each of the studies are illustrated in the table below. 

Table 12. School effect on numeracy outcomes 


principal preparedness 2.762 2.743 2.782 25 Medium 

Committee and village apparatus collaboration 0.334 0.317 0.35 25 High 

percentage of permanent teacher -1.762 -1.801 -1.722 25 High 

use Indonesian 2.188 2.113 2.263 24.99 High 


Likewise, the meta-analysis reported the overall school factors had an insignificant effect on students’ 

literacy learning outcomes (d= 0.102, v= 0.464). Only one factor had a large positive effect on the 

literacy outcomes, which is school policy to elect school committee in a democratic manner (d= 1.5, 

v=0.02). The rest of the studies show medium negative effects and small positive effects. The effects 

are presented in the table below. 

25 


Table 13. School effect on literacy outcomes 


principal preparedness 0.37 0.369 0.371 20 medium 

Democratic 1.05 1.019 1.081 19.99 High 

Committee and village apparatus collaboration 0.448 0.43 0.465 20 High 

Language of instruction different from mother tongue -0.693 -0.702 -0.683 20 High 

Student : teacher ratio -0.666 -0.676 -0.657 20 High 


Home Effect 

This category comprises several independent variables that relate to the home environment and 

family life. In this category, we only focus on factors that can possibly be interfered with or factors 

that provide scopes for improvements. Other factors related to home environment that are more 

static, for example parent educational background or ethnicity background, will be discussed in the 

next category. 

Home factors have significant but small effects on both literacy (d= 0.322, v= 0.017) and numeracy 

outcomes (d= 0.463, v= 0.039). Looking at the individual studies, only one variable had a large 

effect on literacy; the duration of reading activities done by students at home (d= 0.84, v= 0.02). 

On the other hand, all individual variables under home factors only had small effects on numeracy 

outcomes. Tables that summarize the effect of home factors are presented below. 

Table 14. Home effect to numeracy outcomes 


number of family members who helped 0.43 0.399 0.46 33.37 Medium 

Number of family members who encouraged 0.372 0.343 0.402 33.4 Medium 

parent involvement 0.165 0.129 0.2 33.22 Medium 


Table 15. Home effect to literacy outcomes 


Number of family members who helped 0.331 0.302 0.36 25.04 Medium 

Number of family members who encouraged 0.327 0.298 0.356 25.04 Medium 

reading activity 0.84 0.797 0.884 24.87 Medium 

reading facility 0.355 0.326 0.384 25.04 Medium 


26 


Demographic findings 

This category covers variables that do not provide room for improvements and needs to be considered 

as they are. Listed in this category, for example, are region of domicile, family education and SES 

background, students’ age, and resources owned by the household or school. For numeracy 

outcomes, the pooled factors had a small effect size (d= 0.443, v= 0.24). Meanwhile, the metaanalysis 

found that there is no effect of the pooled variable on literacy (d= -0.092, v= 0.414). The 

summary of effects can be seen in the tables below. 

Table 16. Other factors linked to numeracy outcomes 


Age -1.876 -12.013 8.261 0.13 High 

International-equivalent curriculum 0.73 0.729 0.731 13.32 Medium 

home background 1.336 0.329 2.343 6.68 Medium 

Kalimantan vs Java -0.349 -0.349 -0.348 13.32 Medium 

number of people in home 0.403 0.373 0.433 13.31 Medium 

number of school resources 0.505 0.439 0.571 13.27 High 

Papua vs Java -0.482 -0.483 -0.481 13.32 Medium 

Religion -0.331 -0.36 -0.302 13.31 Medium 

Sulawesi vs Java -0.482 -0.483 -0.481 13.32 Medium 


Table 17. Other factors linked to literacy outcomes 


age 0.295 0.267 0.324 14.28 Medium 

Bali vs Java -0.843 -0.844 -0.842 14.29 Medium 

Maluku vs Java -0.935 -0.937 -0.934 14.29 Medium 

parent education 0.459 0.458 0.46 14.29 Medium 

Sulawesi vs Java -0.37 -0.371 -0.369 14.29 Medium 

n people in home 0.355 0.326 0.384 14.28 Medium 

possession 0.394 0.364 0.424 14.28 Medium 

Overall effect (dl) -0.092 -0.569 0.384 100 

27 


Discussion 

WHAT WORKS TO IMPROVE STUDENTS’ LITERACY AND NUMERACY 

OUTCOMES? 

Comparing pooled effect sizes between types of interventions, we can see that teaching and school 

factors have the highest pooled effect on both literacy and numeracy learning outcomes. Under 

school factors, principal preparedness to implement school-based management contributed the 

highest to numeracy learning outcomes. A good school management is determined by a principal 

who can provide supervision to the school staffs and have a clear plan in terms of academic 

improvement and school finances. These principals are also capable of supervising and evaluating 

teachers’ performance (Vernez, Karam, & Marshall, 2012) 

Findings from the study by Suryadarma, Suryahadi, Sumarto, and Rogers (2006), show that having 

a higher proportion of permanent or civil servant teachers have a large negative impact on students’ 

math scores. This finding may indicate that secure position lowers teacher productivity in comparison 

to non-permanent teachers who need to maintain their performance to secure their jobs. The school 

level measure school committee democratic elections have been found to also have a large impact 

on learning outcomes. A study conducted by Pradhan et al. (2014) explained that the reform of 

school policy related to school committee organisations improve community engagement to support 

learning. To summarise, findings from school factors reveal that the principals’ leadership skills, 

including how they boost teachers’ performance and community engagement, positively impact 

learning outcomes. 

Let us now take a look at the broader implications for the variables under teacher and home categories 

that contribute largely to the learning outcomes. Most home factors have a small effect on student 

learning outcomes. Looking into the individual variables, factors that influence both numeracy and 

literacy learning outcomes is support from family in a form of encouragement and help. Specific to 

the literacy outcomes, the possession of books and how often students read at home contribute 

largely to their literacy outcomes (Johnstone, 1983). A study conducted by van Bergen, van Zuijen, 

Bishop, and de Jong (2016), that also found a relationship between access to books and literacy 

ability, explained that a number of books possessed by a family indicates the value placed on 

literacy. Furthermore, this study found a strong correlation between a number of book possessed 

and children’s literacy level even when variables such as parents’ reading fluency and environmental 

factors had been controlled. 

Turning now to the teacher effect, the current meta-analysis found that experience and doublepayment-scheme 

for teachers did not have any effect on their students’ learning outcomes. Teacher 

competence, as assessed using a set of tests that covered mathematics, science, Indonesian 

language, pedagogical knowledge, and social competencies (van Trotsenburg et al., 2015), had 

a huge effect on numeracy outcomes but only a small effect on literacy. This finding is in line with 

what Hattie emphasised in his synthesis of meta-analyses that teacher is the most powerful factor 

in learning due to their content knowledge, social competencies, and teaching skills (Hattie, 2009). 

28 


However, the limitations of the studies included in our review is that they only suggest positive 

relations between teachers’ content and pedagogical knowledge to students’ learning outcomes 

without knowing exactly whether the teachers have already used their knowledge and pedagogical 

skills according to students’ needs or classroom context (Depaepe, Verschaffel, & Kelchtermans, 

2013). 

With regards to the student’s effect on learning outcomes, the pooled meta-analysis showed a small 

effect of the mean of weighted effect size under student’s category on literacy (d= -0.255) and 

medium effect on numeracy learning outcomes (0.486). These findings are similar to observations 

made by Hattie (2009) that what children bring at the start of their schooling is critical to their later 

achievement. Looking into each finding in this meta-analysis, we can see that motivation and 

aspiration or expectation of highest education level that they plan to pursue has the highest effect 

on numeracy outcomes. In their review, Miller & Brickman (2004) explained the mechanism behind 

the impact that motivation and aspiration have on learning outcomes. A clear aspiration or target 

goal serves three functions in student learning: (i) define immediate performance that the child hope 

to produce, (ii) give the child signal or awareness on what aspects of learning performance they 

should focus on and regulate, and (iii) provide the child standards to be used to evaluate their own 

performance. These three aspects can be the source for drive for students to learn and regulate 

themselves (Miller & Brickman, 2004). Reviews suggest that these personal factors are influenced 

by both genetics and nurture factors from experience, knowledge, and value that the children gained 

from home environment or during pre-school (Hattie, 2009; Miller & Brickman, 2004). However, such 

factors can also be shaped by teachers, schools, or policies. This is exemplified in the recent review 

undertaken by Snilstveit et al. (2016) that students’ motivation to learn can be effectively induced by 

giving a merit-based scholarship to students in the countries with low to medium income. 

If the goal of education is to equip students with basic skills that can be partially translated into 

learning outcomes, providing interventions that focus on students’ personal characteristic (e.g., 

motivation, aspiration, self-regulated learning, etc.) may not translate into academic learning gains 

if the children are still poorly taught at schools (Evans & Popova, 2015). In line with what earlier 

reviews suggested (e.g. Evans & Popova, 2015; Hattie, Biggs, & Purdie, 1996; Hattie, 2009), the 

current study also found that amongst the categories of variables, teaching category has the highest 

pooled effect size on learning outcomes for both numeracy and literacy. However, many studies that 

fall into this category suffer from methodological issues. 

Other interesting findings under teaching category are how teacher use formative assessment to 

give themselves as well as the students feedback on the learning progress and using learning media 

to introduce new concepts in math. These two variables are supported by Evans and Popova (2015) 

who found in their review that teacher-led methods, such as formative assessment and targeted 

instruction, had a large effect on students’ learning outcomes in the low and middle-income country. 

Our review also found that inquiries help children to understand reading, while making collage from 

series of pictures or texts help students to compose a writing. Nevertheless, we need to keep in mind 

that the teaching strategies investigated in the current review are mostly coming from classroom 

action research reports where the interventions were designed based on problem identification 

29 


undertaken by the teachers. Considering the methods employed in these studies, a likely explanation 

of the positive effects of teaching strategies is they were tailored to students’ needs, abilities and 

problems. Snilstveit and colleagues’ (2015) review also suggest that the provided materials will not 

have the desired impact if it is not tailored to students’ needs and ability. 

INTERVENTION, PRACTICES, AND CONTEXTUAL FACTORS 

The recent meta-analyses on the quality of education in developing countries largely focus on 

impact evaluations of intervention or policy reforms (e.g. Asim et al., 2015; Evans & Popova, 2015; 

Masino & Niño-Zarazúa, 2015)at least six systematic reviews or meta-analyses have examined 

the interventions that improve learning outcomes in low- and middle-income countries. However, 

these reviews have sometimes reached starkly different conclusions: reviews, in turn, recommend 

information technology, interventions that provide information about school quality, or even basic 

infrastructure (such as desks. The current review aims to see the linkage between interventions, 

practices, and contextual factors. 

Looking at medium and large effects in each category of independent factors, there are several factors 

that can be considered as interventions, such as introduction of formative assessment for teachers, 

language of instruction policy, policy reform in school committee democratic election for boosting 

parental involvement in students’ learning, and the usage of international-equivalent curriculum. 

Some factors that fall under teaching categories were evidence from practices that are reported by 

teachers as their classroom action research. Under practice category, we can list several activities 

that have been done by the teacher, for example changing the strategy of delivering materials 

or illustrating the concepts using media. A problem-based approach to learning that is done in a 

collaborative way or how the students’ family members help and encourage them at home are also 

considered to be practices that support students’ learning process. Under contextual factor, we listed 

things that students bring to the school for instance, motivation, self-efficacy, and aspiration as well 

as their teachers’ competency in terms of the subject they teach and their pedagogical knowledge. 

We also found students having particular demographic characteristics have lower learning outcomes 

for example, males in comparison with females, students living in Bali and Maluku in comparison 

with those who live in Java, and students who come from less educated parents in comparison with 

those who come from highly educated families. 

If we look deeper into the mechanism behind each of the effect, the three categories (interventions, 

practices, and contextual factors) actually overlap. For example, the meta-analysis shows a significant 

and large effect on teachers’ competence on learning outcomes. Looking deeper into the mechanism 

behind this effect, Trotsenburg et al. (2015) highlighted that the top 10% teachers are concentrated 

only in a few particular school that then we can consider those as the “good schools”. This fact 

leads us to a question whether there is a particular system or intervention being employed by the 

“good schools” that create a motivating atmosphere for the teachers to grow or there are particular 

contexts attached to the “good schools” that facilitate teacher to grow. Another example is how some 

students can have more motivation and better learning outcomes than others (Husaini & Wirawan, 

1998). In Husaini and Wirawan’s study, we can see the same patterns as teachers’ competence 

30 


where “good students” concentrated in the “good schools”. The study reported by Kasmaja (2016) 

suggested that particular classroom practices and interventions can increase students’ motivation. 

On the other hand, role models that the students have in their home environment are also highly 

correlated with their aspiration and motivation (Suryadi, 1992) libraries and library books, and other 

learning equipment. School management and teacher performance have also been emphasized. 

The need to improve the quality of primary education was reaffirmed in 1986 and is included in 

the Fifth Five-Year Development Plan. The aim of this study was to produce information and policy 

recommendations on how to improve primary schools. This study examined communities and schools 

to estimate primary school attendance, identify factors that influence participation, measure sixthgrade 

achievement, gauge the progress in different provinces, rank schools by achievement, and 

identify factors responsible for differential achievement in schools. Four major research questions 

were posed in this study: (1. These findings are also in line with numerous studies undertaken in 

other developing countries. A synthesis of reviews conducted by Evans and Popova (2015) also 

highlighted that the same intervention may provide different results depending on effectiveness 

of implementation across different contexts. The interaction between interventions, practices, and 

contextual factors is summarised in the picture below. 

Figure 3. Summary of interaction among interventions, practices, and contextual factors 

Direct relationship with learning outcome 

Indirect relationship with learning outcome 

31 


QUALITY OF EDUCATION RESEARCH IN INDONESIA 

This review not only mapped interventions, practices, and other factors that positively and negatively 

linked to students’ learning outcomes, but also maps the quality of the studies that were obtained 

from a systematic search. Only nine studies identified from the depository of all screened studies did 

not suffer from methodological issues. The most common problem found in the studies is bias due to 

incomplete reporting (55%), such as unreported variables, and unclear or incomplete estimation or 

statistical report. The other common source of bias is that the author did not identify or try to tackle 

possible confounding factors that might contaminate the result (52%). 

Even though studies with low qualities were eliminated from the analysis, some medium quality 

studies were taken into account although they suffer from issues in terms of the rigorousness of 

selecting the counterfactual comparison in the experiment. Several studies employed a beforeafter 

without control design or only measured the outcomes in treated and control groups after the 

intervention was complete, without prior knowledge of groups being balanced before the treatment. 

With regards to the limitations of the individual primary studies, the findings therefore, must be 

interpreted with caution. 

The scarcity of high-quality educational research in Indonesia was also reported by a study that 

incorporate six systematic reviews in developing countries (Evans and Popova, 2016). The current 

review strongly suggests that further in-depth studies on learning quality should be undertaken in 

Indonesia. Moreover, only quantitative studies were considered for this meta-analysis, and thus, 

further systematic reviews should also take qualitative studies into account. 

Implication for education programming 

Synthesising the findings of this review, there is a strong evidence of the effect of using teaching 

strategies based on students’ needs and difficulties to improve learning outcomes. To be able 

to understand the problems in the classroom and to choose and implement the best-fit teaching 

strategy, our findings suggest that teachers should be equipped with sufficient comprehension of the 

subject matter and possess pedagogical knowledge as well as social skills. Supported by evidence 

from other factors, teaching practice should also include strategy to boosts students’ psychological 

aspects, such as educational aspiration, motivation, and self-regulation. 

With regards to support for teachers that leads to a better learning outcomes, evidence has shown 

that contractual-based teachers performed better than civil servant teachers. In addition to this, 

payment increase had no effect on teachers’ performance 6 . Generally, therefore, the current system 

of providing higher incentives to teacher has not worked to increase learning. Instead, we find that 

support from the principal, in the form of supervision in teaching, and policies that increase parent 

engagement with teacher had a larger effect to the increase of the quality of learning. 

6 

One explanation is that job security and wages policies are typically focused on medium- and long- term improvements 

to attract better teachers and compete with other sectors on the labour market. Therefore, the impact should be 

evaluated on new cohorts of teacher instead of the current in-service teacher. It is therefore not surprising these impact 

evaluations have not found any effect yet. 

32 


Implication for policy 

Drawing a connection between the evidence across the different categories, we can conclude that 

teachers’ ability to understand the problems and needs of students in the classroom, and to choose 

the best fitting strategies to address those needs are the keys to improving student learning. At the 

school level, our results show that supervision helped teachers to undertake appropriate teaching 

strategies in their classrooms. Finally, the current teacher incentive and employment policies do not 

seem to enhance the quality of teacher and learning in schools. Given that these policies require 

significant budget allocations, they need to be re-assessed or modified in order to achieve their 

intended purpose to increase student learning outcomes. 

Limitation of the review 

As discussed above, the reviewers found a very limited number of high quality studies and decided 

to include the medium quality studies to the review. Some medium quality studies did not employ a 

double blind experimental design and/or did not provide a complete report of the variables they had 

tested. In addition, some of the studies did not provide detail description of the population. Thus, it is 

important to bear in mind possible bias given by the medium quality studies. 

The other consideration is related to the systematic search process employed in this study. Despite 

various combination of search terms used in various electronic databases, university repositories, 

donors and research institutes’ websites or internal libraries, the review may still miss some donor 

funded project reports or other relevant resources. 

Based on the above limitation, caution must be applied in interpreting the effect sizes of the factors, 

as the finding may not be universally applicable. 

33 


References 

Asim, S., Chase, R. S., Dar, A., & Schmillen, A. (2015). Improving Education Outcomes in 

South Asia Findings from a Decade of Impact Evaluations (Policy Research Working 

Paper No. 7362). Washington DC. Retrieved from http://documents.worldbank.org/curated/ 

en/254411468188375892/pdf/WPS7362.pdf 

Breakspear, S. (2012). The Policy Impact of PISA: An Exploration of the Normative Effects of 

International Benchmarking in School System Performance (OECD Education Working Papers 

No. 71). Paris. http://doi.org/10.1787/5k9fdfqffr28-en 

Chang, M. C., Shaeffer, S., Al-Samarrai, S., Ragatz, A. B., de Ree, J., & Stevenson, R. (2014). 

Teacher Reform in Indonesia: The Role of Politics and Evidence in Policy Making - Mae Chu 

Chang, Samer Al-Samarrai, Andrew B. Ragatz, Sheldon Shaeffer, Joppe de Ree, Ritchie 

Stevenson - Google Books. Washington DC: The World Bank. 

de Ree, J., Muralidharan, K., Pradhan, M. P., & Rogers, F. H. (2015). Double for Nothing? Experimental 

Evidence on the Impact of an Unconditional Teacher Salary Increase on Student Performance in 

Indonesia. 

Depaepe, F., Verschaffel, L., & Kelchtermans, G. (2013). Pedagogical Content Knowledge: A 

Systematic Review of the Way in which the Concept has Pervaded Mathematics Educational 

Research. Teaching and Teacher Education, 34, 12–25. http://doi.org/10.1016/j.tate.2013.03.001 

Dersimonian, R., & Kacker, R. (2007). Random-effects Model for Meta-analysis of Clinical Trials: An 

update. Contemporary Clinical Trials, 28(2), 105–114. http://doi.org/10.1016/j.cct.2006.04.004 

Education at a Glance 2015: OECD Indicators. (2015). OECD Publishing. 

Evans, D., & Popova, A. (2015). What Really Works to Improve Learning in Developing Countries? 

An Analysis of Divergent Findings in Systematic Reviews. World Bank Policy Research Working 

Paper (Vol. 7203). Africa Region. Retrieved from http://papers.ssrn.com/abstract=2571082 

Hanushek, E. A., & Woessmann, L. (2009). Do Better Schools Lead to More Growth? Cognitive 

Skills, Economic Outcomes, and Causation (IZA Discussion Papers No. 4575). Bonn. Retrieved 

from http://hdl.handle.net/10419/36178 

Hattie, J. (2009). Visible Learning. Oxon, Oxforshire: Routledge. Retrieved from https://www.readpbn. 

com/pdf/Visible-Learning-A-Synthesis-of-Over-800-Meta-Analyses-Relating-to-Achievement- 

Sample.pdf 

34 


Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of Learning Skills Interventions on Student Learning: 

A Meta-Analysis. Review of Educational Research, 66(2), 99–136. Retrieved from http://www. 

jstor.org/stable/1170605 

HUSAINI, & Dr. Yapsir Gandi Wirawan, M. A. (1998). Hubungan Motivasi Belajar, NEM SD/MI dan Nilai 

STTB SD/MI Mata Pelajaran Matematika dengan Prestasi Belajar Mata Pelajaran Matematika 

Siswa SLTP N dan Siswa MTS Negeri Kelas II Catur Wulan I Tahun Pelajaran 1998/1999 di 

Kotamadia Banda Aceh (Relation between Motivation of Learning, Math National Exam Mark for 

Public/Islamic Elementary School and Math Learning Performance of Public/Junior High School 

Grade 2 Student in First Quarter of Fiscal Year 1998/1999 in Banda Aceh City). Retrieved from 

http://ugm.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwhV1LT4NAEN5UvRgvGjW-Mx 

d7si3yajn0YG2bEoU0gYN6IQNsWyzQRiCN_9qf4A5U03ioJ8IwO5vssDvMMPMNY4rclBp_zgR 

DNXxdNtQJttsoLLakhdzQFERDOEmTeyolfh0r1pvy4qpWjf3E93keNotpQqKFakr8wBI0IlmEXeH 

tc8oQw9Srsi3rGfXt-aX2KyI 

Johnstone, J. N. (1983). Out-of-School Factors and Educational Achievement in Indonesia. 

Comparative Education Review. 

Kasmaja, H. (2016). Efektivitas Implementasi Metode Hypnoteaching Untuk Meningkatkan Motivasi 

dan Hasil Belajar Matematika pada Siswa SMP NEGERI (The effectivity of the Implementation of 

Hypnoteaching Method to Improve Motivation and Math Learning Outcome for Public Junior High 

School Students). Journal of Educational Science and Technology (EST), 2(1), 33–45. 

Kementrian Pendidikan dan Kebudayaan (The Ministry of Education and Culture). (2015). Rencana 

Strategis Kementrian Pendidikan dan Kebudayaan 2015-2019. Jakarta: Kementrian Pendidikan 

dan Kebudayaan (Strategic Planning of the Ministry of Education and Culture 2015-2019). 

Retrieved from http://luk.staff.ugm.ac.id/atur/RenstraKemdikbud2015-2019.pdf 

Krishnaratne, S., White, H., & Carpenter, E. (2013). Quality education for all children? What works 

in education in developing countries (3ie Working Paper No. 20). New Delhi. Retrieved from 

www.3ieimpact.org 

Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-analysis. CA. Retrieved from http://www. 

metafor-project.org/doku.php/analyses:lipsey2001 

Masino, S., & Niño-Zarazúa, M. (2015). What Works to Improve the Quality of Student Learning in 

Developing Countries? (Wider Working Paper No. 033). Helsinki. Retrieved from http://ssrn.com/ 

abstract=2579699 

35 


Mcewan, P. J., Lucas, A., O ’gara, C., Perez, M., Scheid, P., Schmidt, D., & Thornton, R. (2013). 

Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized 

Experiments. Review of Educational Research, 85(3), 353–394. Retrieved from www. 

patrickmcewan.net/meta. 

Miller, R. B., & Brickman, S. J. (2004). A Model of Future-Oriented Motivation and Self-Regulation. 

Educational Psychology Review, 16(7), 954–969. Retrieved from http://www.prospectivepsych. 

org/sites/www.prospectivepsych.org/files/pictures/Miller-Brickman_Model-of-future-orientedmotivation-and-self-regulation-2004.pdf 

Pradhan, M., Suryadarma, D., Beatty, A., Wong, M., Gaduh, A., Alisjahbana, A., & Artha, R. P. (2014). 

Improving Educational Quality through Enhancing Community Participation: Results from a 

Randomized Field Experiment in Indonesia. American Economic Journal. Applied Economics, 

6(2), 105–126. http://doi.org/http://dx.doi.org/10.1257/app.6.2.105 

Snilstveit, B., Stevenson, J., Menon, R., Phillips, D., Gallagher, E., Geleen, M., … Jimenez, E. (2016). 

Systematic Review Summary. The impact of Education Programmes on Learning and School 

Participation in Low-and Middle-income countries. London. Retrieved from http://www.3ieimpact. 

org/media/filer_public/2016/09/20/srs7-education-report.pdf 

Suryadarma, D., Suryahadi, A., Sumarto, S., Rogers, F. H., & Halsey Rogers, F. Improving Student 

Performance in Public Primary Schools in Developing Countries: Evidence from Indonesia, 14 

Education Economics 401–429 (2006). http://doi.org/10.1080/09645290600854110 

Suryadi, A. (1992). Improving the Educational Quality of Primary Schools. Educational Policy and 

Planning Project. A Government of Indonesia-USAID Project. 

The Joanna Briggs Institute. (2014). Joanna Briggs Institute Reviewers’ Manual 2014 Edition. 

(Professor Alan Pearson AM, Ed.) (1st ed.). Adelaide: the Joanna Briggs Institute. Retrieved from 

https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf 

van Bergen, E., van Zuijen, T., Bishop, D., & de Jong, P. F. (2016). Why are Home Literacy 

Environment and Children’s Reading Skills Associated? What Parental Skills Reveal? Reading 

Research Quarterly, 0(0), 1–14. http://doi.org/10.1002/rrq.160 

van Trotsenburg, A., Chaves, R. A., Costin, C., Patinos, H., & Ragatz, A. (2015). Teacher Certification 

and beyond. An Empirical Evaluation of the Teacher Certification Program and Education Quality 

Improvements in Indonesia. Jakarta. 

Vernez, G., Karam, R., & Marshall, J. H. (2012). Implementation of School-Based Management in 

Indonesia. Monograph. RAND Corporation. 

36 


37 


Ratu Plaza Office Tower 19th Floor, 

Jl. Jend. Sudirman Kav 9, 

Jakarta Pusat, 10270 

Indonesia 

Tel : (+6221) 720 6616 ext. 304 

Fax : (+6221) 720 6616 

info@inovasi.or.id 

www.facebook.com/InovasiPendidikanAIP

Interventions, Practices and Contextual Factors Linked to Indonesian Students’ Literacy and Numeracy Outcomes: A Systematic Review

Transform your PDFs into Flipbooks and boost your revenue!

Delete template?

Save as template ?