TY - JOUR A1 - Wenger, Marina A1 - Lüdtke, Oliver A1 - Brunner, Martin T1 - Übereinstimmung, Variabilität und Reliabilität von Schülerurteilen zur Unterrichtsqualität auf Schulebene T1 - Interrater agreement, variability and reliability of student ratings of instructional quality at the school-level BT - Ergebnisse aus 81 Ländern BT - Results from 81 countries JF - Zeitschrift für Erziehungswissenschaft N2 - Für die Analyse der Unterrichtsqualität von Schulen durch Schülerurteile sollten drei Voraussetzungen erfüllt sein: (1) eine angemessene Übereinstimmung der Schülerurteile innerhalb der Schulen, (2) systematische Variabilität der Schülerurteile zwischen Schulen, (3) ein ausreichendes Maß an Reliabilität der aggregierten Urteile. Diese Studie untersucht mit internationalen PISA-Daten (Zyklen 2000–2012; 81 Länder, über 55.300 Schulen, über 1,3 Millionen 15-Jährige), inwiefern dies für Indikatoren der Qualitätsdimensionen des Unterrichts (Klassenführung, Kognitive Aktivierung, Konstruktive Unterstützung) zutrifft. Dafür bestimmten wir das Übereinstimmungsmaß rWG(J) sowie die Intraklassenkorrelationen ICC(1) und ICC(2). Es zeigte sich, dass (1) die Mehrzahl der Unterrichtsmerkmale eine moderate oder starke Übereinstimmung in Schulen aufwies, (2) sich Unterrichtsmerkmale aus Sicht der Schülerschaft systematisch zwischen Schulen unterschieden, jedoch (3) die Reliabilität der aggregierten Schülerurteile in vielen Ländern nicht ausreichte. Die Ergebnisse diskutieren wir vor dem Hintergrund von Konventionen zur Beurteilung der Übereinstimmung, Variabilität und Reliabilität auf Schulebene. N2 - Using student ratings to assess instructional quality of schools should fulfill three requirements: (1)an appropriate level of inter-rater agreement within schools, (2)systematic variance of student ratings between schools, (3)an adequate reliability level of aggregated student ratings. Using international PISA-data (2000-2012; 81countries, over 55,300 schools, over 1.3million 15-year olds) this study investigated how these requirements were met regarding indicators of instructional quality (classroom management, cognitive activation, individual learning support). We computed the interrater agreement index r(WG(J)), as well as the intraclass correlations ICC(1) and ICC(2). Our results showed that (1)student ratings demonstrated amoderate or strong level of agreement for most indicators of instructional quality and (2)instructional quality assessed by students varied systematically between schools. Yet, (3)reliability of aggregated student ratings was not sufficient in many countries. We discuss these results regarding conventions to evaluate agreement, variability, and reliability of student ratings at the school level. KW - Instructional quality KW - Student ratings KW - PISA Y1 - 2018 U6 - https://doi.org/10.1007/s11618-018-0813-3 SN - 1434-663X SN - 1862-5215 VL - 21 IS - 5 SP - 929 EP - 950 PB - Springer CY - Wiesbaden ER - TY - JOUR A1 - Brunner, Martin A1 - Keller, Ulrich A1 - Wenger, Marina A1 - Fischbach, Antoine A1 - Lüdtke, Oliver T1 - Between-School Variation in Students' Achievement, Motivation, Affect, and Learning Strategies BT - Results from 81 Countries for Planning Group-Randomized Trials in Education JF - Journal of research on educational effectiveness / Society for Research on Educational Effectiveness (SREE) N2 - To plan group-randomized trials where treatment conditions are assigned to schools, researchers need design parameters that provide information about between-school differences in outcomes as well as the amount of variance that can be explained by covariates at the student (L1) and school (L2) levels. Most previous research has offered these parameters for U.S. samples and for achievement as the outcome. This paper and the online supplementary materials provide design parameters for 81 countries in three broad outcome categories (achievement, affect and motivation, and learning strategies) for domain-general and domain-specific (mathematics, reading, and science) measures. Sociodemographic characteristics were used as covariates. Data from representative samples of 15-year-old students stemmed from five cycles of the Programme for International Student Assessment (PISA; total number of students/schools: 1,905,147/70,098). Between-school differences as well as the amount of variance explained at L1 and L2 varied widely across countries and educational outcomes, demonstrating the limited generalizability of design parameters across these dimensions. The use of the design parameters to plan group-randomized trials is illustrated. KW - student achievement KW - motivation KW - affect KW - learning styles KW - intraclass correlation KW - large-scale assessment KW - multilevel models KW - design parameters Y1 - 2017 U6 - https://doi.org/10.1080/19345747.2017.1375584 SN - 1934-5747 SN - 1934-5739 VL - 11 IS - 3 SP - 452 EP - 478 PB - Routledge, Taylor & Francis Group CY - Abingdon ER - TY - JOUR A1 - Wenger, Marina A1 - Gärtner, Holger A1 - Brunner, Martin T1 - To what extent are characteristics of a school's student body, instructional quality, school quality, and school achievement interrelated? JF - School effectiveness and school improvement N2 - The aim of educational policy should be to provide a good education to all students. Thus, a key question arises regarding the extent to which key characteristics of school composition (proportion of students with migration background, socioeconomic status [SES], prior school achievement, and achievement heterogeneity), instructional quality, school quality, and later school achievement are interrelated. The present study addressed this research question by examining school inspection data, official school statistics, and large-scale achievement data from all primary schools in Berlin, Germany (N = 343). The results of correlation and path analyses showed that school composition (average SES, average prior school achievement) predicted components of instructional quality (SES: classroom management, cognitive activation; achievement: cognitive activation, individual learning support). The relation between school composition characteristics and most components of school quality was close to zero. Contrary to our expectations, only the effect of school SES on later achievement was mediated by instructional quality. KW - school composition KW - instructional quality KW - school quality Y1 - 2020 U6 - https://doi.org/10.1080/09243453.2020.1754243 SN - 0924-3453 SN - 1744-5124 VL - 31 IS - 4 SP - 548 EP - 575 PB - Routledge, Taylor & Francis Group CY - Abingdon ER - TY - GEN A1 - Brunner, Martin A1 - Keller, Ulrich A1 - Wenger, Marina A1 - Fischbach, Antoine A1 - Lüdtke, Oliver T1 - Between-school variation in students' achievement, motivation, affect, and learning strategies BT - results from 81 countries for planning group-randomized trials in education T2 - Postprints der Universität Potsdam : Humanwissenschaftliche Reihe N2 - To plan group-randomized trials where treatment conditions are assigned to schools, researchers need design parameters that provide information about between-school differences in outcomes as well as the amount of variance that can be explained by covariates at the student (L1) and school (L2) levels. Most previous research has offered these parameters for U.S. samples and for achievement as the outcome. This paper and the online supplementary materials provide design parameters for 81 countries in three broad outcome categories (achievement, affect and motivation, and learning strategies) for domain-general and domain-specific (mathematics, reading, and science) measures. Sociodemographic characteristics were used as covariates. Data from representative samples of 15-year-old students stemmed from five cycles of the Programme for International Student Assessment (PISA; total number of students/schools: 1,905,147/70,098). Between-school differences as well as the amount of variance explained at L1 and L2 varied widely across countries and educational outcomes, demonstrating the limited generalizability of design parameters across these dimensions. The use of the design parameters to plan group-randomized trials is illustrated. T3 - Zweitveröffentlichungen der Universität Potsdam : Humanwissenschaftliche Reihe - 465 KW - student achievement KW - motivation KW - affect KW - learning styles KW - intraclass correlation KW - large-scale assessment KW - multilevel models KW - design parameters Y1 - 2018 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:kobv:517-opus4-412662 IS - 465 ER -