TY - JOUR A1 - Wenger, Marina A1 - Lüdtke, Oliver A1 - Brunner, Martin T1 - Übereinstimmung, Variabilität und Reliabilität von Schülerurteilen zur Unterrichtsqualität auf Schulebene T1 - Interrater agreement, variability and reliability of student ratings of instructional quality at the school-level BT - Ergebnisse aus 81 Ländern BT - Results from 81 countries JF - Zeitschrift für Erziehungswissenschaft N2 - Für die Analyse der Unterrichtsqualität von Schulen durch Schülerurteile sollten drei Voraussetzungen erfüllt sein: (1) eine angemessene Übereinstimmung der Schülerurteile innerhalb der Schulen, (2) systematische Variabilität der Schülerurteile zwischen Schulen, (3) ein ausreichendes Maß an Reliabilität der aggregierten Urteile. Diese Studie untersucht mit internationalen PISA-Daten (Zyklen 2000–2012; 81 Länder, über 55.300 Schulen, über 1,3 Millionen 15-Jährige), inwiefern dies für Indikatoren der Qualitätsdimensionen des Unterrichts (Klassenführung, Kognitive Aktivierung, Konstruktive Unterstützung) zutrifft. Dafür bestimmten wir das Übereinstimmungsmaß rWG(J) sowie die Intraklassenkorrelationen ICC(1) und ICC(2). Es zeigte sich, dass (1) die Mehrzahl der Unterrichtsmerkmale eine moderate oder starke Übereinstimmung in Schulen aufwies, (2) sich Unterrichtsmerkmale aus Sicht der Schülerschaft systematisch zwischen Schulen unterschieden, jedoch (3) die Reliabilität der aggregierten Schülerurteile in vielen Ländern nicht ausreichte. Die Ergebnisse diskutieren wir vor dem Hintergrund von Konventionen zur Beurteilung der Übereinstimmung, Variabilität und Reliabilität auf Schulebene. N2 - Using student ratings to assess instructional quality of schools should fulfill three requirements: (1)an appropriate level of inter-rater agreement within schools, (2)systematic variance of student ratings between schools, (3)an adequate reliability level of aggregated student ratings. Using international PISA-data (2000-2012; 81countries, over 55,300 schools, over 1.3million 15-year olds) this study investigated how these requirements were met regarding indicators of instructional quality (classroom management, cognitive activation, individual learning support). We computed the interrater agreement index r(WG(J)), as well as the intraclass correlations ICC(1) and ICC(2). Our results showed that (1)student ratings demonstrated amoderate or strong level of agreement for most indicators of instructional quality and (2)instructional quality assessed by students varied systematically between schools. Yet, (3)reliability of aggregated student ratings was not sufficient in many countries. We discuss these results regarding conventions to evaluate agreement, variability, and reliability of student ratings at the school level. KW - Instructional quality KW - Student ratings KW - PISA Y1 - 2018 U6 - https://doi.org/10.1007/s11618-018-0813-3 SN - 1434-663X SN - 1862-5215 VL - 21 IS - 5 SP - 929 EP - 950 PB - Springer CY - Wiesbaden ER - TY - JOUR A1 - Brunner, Martin A1 - Keller, Ulrich A1 - Wenger, Marina A1 - Fischbach, Antoine A1 - Lüdtke, Oliver T1 - Between-School Variation in Students' Achievement, Motivation, Affect, and Learning Strategies BT - Results from 81 Countries for Planning Group-Randomized Trials in Education JF - Journal of research on educational effectiveness / Society for Research on Educational Effectiveness (SREE) N2 - To plan group-randomized trials where treatment conditions are assigned to schools, researchers need design parameters that provide information about between-school differences in outcomes as well as the amount of variance that can be explained by covariates at the student (L1) and school (L2) levels. Most previous research has offered these parameters for U.S. samples and for achievement as the outcome. This paper and the online supplementary materials provide design parameters for 81 countries in three broad outcome categories (achievement, affect and motivation, and learning strategies) for domain-general and domain-specific (mathematics, reading, and science) measures. Sociodemographic characteristics were used as covariates. Data from representative samples of 15-year-old students stemmed from five cycles of the Programme for International Student Assessment (PISA; total number of students/schools: 1,905,147/70,098). Between-school differences as well as the amount of variance explained at L1 and L2 varied widely across countries and educational outcomes, demonstrating the limited generalizability of design parameters across these dimensions. The use of the design parameters to plan group-randomized trials is illustrated. KW - student achievement KW - motivation KW - affect KW - learning styles KW - intraclass correlation KW - large-scale assessment KW - multilevel models KW - design parameters Y1 - 2017 U6 - https://doi.org/10.1080/19345747.2017.1375584 SN - 1934-5747 SN - 1934-5739 VL - 11 IS - 3 SP - 452 EP - 478 PB - Routledge, Taylor & Francis Group CY - Abingdon ER - TY - JOUR A1 - Wenger, Marina A1 - Gärtner, Holger A1 - Brunner, Martin T1 - To what extent are characteristics of a school's student body, instructional quality, school quality, and school achievement interrelated? JF - School effectiveness and school improvement N2 - The aim of educational policy should be to provide a good education to all students. Thus, a key question arises regarding the extent to which key characteristics of school composition (proportion of students with migration background, socioeconomic status [SES], prior school achievement, and achievement heterogeneity), instructional quality, school quality, and later school achievement are interrelated. The present study addressed this research question by examining school inspection data, official school statistics, and large-scale achievement data from all primary schools in Berlin, Germany (N = 343). The results of correlation and path analyses showed that school composition (average SES, average prior school achievement) predicted components of instructional quality (SES: classroom management, cognitive activation; achievement: cognitive activation, individual learning support). The relation between school composition characteristics and most components of school quality was close to zero. Contrary to our expectations, only the effect of school SES on later achievement was mediated by instructional quality. KW - school composition KW - instructional quality KW - school quality Y1 - 2020 U6 - https://doi.org/10.1080/09243453.2020.1754243 SN - 0924-3453 SN - 1744-5124 VL - 31 IS - 4 SP - 548 EP - 575 PB - Routledge, Taylor & Francis Group CY - Abingdon ER - TY - GEN A1 - Brunner, Martin A1 - Keller, Ulrich A1 - Wenger, Marina A1 - Fischbach, Antoine A1 - Lüdtke, Oliver T1 - Between-school variation in students' achievement, motivation, affect, and learning strategies BT - results from 81 countries for planning group-randomized trials in education T2 - Postprints der Universität Potsdam : Humanwissenschaftliche Reihe N2 - To plan group-randomized trials where treatment conditions are assigned to schools, researchers need design parameters that provide information about between-school differences in outcomes as well as the amount of variance that can be explained by covariates at the student (L1) and school (L2) levels. Most previous research has offered these parameters for U.S. samples and for achievement as the outcome. This paper and the online supplementary materials provide design parameters for 81 countries in three broad outcome categories (achievement, affect and motivation, and learning strategies) for domain-general and domain-specific (mathematics, reading, and science) measures. Sociodemographic characteristics were used as covariates. Data from representative samples of 15-year-old students stemmed from five cycles of the Programme for International Student Assessment (PISA; total number of students/schools: 1,905,147/70,098). Between-school differences as well as the amount of variance explained at L1 and L2 varied widely across countries and educational outcomes, demonstrating the limited generalizability of design parameters across these dimensions. The use of the design parameters to plan group-randomized trials is illustrated. T3 - Zweitveröffentlichungen der Universität Potsdam : Humanwissenschaftliche Reihe - 465 KW - student achievement KW - motivation KW - affect KW - learning styles KW - intraclass correlation KW - large-scale assessment KW - multilevel models KW - design parameters Y1 - 2018 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:kobv:517-opus4-412662 IS - 465 ER - TY - JOUR A1 - Wenger, Marina A1 - Gärtner, Holger A1 - Brunner, Martin T1 - Wie verändert sich die Unterrichts- und Schulqualität, die Leistung und die Schülerzusammensetzung nach der Schulinspektionsdiagnose „erheblicher Entwicklungsbedarf“? T1 - How does instruction and school quality, school performance, and composition of the student body change after the school inspection diagnosis "considerable need for development"? JF - Zeitschrift für Erziehungswissenschaft N2 - Die Schulinspektion evaluiert Schulen mit dem Ziel der Qualitätssicherung von Unterrichts- und Schulqualität. Dies gilt insbesondere für Schulen, an denen „erheblicher Entwicklungsbedarf“ festgestellt wurde. Diese Schulen bekommen zusätzliche Unterstützung, erfahren aber auch zusätzlichen Druck durch diese Einordnung. Die weitere Entwicklung dieser Schulen ist bisher kaum erforscht. Diese Studie untersucht mit Daten der Schulinspektion, der amtlichen Statistik und Leistungsdaten von 333 Berliner Grundschulen Veränderungen in Indikatoren der Unterrichts- und Schulqualität, der Schulleistung, und der Zusammensetzung der Schülerschaft (SES und Anteil mit nicht-deutscher Herkunftssprache) nach der Diagnose „erheblicher Entwicklungsbedarf“. Die empirischen Analysen zeigten, dass sich bei diesen Schulen die Unterrichts- und Schulqualität nur geringfügig veränderte, sich der Leistungsabstand zu allen anderen Grundschulen nicht statistisch signifikant verringerte, und sich die Zusammensetzung der Schülerschaft hinsichtlich des sozioökonomischen Status (SES) nicht veränderte. Jedoch erhöhte sich der Anteil von Kindern mit nicht-deutscher Herkunftssprache statistisch signifikant. N2 - The School Inspectorate evaluates schools with the aim of quality assurance and quality development. This applies in particular to schools where "considerable need for development" has been identified. These schools receive additional support, but also experience additional pressure as a result. The further development of these schools has hardly been researched so far. Therefore, using data from the school inspectorate, official statistics and performance data from 333 Berlin primary schools, this study examines changes in indicators of instruction and school quality, school performance, and composition of the student body (SES and percentage of students from nonGerman-speaking homes) following the diagnosis "considerable need for development". The empirical analyses showed that at these schools instruction and school quality changed only slightly, the achievement gap did not decrease significantly in statistical terms (in comparison to all other primary schools), and the composition of the student body did not change with regard to socio-economic status (SES). However, a statistically significant increase of the proportion of students from non-German-speaking homes was observed. KW - school inspection KW - special measure KW - student performance KW - Schulinspektion KW - Entwicklungsbedarf KW - Schülerleistungen Y1 - 2022 U6 - https://doi.org/10.1007/s11618-022-01066-3 SN - 1434-663X SN - 1862-5215 VL - 25 IS - 4 SP - 1031 EP - 1058 PB - VS Verlag für Sozialwissenschaften ; Springer CY - Wiesbaden ER -