Refine
Year of publication
Document Type
- Article (20)
- Postprint (9)
- Doctoral Thesis (6)
Language
- English (35) (remove)
Is part of the Bibliography
- yes (35)
Keywords
- eye-tracking (35) (remove)
Institute
- Department Linguistik (20)
- Strukturbereich Kognitionswissenschaften (4)
- Department Psychologie (3)
- Department Sport- und Gesundheitswissenschaften (2)
- Department Erziehungswissenschaft (1)
- Department Grundschulpädagogik (1)
- Hasso-Plattner-Institut für Digital Engineering GmbH (1)
- Humanwissenschaftliche Fakultät (1)
- Institut für Romanistik (1)
- Strukturbereich Bildungswissenschaften (1)
The ZuCo benchmark on cross-subject reading task classification with EEG and eye-tracking data
(2023)
We present a new machine learning benchmark for reading task classification with the goal of advancing EEG and eye-tracking research at the intersection between computational language processing and cognitive neuroscience. The benchmark task consists of a cross-subject classification to distinguish between two reading paradigms: normal reading and task-specific reading. The data for the benchmark is based on the Zurich Cognitive Language Processing Corpus (ZuCo 2.0), which provides simultaneous eye-tracking and EEG signals from natural reading of English sentences. The training dataset is publicly available, and we present a newly recorded hidden testset. We provide multiple solid baseline methods for this task and discuss future improvements. We release our code and provide an easy-to-use interface to evaluate new approaches with an accompanying public leaderboard: .
It is a common finding that preschoolers have difficulties in identifying who is doing what to whom in non-canonical sentences, such as (object-verb-subject) OVS and passive sentences in German. This dissertation investigates how German monolingual and German-Italian simultaneous bilingual children process German OVS sentences in Study 1 and German passives in Study 2. Offline data (i.e., accuracy data) and online data (i.e., eye-gaze and pupillometry data) were analyzed to explore whether children can assign thematic roles during sentence comprehension and processing. Executive functions, language-internal and -external factors were investigated as potential predictors for children’s sentence comprehension and processing.
Throughout the literature, there are contradicting findings on the relation between language and executive functions. While some results show a bilingual cognitive advantage over monolingual speakers, others suggest there is no relationship between bilingualism and executive functions. If bilingual children possess more advanced executive function abilities than monolingual children, then this might also be reflected in a better performance on linguistic tasks. In the current studies monolingual and bilingual children were tested by means of two executive function tasks: the Flanker task and the task-switching paradigm. However, these findings showed no bilingual cognitive advantages and no better performance by bilingual children in the linguistic tasks. The performance was rather comparable between bilingual and monolingual children, or even better for the monolingual group. This may be due to cross-linguistic influences and language experience (i.e., language input and output). Italian was used because it does not syntactically overlap with the structure of German OVS sentences, and it only overlapped with one of the two types of sentence condition used for the passive study - considering the subject-(finite)verb alignment. The findings showed a better performance of bilingual children in the passive sentence structure that syntactically overlapped in the two languages, providing evidence for cross-linguistic influences.
Further factors for children’s sentence comprehension were considered. The parents’ education, the number of older siblings and language experience variables were derived from a language background questionnaire completed by parents. Scores of receptive vocabulary and grammar, visual and short-term memory and reasoning ability were measured by means of standardized tests. It was shown that higher German language experience by bilinguals correlates with better accuracy in German OVS sentences but not in passive sentences. Memory capacity had a positive effect on the comprehension of OVS and passive sentences in the bilingual group. Additionally, a role was played by executive function abilities in the comprehension of OVS sentences and not of passive sentences. It is suggested that executive function abilities might help children in the sentence comprehension task when the linguistic structures are not yet fully mastered.
Altogether, these findings show that bilinguals’ poorer performance in the comprehension and processing of German OVS is mainly due to reduced language experience in German, and that the different performance of bilingual children with the two types of passives is mainly due to cross-linguistic influences.
Exercise or not?
(2023)
Objective: Individuals’ decisions to engage in exercise are often the result of in-the-moment choices between exercise and a competing behavioral alternative. The purpose of this study was to investigate processes that occur in-the-moment (i.e., situated processes) when individuals are faced with the choice between exercise and a behavioral alternative during a computerized task. These were analyzed against the background of interindividual differences in individuals’ automatic valuation and controlled evaluation of exercise.
Method: In a behavioral alternatives task 101 participants were asked whether they would rather choose an exercise option or a behavioral alternative in 25 trials. Participants’ gaze behavior (first gaze and fixations) was recorded using eye-tracking. An exercise-specific affect misattribution procedure (AMP) was used to assess participants’ automatic valuation of exercise before the task. After the task, self-reported feelings towards exercise (controlled evaluation) and usual weekly exercise volume were assessed. Mixed effects models with random effects for subjects and trials were used for data analysis.
Results: Choosing exercise was positively correlated with individuals’ automatic valuation (r = 0.20, p = 0.05), controlled evaluation (r = 0.58, p < 0.001), and their weekly exercise volume (r = 0.43, p < 0.001). Participants showed no bias in their initial gaze or number of fixations towards the exercise or the non-exercise alternative. However, participants were 1.30 times more likely to fixate on the chosen alternative first and more frequently, but this gaze behavior was not related to individuals’ automatic valuation, controlled evaluation, or weekly exercise volume.
Conclusion: The results suggest that situated processes arising from defined behavioral alternatives may be independent of individuals’ general preferences. Despite one’s best general intention to exercise more, the choice of a non-exercise alternative behavior may seem more appealing in-the-moment and eventually be chosen. New psychological theories of health behavior change should therefore better consider the role of potentially conflicting alternatives when it comes to initiating physical activity or exercise.
Exercise or not?
(2023)
Objective: Individuals’ decisions to engage in exercise are often the result of in-the-moment choices between exercise and a competing behavioral alternative. The purpose of this study was to investigate processes that occur in-the-moment (i.e., situated processes) when individuals are faced with the choice between exercise and a behavioral alternative during a computerized task. These were analyzed against the background of interindividual differences in individuals’ automatic valuation and controlled evaluation of exercise.
Method: In a behavioral alternatives task 101 participants were asked whether they would rather choose an exercise option or a behavioral alternative in 25 trials. Participants’ gaze behavior (first gaze and fixations) was recorded using eye-tracking. An exercise-specific affect misattribution procedure (AMP) was used to assess participants’ automatic valuation of exercise before the task. After the task, self-reported feelings towards exercise (controlled evaluation) and usual weekly exercise volume were assessed. Mixed effects models with random effects for subjects and trials were used for data analysis.
Results: Choosing exercise was positively correlated with individuals’ automatic valuation (r = 0.20, p = 0.05), controlled evaluation (r = 0.58, p < 0.001), and their weekly exercise volume (r = 0.43, p < 0.001). Participants showed no bias in their initial gaze or number of fixations towards the exercise or the non-exercise alternative. However, participants were 1.30 times more likely to fixate on the chosen alternative first and more frequently, but this gaze behavior was not related to individuals’ automatic valuation, controlled evaluation, or weekly exercise volume.
Conclusion: The results suggest that situated processes arising from defined behavioral alternatives may be independent of individuals’ general preferences. Despite one’s best general intention to exercise more, the choice of a non-exercise alternative behavior may seem more appealing in-the-moment and eventually be chosen. New psychological theories of health behavior change should therefore better consider the role of potentially conflicting alternatives when it comes to initiating physical activity or exercise.
The other-race effect (ORE) can be described as difficulties in discriminating between faces of ethnicities other than one's own, and can already be observed at approximately 9 months of age. Recent studies also showed that infants visually explore same-and other-race faces differently. However, it is still unclear whether infants' looking behavior for same- and other-race faces is related to their face discrimination abilities. To investigate this question we conducted a habituation-dishabituation experiment to examine Caucasian 9-month-old infants' gaze behavior, and their discrimination of same- and other-race faces, using eye-tracking measurements. We found that infants looked longer at the eyes of same-race faces over the course of habituation, as compared to other-race faces. After habituation, infants demonstrated a clear other-race effect by successfully discriminating between same-race faces, but not other-race faces. Importantly, the infants' ability to discriminate between same-race faces significantly correlated with their fixation time towards the eyes of same-race faces during habituation. Thus, our findings suggest that for infants old enough to begin exhibiting the ORE, gaze behavior during habituation is related to their ability to differentiate among same-race faces, compared to other-race faces.
German orthography systematically marks all nouns (even other nominalized word classes) by capitalizing their first letter. It is often claimed that readers benefit from the uppercase-letter syntactic and semantic information, which makes the processing of sentences easier (e.g., Bock et al., 1985, 1989). In order to test this hypothesis, we asked 54 German readers to read single sentences systematically manipulated by a target word (N). In the experimental condition (EXP), we used semantic priming (in the following example: sick -> cold) in order to build up a strong expectation of a noun, which was actually an attribute for the following noun (N+1) (translated to English e.g., "The sick writer had a cold (N) nose (N+1) ..."). The sentences in the control condition were built analogously, but word N was purposefully altered (keeping word length and frequency constant) to make its interpretation as a noun extremely unlikely (e.g., "The sick writer had a blue (N) nose (N+1) ..."). In both conditions, the sentences were presented either following German standard orthography (Cap) or in lowercase spelling (NoCap). The capitalized nouns in the EXP/Cap condition should then prevent garden-path parsing, as capital letters can be recognized parafoveally. However, in the EXP/NoCap condition, we expected a garden-path effect on word N+1 affecting first-pass fixations and the number of regressions, as the reader realizes that word N is instead an adjective. As the control condition does not include a garden-path, we expected to find (small) effects of the violation of the orthographic rule in the CON/NoCap condition, but no garden-path effect. As a global result, it can be stated that reading sentences in which nouns are not marked by a majuscule slows a native German reader down significantly, but from an absolute point of view, the effect is small. Compared with other manipulations (e.g., transpositions or substitutions), a lowercase letter still represents the correct allograph in the correct position without affecting phonology. Furthermore, most German readers do have experience with other alphabetic writing systems that lack consistent noun capitalization, and in (private) digital communication lowercase nouns are quite common. Although our garden-path sentences did not show the desired effect, we found an indication of grammatical pre-processing enabled by the majuscule in the regularly spelled sentences: In the case of high noun frequency, we post hoc located parafovea-on-fovea effects, i.e., longer fixation durations, on the attributive adjective (word N). These benefits of capitalization could only be detected under specific circumstances. In other cases, we conclude that longer reading durations are mainly the result of disturbance in readers' habituation when the expected capitalization is missing.
Agrammatic speakers of languages with overt grammatical case show impaired use of the morphological cues to establish theta-role relations in sentences presented in non-canonical word orders. We analysed the effect of word order on the sentence comprehension of aphasic speakers of Basque, an ergative, free word order and head-final (SOV) language. Ergative languages such as Basque establish a one-to-one mapping of the thematic role and the case marker. We collected behavioural and gaze-fixation data while agrammatic speakers performed a picture-matching task with auditorily presented sentences with different word orders. We found that people with aphasia (PWA) had difficulties in assigning theta-roles in Theme-Agent order. This result is in line with processing accounts. Contrary to previous findings, our data do not suggest a systematic delay in the integration of morphological information in the PWA group, but strong reliance on the ergative case morphology and difficulties assigning thematic roles into the determiner phrases.
The aim of this dissertation was to conduct a larger-scale cross-linguistic empirical investigation of similarity-based interference effects in sentence comprehension.
Interference studies can offer valuable insights into the mechanisms that are involved in long-distance dependency completion.
Many studies have investigated similarity-based interference effects, showing that syntactic and semantic information are employed during long-distance dependency formation (e.g., Arnett & Wagers, 2017; Cunnings & Sturt, 2018; Van Dyke, 2007, Van Dyke & Lewis, 2003; Van Dyke & McElree, 2011). Nevertheless, there are some important open questions in the interference literature that are critical to our understanding of the constraints involved in dependency resolution.
The first research question concerns the relative timing of syntactic and semantic interference in online sentence comprehension. Only few interference studies have investigated this question, and, to date, there is not enough data to draw conclusions with regard to their time course (Van Dyke, 2007; Van Dyke & McElree, 2011).
Our first cross-linguistic study explores the relative timing of syntactic and semantic interference in two eye-tracking reading experiments that implement the study design used in Van Dyke (2007). The first experiment tests English sentences. The second, larger-sample experiment investigates the two interference types in German.
Overall, the data suggest that syntactic and semantic interference can arise simultaneously during retrieval.
The second research question concerns a special case of semantic interference: We investigate whether cue-based retrieval interference can be caused by semantically similar items which are not embedded in a syntactic structure.
This second interference study builds on a landmark study by Van Dyke & McElree (2006). The study design used in their study is unique in that it is able to pin down the source of interference as a consequence of cue overload during retrieval, when semantic retrieval cues do not uniquely match the retrieval target. Unlike most other interference studies, this design is able to rule out encoding interference as an alternative explanation. Encoding accounts postulate that it is not cue overload at the retrieval site but the erroneous encoding of similar linguistic items in memory that leads to interference (Lewandowsky et al., 2008; Oberauer & Kliegl, 2006). While Van Dyke & McElree (2006) reported cue-based retrieval interference from sentence-external distractors, the evidence for this effect was weak. A subsequent study did not show interference of this type (Van Dyke et al., 2014). Given these inconclusive findings, further research is necessary to investigate semantic cue-based retrieval interference.
The second study in this dissertation provides a larger-scale cross-linguistic investigation of cue-based retrieval interference from sentence-external items. Three larger-sample eye-tracking studies in English, German, and Russian tested cue-based interference in the online processing of filler-gap dependencies. This study further extends the previous research by investigating interference in each language under varying task demands (Logačev & Vasishth, 2016; Swets et al., 2008).
Overall, we see some very modest support for proactive cue-based retrieval interference in English. Unexpectedly, this was observed only under a low task demand. In German and Russian, there is some evidence against the interference effect. It is possible that interference is attenuated in languages with richer case marking.
In sum, the cross-linguistic experiments on the time course of syntactic and semantic interference from sentence-internal distractors support existing evidence of syntactic and semantic interference during sentence comprehension. Our data further show that both types of interference effects can arise simultaneously. Our cross-linguistic experiments investigating semantic cue-based retrieval interference from sentence-external distractors suggest that this type of interference may arise only in specific linguistic contexts.
Background: Comprehension of non-canonical sentences is frequently characterised by chance level performance in people with aphasia (PWA). Chance level performance has been interpreted as guessing, but online data does not support this rendering. It is still not clear whether the incorrect sentence processing is guided by the compensatory strategies that PWA might employ to overcome linguistic difficulties.Aims: We aim to study to what extent people with non-fluent aphasia are aware of their sentence comprehension deficits.Methods & Procedures: This study combined offline and online data to investigate the effect of word order and error-awareness on sentence comprehension in a group of PWA and non-brain damaged (NBD) participants. The offline tasks involved auditory sentence picture-matching immediately followed by a confidence rating (CR). Participants were asked to judge the perceived correctness of their previous answer. Online data consisted of eye-tracking.Outcomes & Results: Replicating previous findings, PWA had significantly worse comprehension of Theme-Agent order compared to Agent-Theme order sentences. Controls showed ceiling level sentence comprehension. CR was a poor predictor of response accuracy in PWA, but moderate-good in NBD. A total of 6.8% of judgements were classified as guessing by PWA. Post hoc gaze data analysis indicated that CR was a predictor of the fixation pattern during the presentation of the linguistic stimuli.Conclusions: Results suggest that PWA were mostly unaware of their sentence comprehension errors and did not consciously employ strategies to compensate for their difficulties.
With the spread of smart phones capable of taking high-resolution photos and the development of high-speed mobile data infrastructure, digital visual media is becoming one of the most important forms of modern communication. With this development, however, also comes a devaluation of images as a media form with the focus becoming the frequency at which visual content is generated instead of the quality of the content. In this work, an interactive system using image-abstraction techniques and an eye tracking sensor is presented, which allows users to experience diverting and dynamic artworks that react to their eye movement. The underlying modular architecture enables a variety of different interaction techniques that share common design principles, making the interface as intuitive as possible. The resulting experience allows users to experience a game-like interaction in which they aim for a reward, the artwork, while being held under constraints, e.g., not blinking. The co nscious eye movements that are required by some interaction techniques hint an interesting, possible future extension for this work into the field of relaxation exercises and concentration training.