Efficient and exact computation of inclusion dependencies for data integration

  • Data obtained from foreign data sources often come with only superficial structural information, such as relation names and attribute names. Other types of metadata that are important for effective integration and meaningful querying of such data sets are missing. In particular, relationships among attributes, such as foreign keys, are crucial metadata for understanding the structure of an unknown database. The discovery of such relationships is difficult, because in principle for each pair of attributes in the database each pair of data values must be compared. A precondition for a foreign key is an inclusion dependency (IND) between the key and the foreign key attributes. We present with Spider an algorithm that efficiently finds all INDs in a given relational database. It leverages the sorting facilities of DBMS but performs the actual comparisons outside of the database to save computation. Spider analyzes very large databases up to an order of magnitude faster than previous approaches. We also evaluate in detail the effectivenessData obtained from foreign data sources often come with only superficial structural information, such as relation names and attribute names. Other types of metadata that are important for effective integration and meaningful querying of such data sets are missing. In particular, relationships among attributes, such as foreign keys, are crucial metadata for understanding the structure of an unknown database. The discovery of such relationships is difficult, because in principle for each pair of attributes in the database each pair of data values must be compared. A precondition for a foreign key is an inclusion dependency (IND) between the key and the foreign key attributes. We present with Spider an algorithm that efficiently finds all INDs in a given relational database. It leverages the sorting facilities of DBMS but performs the actual comparisons outside of the database to save computation. Spider analyzes very large databases up to an order of magnitude faster than previous approaches. We also evaluate in detail the effectiveness of several heuristics to reduce the number of necessary comparisons. Furthermore, we generalize Spider to find composite INDs covering multiple attributes, and partial INDs, which are true INDs for all but a certain number of values. This last type is particularly relevant when integrating dirty data as is often the case in the life sciences domain - our driving motivation.show moreshow less

Download full text files

Export metadata

  • Export Bibtex
  • Export RIS
  • Export XML

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Jana Bauckmann, Ulf Leser, Felix Naumann
URN:urn:nbn:de:kobv:517-opus-41396
Series (Serial Number):Technische Berichte des Hasso-Plattner-Instituts für Softwaresystemtechnik an der Universität Potsdam (34)
Publisher:Universitätsverlag Potsdam
Place of publication:Potsdam
Document Type:Book
Language:English
Date of Publication (online):2010/04/15
Year of Completion:2010
Publishing Institution:Universität Potsdam
Release Date:2010/04/15
Tag:Datenanalyse; Datenintegration; Metadatenentdeckung; Metadatenqualität; Schemaentdeckung
data integration; data profiling; metadata discovery; metadata quality; schema discovery
Pagenumber:36
RVK - Regensburg Classification:ST 230
Organizational units:Extern / Extern
An-Institute / Hasso-Plattner-Institut für Softwaresystemtechnik GMBH
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Publication Way:Universitätsverlag Potsdam
Licence (German):License LogoKeine Nutzungslizenz vergeben - es gilt das deutsche Urheberrecht
Notes extern:
In Printform erschienen im Universitätsverlag Potsdam:

Bauckmann, Jana: Efficient and exact computation of inclusion dependencies for data integration / Jana Bauckmann, Ulf Leser, Felix Naumann. - Potsdam : Universitätsverlag Potsdam, 2010. - 36 S. : graph. Darst.
(Technische Berichte des Hasso-Plattner-Instituts für Softwaresystemtechnik an der Universität Potsdam ; 34)
ISSN 1613-5652
ISBN 978-3-86956-048-9
--> bestellen