Information integration (II) is the process of merging information from heterogeneous sources with different conceptual, contextual, and typographical representations. It is a critical aspect of data management that enables organizations to consolidate data from various sources, such as databases, legacy systems, web services, and flat files, into a coherent and unified dataset. This process is essential for various applications, including data mining, data analysis, business intelligence (BI), and decision-making.
The integration of information often involves handling data with varying structures, formats, and origins, and it may include both structured and unstructured data. The goal is to create a single, comprehensive view of the data that is accurate, up-to-date, and readily available for analysis and reporting.
Information integration is closely related to data integration, which is the process of combining data from multiple sources into a centralized location, such as a data warehouse. This centralized data repository should be flexible enough to accommodate different types of data and support analytical use cases.
Technologies and methods used in information integration include deduplication, string metrics for detecting similar text across different data sources, and causal estimates of outcomes based on models of the sources. These technologies help reduce redundancy and improve the quality of the integrated data.
Information integration is beneficial for businesses as it allows for smarter decision-making, improved customer experiences, cost reduction, increased revenue potential, and enhanced efficiency. By having a unified view of data, organizations can ensure that all stakeholders have access to the same information, which fosters collaboration and innovation.
In the context of AI, information integration is crucial for combining data from multiple sources to build models that reason about the data and make inferences about missing or uncertain information. This process supports AI applications that require a comprehensive view of data to function effectively.
Overall, information integration is a foundational element for organizations looking to leverage their data assets fully, enabling them to become more data-driven and competitive in their respective industries.
Some of the common techniques used for information integration include:
Data Consolidation — This technique involves combining data from multiple sources into a single database or data warehouse. The data is extracted, transformed to match the target system's format, and then loaded into the central repository.
Data Federation — This method provides a virtual view of the integrated data, allowing users to access and retrieve data from multiple sources as if it were from a single source. It does not require physical movement or duplication of data.
Data Propagation — Data propagation involves copying data from one location to another using applications or programmed processes. This can be done in real-time or on a scheduled basis.
Middleware Data Integration — Middleware, such as an enterprise service bus (ESB), is used to connect different systems and allow them to communicate with each other, facilitating the movement and transformation of data.
Data Warehousing — This is a central repository where data from various sources is stored after being cleaned, transformed, and standardized. It supports analytical reporting and structured queries.
Manual Data Integration — In this approach, engineers manually write code to move and manipulate data. This method is often used for smaller projects or when a high degree of customization is required.
Extract, Transform, Load (ETL) — This is a traditional data integration process where data is extracted from the source, transformed into the required format, and then loaded into the target system.
Extract, Load, Transform (ELT) — Similar to ETL, but the transformation occurs after the data is loaded into the data warehouse, taking advantage of the processing power of modern data storage systems.
Application-Based Integration — This method involves linking applications directly so they can move and transform data based on event triggers.
Data Virtualization — This technique creates a virtual database that provides a unified view of data from different sources, allowing end-users to access and analyze data without requiring technical details about the source data.
Real-Time Data Streaming — This approach involves continuously capturing and integrating data as it is generated, allowing for real-time analysis and decision-making.
These techniques can be used individually or in combination, depending on the specific requirements of the integration project, such as the volume of data, the need for real-time processing, and the complexity of the data sources.
Information integration and data integration are related concepts, but they are not identical. Information integration refers to the process of merging information from heterogeneous sources with different conceptual, contextual, and typographical representations. It is broader in scope and can include the integration of unstructured or semi-structured resources, such as textual knowledge representations and information fusion, which aims to combine information into a new set of information to reduce redundancy.
Data integration, on the other hand, is a subset of information integration that specifically deals with the consolidation of data from different sources. It focuses on creating a unified view of data residing in various systems, applications, cloud platforms, and sources to facilitate analysis, reporting, and forecasting without the risks associated with duplication, error, fragmentation, or disparate data formats. Data integration typically involves replicating data into a data warehouse for analytics and reporting and usually runs in batches at a set cadence.
One key difference is that data integration does not necessarily require knowledge of business processes; it primarily needs data sources and a destination, such as a data warehouse or data lake. The flow of data in data integration is one-way, from sources to an analytics repository. Application integration, which is often discussed alongside data integration, involves connecting different applications and is performed in (near) real-time to complete a business process or transaction.
While both information integration and data integration aim to make data more accessible and functional, information integration has a broader remit that includes various forms of data and knowledge, whereas data integration is more focused on the technical process of consolidating data for analytical purposes.
Because information integration spans conceptual, contextual, and typographical mismatches, not just physical data movement, the practices below focus on reconciling meaning across sources rather than just consolidating them.
Before choosing a technique, identify where sources disagree on meaning: the same field name used for different concepts, or the same concept represented in incompatible formats. Reconciling this schema and semantic mismatch is the core work of information integration, distinct from the physical movement of data.
Federation or data virtualization suits queries that need a live view without duplicating data, while consolidation into a data warehouse suits recurring analytical workloads. Real-time streaming fits operational decisions; batch ETL or ELT fits periodic reporting. Choosing the wrong technique for the workload is a common source of stale or inconsistent integrated views.
Use string metrics and other entity-resolution methods to identify when records from different sources describe the same underlying entity. Without this step, a unified view still contains hidden duplicates and contradictions.
Track which source each piece of integrated information came from and how it was transformed. Provenance makes it possible to trace errors back to their origin and to judge how much confidence to place in an inferred or aggregated value.
Information integration is ongoing, not a one-time project. Revisit access controls, validation rules, and source coverage as new systems and data types are added, so the integrated view stays accurate as the underlying sources evolve.
More terms
Glossary term
Glossary term
Collaborate with your team on reliable Generative AI features.
Want expert guidance? Book a 1:1 onboarding session from your dashboard.