From Data Warehousing to Data Integration Hub: The Role of ETL in Enterprise Data Transformation
Trinity Data Integration Lab
“Data is Business, and Business is Data”
In recent years, as enterprises deepen their data analysis capabilities, data transformation has become an inevitable trend. Enterprise data transformation is not just a slogan—the industry has proposed a clear direction: evolving from traditional data warehousing toward Data Middle Platforms, making the latter a prominent topic. The core concept of the Data Middle Platform is to create a centralized platform (positioned between frontend business services and backend databases) that manages data sources and applications through organizational restructuring and technologies such as data warehousing and data synchronization, supporting various business applications. In essence, the Data Middle Platform is a framework that integrates and governs cross-domain data within the enterprise, assembling it into services with business value.
Amid the rise of Data Middle Platforms, vendors have approached the topic from different perspectives, offering diverse solutions. This has led to confusion about the role of the platform and its distinction from data warehouses. Some even mistakenly believe that by building a Data Middle Platform using data virtualization technologies, data warehouses and the ETL architecture are no longer necessary. Therefore, clarifying the evolution of the Data Middle Platform is essential. Once this evolution is understood, these misconceptions dissolve, and the answers become self-evident.
In recent years, as enterprises deepen their data analysis capabilities, data transformation has become an inevitable trend. Enterprise data transformation is not just a slogan—the industry has proposed a clear direction: evolving from traditional data warehousing toward Data Middle Platforms, making the latter a prominent topic. The core concept of the Data Middle Platform is to create a centralized platform (positioned between frontend business services and backend databases) that manages data sources and applications through organizational restructuring and technologies such as data warehousing and data synchronization, supporting various business applications. In essence, the Data Middle Platform is a framework that integrates and governs cross-domain data within the enterprise, assembling it into services with business value.
Amid the rise of Data Middle Platforms, vendors have approached the topic from different perspectives, offering diverse solutions. This has led to confusion about the role of the platform and its distinction from data warehouses. Some even mistakenly believe that by building a Data Middle Platform using data virtualization technologies, data warehouses and the ETL architecture are no longer necessary. Therefore, clarifying the evolution of the Data Middle Platform is essential. Once this evolution is understood, these misconceptions dissolve, and the answers become self-evident.
The Evolution of the Data Middle Platform
In mid-2015, Jack Ma visited Finland’s Supercell and was inspired by its middle platform architecture. By the end of that year, Alibaba introduced its Data Middle Platform strategy to the industry. As the architecture continued to evolve, Alibaba announced a “dual middle platform” strategy in 2018—separating Business and Data Middle Platforms. Subsequently, the Business Middle Platform was further subdivided into Mobile, Technology, Risk, and R&D Efficiency Middle Platforms. By 2020, the architecture had evolved into a “fragmented middle platform” ecosystem. Among these, the Data Middle Platform—composed of data processing architecture and data warehousing—was responsible for providing data to both frontend applications and the Business Middle Platform.
Throughout Alibaba’s practical evolution from “Data Middle Platform” 🢂 “Business–Data Dual Platforms” 🢂 “Fragmented Middle Platforms,” the role of the data warehouse remained constant—as a provider and integrator of data. Why, then, has enterprise data transformation not displaced the role of the “Data Warehouse–ETL” foundation?
Throughout Alibaba’s practical evolution from “Data Middle Platform” 🢂 “Business–Data Dual Platforms” 🢂 “Fragmented Middle Platforms,” the role of the data warehouse remained constant—as a provider and integrator of data. Why, then, has enterprise data transformation not displaced the role of the “Data Warehouse–ETL” foundation?
Practical Considerations of Data Virtualization
From a conceptual standpoint, data virtualization allows users to interact with multiple databases through a single platform, building a virtual layer to directly access, transform, and integrate data without persisting it—supporting applications with high flexibility. However, while theoretically sound, this approach faces several real-world challenges:
- Database Performance Impact
The virtual layer must establish numerous database connections to meet the Data Middle Platform’s real-time access and processing needs. This imposes significant load on the existing databases, which were originally designed for specific business use cases. The added burden of data platform workloads can degrade performance and impact overall system stability.
- Increased Resource Costs
The ideal of “non-persistent, flexible use” in virtualization leads to inefficiencies when data has not undergone preprocessing. This significantly increases system resource usage—both in compute and network transmission—causing waste and potential bottlenecks.
For example, if personal data requires masking, the same masking process must be repeated every time the data is retrieved. Similarly, if a dataset requires cleaning or transformation (e.g., deriving age or zodiac signs from birth dates), the computation must be re-executed each time it’s accessed.
Thus, the use of data virtualization must be carefully planned. For microservices that do not significantly increase system resource loads or affect the operation of existing systems, direct database connections may be appropriate. For others, a preprocessing mechanism is essential—connecting them instead to a “Data Warehouse–ETL” structure.
ETL Operations and Data Integration Are the Cornerstone of Enterprise Data Assets
In summary, to support the diverse application needs of enterprise data assets, most data must undergo preprocessing before it can be utilized. This includes:
- Data Security (masking/encryption, access control)
- Data Quality (cleansing, transformation, validation)
- Aggregation/Statistics (weekly/monthly summaries, categorization)
No matter how enterprise data transformation evolves, some form of data staging is essential. Whether implemented through ETL or streaming technologies, and whether structured as a data warehouse or data lake, the ultimate goal is to consolidate into one or more data asset warehouses. The “Data Warehouse–ETL” foundation continues to evolve, but it will not disappear.
Let’s revisit the business intelligence (BI) model from nearly 30 years ago:
Application Database 🢂 ETL 🢂 Data Warehouse 🢂 Data Mart 🢂 Business Analytics Applications
compared to today’s:
Backend Application Database 🢂 Data Middle Platform 🢂 Business Middle Platform 🢂 Frontend Business-Oriented Services
The core logic behind enterprise data usage remains remarkably consistent. The tools may change, but the principles endure.
Let’s revisit the business intelligence (BI) model from nearly 30 years ago:
Application Database 🢂 ETL 🢂 Data Warehouse 🢂 Data Mart 🢂 Business Analytics Applications
compared to today’s:
Backend Application Database 🢂 Data Middle Platform 🢂 Business Middle Platform 🢂 Frontend Business-Oriented Services
The core logic behind enterprise data usage remains remarkably consistent. The tools may change, but the principles endure.
ETL Operations and Data Integration Are the Cornerstone of Enterprise Data Assets
From the early days of Business Intelligence, through cloud computing and big data analytics, and now into the Data Middle Platform and AI era, Trinity has remained at the forefront of ETL in Taiwan. We’ve helped numerous enterprises establish robust data assets.
Our mission:
No matter how data analysis trends evolve, Trinity will continue to upgrade with advances in data processing technologies, laying a solid foundation for enterprise data assets.
Our mission:
No matter how data analysis trends evolve, Trinity will continue to upgrade with advances in data processing technologies, laying a solid foundation for enterprise data assets.