Data integration is becoming a hot topic. A 2016 survey found that the integration of multiple systems had an impact on business strategy. In the past, the job of integrating systems was a complex, often costly, frequently cumbersome process, involving on-premise integration tools. These days a new breed of applications, iPaaS (integration Platform-as-a-service), are taking over as the integration platform of choice for new integration projects. So how should developers and IT management approach data integration projects to avoid getting to that point where they ask “This is getting too complex, what do we do?”
Issues to consider in a data integration project
Before beginning a data integration project, you need to understand the environment and implications of the data integration project.
Firstly, there is the ‘why, what, how, who.’
| Why | Trigger or business pain that initiates the integration project |
| What | Technology options and solutions |
| How | Business processes that will need to be changed or added |
| Who | People involved |
After getting past the ‘why’ of an integration project, you can get some results with the ‘what’, but to get real value you need to involve all the other aspects. A Gartner article on the stages of integration maturity shows how organizations can assess and improve their integration competency.
Then there are different types of data integration. Each type has its own issues.
| Type | Justification | Complexity | Value |
|---|---|---|---|
| Transactional | Operational productivity, efficiency | Medium | Time savings, reduction of errors |
| Relational | Streamline business partner interactions | Complex | Faster order processing, improved fulfillment |
| Analytical | Organizational improvement, excellence | Medium- Complex | Data-driven decisions, empowered management |
Data integration architecture using iPaaS
Selecting a data integration solution
Today many data integration projects start with what initially seems a good approach – manually building custom software to handle the integrations. But this nearly always ends up being complicated and time consuming to maintain, and slow to respond to changes.
A dedicated integration solution works much better for increasingly complex business integration requirements. Here are the key criteria should you consider when selecting an integration solution.
- Support for real-time integration – batch data integration is no longer acceptable for the way businesses work
- An architecture based on reusable components – so that you don’t have to re-write integrations every time
- An API management framework that will enable information requests from different data sources and provide translation, aggregation and transfer of data
- Ability to handle both data file and document (e.g. PDF) integration management
- Support for common data standards, and conversion between standards – to enable communication between different systems, and for integrating with business partners
- Efficient and secure data transmission – encryption and compression of data through the integration workflow
- Orchestration and automation – reduce the overhead of managing various integrations
- Data governance – ensure there is secure, controlled access to data and integration workflows
The value of an iPaaS for data integration
An iPaaS allows organizations to move and manage data across different applications and data stores. As an integration platform, it:
- provides access points to data for both human users and other systems,
- enables real-time integration,
- allows integration of multiple ERP systems during and after a company merger,
- reduces the friction of cloud adoption,
- facilitates the on-boarding of partners.
How iPaaS can replace ETL
One major data integration project for many enterprises is populating an enterprise data warehouse (EDW) from transactional systems. This often involves using on-premise extract-transform-load (ETL) software running batch updates. What would be better is a ‘ready-to-go’ data library that can be accessed at any time, in real-time, without the problem of having to prepare and load data beforehand. A modern iPaaS has an easy-to-use interface, which means business users who want data for analysis or reporting can be given access to the iPaaS to extract data as and when they need it.
The value of data integration
As data integration demands grow, IT departments will have to grapple with the challenge of balancing data requests against current budgets, available resources, and other system priorities. By selecting a cloud-based integration platform, like Flowgear, IT departments can implement and support data access capabilities that are easier to manage and use than traditional ETL tools.
Businesses have adopted Flowgear to link into systems to:
- get a better consolidated view
- integrate with customers’ systems in real-time
- access data from systems after a merger
- synchronize data between systems
Data integration has always been a complex area, but new cloud-based integration platforms don’t have the limitations of past legacy applications. Just remember, successful data integration project don’t have programmers writing lots of custom code, and also focus on more than just the technology alone.
