How does the Flowgear integration platform work?
Flowgear is a cloud-based integration, and a question we often get is ‘How does it work?’ It’s not just something we hear from business people, but also from technical people. So here is a brief overview.
What Flowgear is
Gartner, an IT analyst and advisor firm, has included Flowgear in its integration platform-as-a-service (iPaaS) category, and defines iPaaS as providing capabilities to implement data, application and API integration projects spanning cloud and on-premises systems.
“This is achieved by developing, deploying, executing, managing and monitoring ‘integration flows’ – that is, integration applications bridging between multiple endpoints so that they can work together.”
“An iPaaS is typically used for cloud service integration and application to application integration scenarios. Increasingly, they are also being used for business to business integration, mobile application integration, API publishing and Internet of Things integration scenarios.” Source: Gartner
Simply put, Flowgear provides data plumbing for the enterprise.
How you use Flowgear
You create, deploy and manage integration workflows using the Flowgear Console. The Console is a browser-based graphical development interface in which you drag and drop and integration items, called Nodes, onto the design Canvas, specifying links between Nodes to create a Workflow.
Why it’s better than hand coding integrations
Hand coding integrations involves having to know all the details of the endpoints of the applications you want to integrate. If something changes in the endpoints, your integration breaks. It might be quicker to hand code an integration for a simple point-to-point integration, but most business-oriented integrations are not simple.
Other problems with hand coding are:
- for anything more than basic integrations, our customers have shown that using Flowgear is quicker by a factor of 3-4 than hand coding;
- managing multiple integrations points gets increasingly complex – ‘integration spaghetti’ occurs because customized point-to-point connectivity becomes so complex that it is unmanageable and difficult to change;
- hand coding runs the risk of being write-only, i.e., hard to read or understand by anyone other than the developer;
- hand coding integrations are hard to hand over to non-tech users.
Developers can still apply their skills when using Flowgear – the process of creating a Workflow still fundamentally requires developers to recursively break down tasks into smaller steps, which is what software development really is. Scripting is available but this is a feature that developers can fall back on to if no suitable Node exists.
How Flowgear does integration
Connectors and Nodes
The building blocks of a Flowgear integration Workflow are Nodes and Flow Lines.
Nodes represent specific tasks – such as branching of execution based on an expression, or splitting a data set into individual transactions by parsing JSON or XML. A subset of Nodes are Connectors, which enable connection to specific endpoints; there are Product Connectors (e.g. Salesforce), Technology Connectors (e.g. PDF parser), and Protocol Connectors (e.g. SOAP request).
To see our current set of Connectors, see https://flowgearlegacy.wpenginepowered.com/connectors
Flow Lines specify how Nodes are linked. There are two types of Flow Lines.
- Execution Flow Lines – control the sequence of steps
- Data Flow Lines – control the flow of data between Node
Connecting to on-premise applications
Flowgear is a cloud application, so it’s not uncommon for people to ask how we connect to ground on-premise software. The answer is a piece of technology called a DropPoint. A DropPoint is a small software agent that is installed on an application or database server in order to gain access to an on-premise data source. You can also use a DropPoint to connect old COM apps. You then use a Connection in the Flowgear console to specify the endpoint and credentials for a data source.
A Flowgear DropPoint provides some key capabilities.
- All data flowing to or from the DropPoint is encrypted and compressed automatically.
- The secured outbound connection over HTTPS with TLS1.2 means no changes are necessary to the organisation firewall.
- DropPoints can tolerate loss of connectivity and resume where they left off when a connection is restored, including automatically resending corrupt or lost sections of data.
For more technical information, see http://developers.flowgear.net/kb/DropPoints
Connecting to cloud software
Cloud systems usually only require a username, password and a secondary level of security (e.g., an API key) in order to access the system.
Mapping from one system to another
When you are transferring data from one system to another, it’s likely that the systems use different metadata, i.e., one system might use the term ‘customer’ and the other system will call it ‘custname’ for the same data. Flowgear handles the transformation of source data so it’s appropriate for a target endpoint using the QuickMap Node.
In this screen grab, the visual mappings for the QuickMap Node are being shown. Developers have other mapping options available too, including XSL and scripting.
APIs as well
Application program interfaces (APIs) not only make data access easier and more seamless, but they are also an efficient way to control access. You can use Flowgear to build integrations that share your data internally, or with suppliers and customers, via APIs.
By using a subdomain for your business on Flowgear.io, you can set up the endpoint for all APIs. With Flowgear.io you have a place to do API management, and it enables Flowgear Workflows to be bound as APIs. Flowgear allows you to format API output in JSON.
Flowgear provides security features for APIs so that only authorized users have access. You can also build security checks into Workflows for more granular security.
Debugging, deploying and managing integrations
The Flowgear Console provides the environment for managing integrations.
Debug integrations
As a Workflow starts to execute, an activity log is displayed. You can inspect the tasks that are being executed and also see the value of input and output properties.
Deploy and managing (governance)
Flowgear supports the complex integration scenarios in arise in modern organizations.
- Release Management allows separate revisions of Workflows to be simultaneously used in different Environments (e.g. Test or Production Environments). You can promote or demote Workflows between different Flowgear Environments. In a Test Environment, enhancements to integrations can be built and tested without disrupting Production use. Production Workflows can also be copied and demoted to a Test Environment for fixes and enhancements.
- Revision Management displays the complete history of the changes to a Workflow over time, showing which team members were responsible for a contribution, and rollback or forking of a Workflow.
- Dependency Insights provides information about how Workflows, connections and DropPoints interact with each other, with a visual depiction of the relationship between them.
Security and privacy
Security and privacy is managed at the following levels.
- Site – controls the isolation boundary between tenants on the platform. Each organisation is allocated a Site and no communication across sites is permitted. Some organisations use a single Site to represent integration across the whole company; others prefer to separate out departments or business units by assigning a Site to each one.
- Users accounts – stored as salted, one-way PBKDF2 hashes with at least 10 000 iterations. Passwords for a connection defined in Flowgear are encrypted with AES.
- Workflow Logs – provide a granular record of each execution step of a Workflow by recording the state of a Connector or Workflow step both before and after it executes. Field masking ensures that sensitive information such as passwords and credit card numbers are entirely excluded from logging.
- Network – all external platform communication in Flowgear is encrypted. The Flowgear Console and API’s are served over HTTPS and will use up to TLS 1.2. Flowgear DropPoints communicate over WebSockets and are secured up to TLS 1.2. Whenever possible, Flowgear Connectors make use of secured transports (eg. for FTP both FTPS and SFTP are supported, for HTTP, HTTPS is supported).
- Infrastructure – the Flowgear platform is hosted in secure data centers. Physical access to these data centers is strictly controlled while remote administration accounts and permissions are managed by Microsoft Active Directory.
Build your own Connectors
For organizations needing a highly specialized Connector that is not part of the Flowgear toolbox, you can build your own Connector using a test harness. This allows developers to test and debug Connectors that are under development from within their Development Environment as they are invoked from the Flowgear Engine.
For more information about building Connectors, see https://developers.flowgear.net/kb/Building_Nodes
Not just for developers, for users too
“Citizen integrators” is a term for business users with some technical knowledge. Flowgear’s graphical user interface allows citizen integrators to use existing Workflows for other practical applications without always relying on the development team. The resulting solutions provide high ease of use, without compromising security and control.
Flowgear architecture
Flowgear is a cloud-native, multi-tenant application. Here is a high-level architecture diagram of the Flowgear Engine.
What you get from Flowgear
- Create integration processes immediately with the browser-based designer, no coding required
- Easily map data fields between applications
- Monitor and maintain all your deployed integration processes with notifications for successful and failed processes
- Transfer information between existing on-premise and cloud-based applications, ensuring processes are reliable, and data is accurate
- Eliminate the cost and complexity associated with integration, and decrease integration time
- Enable rapid implementation cycles
- Frequent updates delivered automatically
- No hardware or software to install and maintain
