Working with web services

In Integrator it is possible to extract data from and push data into practicably any REST or SOAP based web service. Integrator also includes a growing number of the built-in connectors to the commonly used APIs.

Connecting to the web services

Using HTTP connector

Create HTTP connection to connect to the web service.

Authentication

Integrator supports the following types of the authentication:

Read how to configure authentication for the web service.

Formats

Web services use different data exchange formats, which must be described in Integrator:

Testing and exploring API endpoints

Use Explorer add-on to test connection to the web service and explore API endpoints.

Step 1. Create connection to the web service.

Step 2. Create format for the response. The most commonly used formats are JSON, XML and CSV.

Step 3. Open Explorer, select connection created in step 1 and link format, created in step 2. Read more how to link format to the connection in Explorer.

Step 4. Explore metadata (endpoints and fields), view data in a grid, query data and discover dependencies using SQL.

API endpoint is a source

Integrator can extract data from the web services, transform it and load into any destination, for example database, files, third-party system via the API, etc.

Read how to configure transformation when the source is API endpoint.

Extracting data from complex nested datasets

Read more how to extract data from the nested source data object using SQL and specifically.

Also, read how to extract data from the nested source data object using built-in transformations. Note: there are multiple options, so please read all the paragraphs under the link.

API endpoint is a destination

Integrator can extract data from files, API endpoints, social web sites, databases, etc., transform it and load into third-party system via API endpoint .

Read how to configure transformation when the destination is API endpoint.

Creating complex nested payload

Read how to create a complex nested payload when the destination is an API endpoint.

If the payload is XML, read how to use XST transformation to create complex XML.

Using template for the request

In Integrator it is possible to use a template to define fields for the request. You can then use visual designer to map source columns to the request (payload) columns.

Read how to use template for the request.

Creating payload using template with tokens

Read how to create a payload using template with tokens.

Testing payload

Read how to test payload without actually calling web service.

Other use cases

Saving response from the HTTP request to the file

In Integrator it is possible to save unmodified response from the HTTP request to the file.

Sending payload to HTTP endpoint

If there is no source to destination transformation, and you just want to send a payload to the HTTP endpoint, use flow type Send payload to HTTP connection.

Read how to send payload to HTTP endpoint.

Exposing any dataset as an API endpoint

Read how to create an endpoint in Integrator, which exposes any dataset as a web service.

Calling API endpoint in a loop

Step 1. Create http connection with a tokenized payload.

Step 2. Create flow which sends payload to HTTP endpoint.

Step 3. Create nested flow and add flow created in step 2.

Step 4. Configure to run flow 2 in a loop.

Step 5. For loop SQL use the same field names in the select statement as token names in the payload.

Example (payload):

{
  "Device": "{name}",   
  "HostName": "{host}",
  "DomainName": "{domain}",
  "FullyQualifiedDomainName": "{domain_full}",
  "Description": "{description}"
}  

Example (loop sql):

select name, host, domain, domain_full, description from staging_table

Loading data from web service into Snowflake

Use flow type "Extract data from Web service, load into Snowflake".

web service to snowflake

Read how to configure snowflake flow.

Integrations ans APIs

Read about Integrator's APIs.

Built-in connectors

Google Analytics

Read how to connect to the Google Analytics in Integrator.

Amazon MWS

Read how to work with Amazon MWS API in Integrator.

Social Networks

Read how to connect to the social networks, such as Twitter and Facebook.

Tips and tricks

Tips and tricks when working with web services

Read about tips and tricks when working with web services.

Common tips and tricks when working with data

Read about common tips and tricks when working with data.

Performance tips

Read about common performance tips.