Data Sources Resource

Endpoints under /datasource provide access to DataSource objects. A Waterline Data data source is the location of an HDFS directory, a Hive instance or specific database, a cloud storage location, or a relational database instance or specific database. A data source hosts data resources, in the form of folders, files, databases, and tables.
DataSource objects are uniquely identified by a "key" value, assigned on creation. In addition, data sources can be referenced by the "name" property, specified by the creator.
Data source calls return results based on the user credentials for the current authentication session.

GET /v2/datasource

A GET request to the /datasource endpoint retrieves a list of all data sources configured in the catalog. A DataSource object includes:

  • The data source "name", a string set by the creator to identify the data source.
  • The data source "uri", a string indicating the connection point of the data source. This URI may contain parameters to support authentication or may point to a symbolic host to support high-availability access. For example, "hdfs://acme01:8020" or "jdbc:hive2://acme04:10000"
  • The data source "type" (such as HDFS or Hive)
  • Connection credentials including a username and password used to connect to the data source instance.
  • A description of the data source visible to users in the Waterline Data UI.
  • A specific folder or database within the instance that this data source represents.

Response Body
media type data type description
application/json array of DataSource (JSON) A list of DataSource objects in JSON format.

Example

Request
GET /v2/datasource
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
[ {
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
} ]
                
              

POST /v2/datasource

A POST request to the /datasource endpoint creates a DataSource object.

Request Body
media type data type description
application/json DataSource (JSON) The data source content ("transfer object") in JSON format, including the "name" string, URI, and connection credentials.
Response Body
media type data type description
application/json DataSource (JSON) The new DataSource object, with its assigned unique key.

Example

Request
POST /v2/datasource
Content-Type: application/json
Accept: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              
Response
HTTP/1.1 201 Created
Content-Type: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              

GET /v2/datasource/list-configs

A GET request to the /datasource/list-configs endpoint retrieves a list of all data sources configured in the catalog. A DataSource object includes:

  • The data source "name", a string set by the creator to identify the data source.
  • The data source "uri", a string indicating the connection point of the data source. This URI may contain parameters to support authentication or may point to a symbolic host to support high-availability access. For example, "hdfs://acme01:8020" or "jdbc:hive2://acme04:10000"
  • The data source "type" (such as HDFS or Hive)
  • Connection credentials including a username and password used to connect to the data source instance.
  • A description of the data source visible to users in the Waterline Data UI.

Response Body
media type data type description
application/json array of DataSource (JSON) A list of DataSource objects in JSON format.

Example

Request
GET /v2/datasource/list-configs
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
[ {
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
} ]
                
              

POST /v2/datasource/validate

A POST request to the /datasource/validate endpoint tests the connection for a DataSource object.

Request Body
media type data type description
application/json DataSource (JSON) The data source content ("transfer object") in JSON format, including the "name" string, URI, and connection credentials.
Response Body
media type data type description
application/json map of string (JSON)

Example

Request
POST /v2/datasource/validate
Content-Type: application/json
Accept: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              
Response
HTTP/1.1 201 Created
Content-Type: application/json

                
...
                
              

DELETE /v2/datasource/{key}

Request Parameters
name type description
key path

Example

Request
DELETE /v2/datasource/{key}
Content-Type: */*

                
...
                
              
Response
HTTP/1.1 204 No Content

              

GET /v2/datasource/{key}

A GET request to the /datasource/{key} endpoint retrieves details of a specific data source.

Request Parameters
name type description
key path The identifying key of a data source, as returned in the list from a call to listDataSources().
Response Body
media type data type description
application/json DataSource (JSON) The DataSource object in JSON format.

Example

Request
GET /v2/datasource/{key}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              

PUT /v2/datasource/{key}

A PUT request to the /datasource/{key} endpoint updates properties of a DataSource object. Warning: Updating a data source after contents of the data source have been added to the Waterline Data catalog may cause unexpected results.

Request Parameters
name type description
key path The unique identifier of the DataSource object.
Request Body
media type data type description
application/json DataSource (JSON) The new data source content in JSON format. Properties that are not present are not changed.
Response Body
media type data type description
application/json DataSource (JSON) The updated DataResource object in JSON format.

Example

Request
PUT /v2/datasource/{key}
Content-Type: application/json
Accept: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              
Response
HTTP/1.1 204 No Content
Content-Type: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              

GET /v2/datasource/dependencies/{key}

Request Parameters
name type description
key path
Response Body
media type data type description
application/json DataSourceDependencies (JSON)

Example

Request
GET /v2/datasource/dependencies/{key}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "reviewCount" : 12345,
  "resourceCount" : 12345,
  "associationCount" : 12345,
  "roleCount" : 12345,
  "favoriteCount" : 12345,
  "virtualFolderCount" : 12345,
  "dataSetCount" : 12345
}
                
              

GET /v2/datasource/gethint/{datasourcetype}

Request Parameters
name type description
datasourcetype path
Response Body
media type data type description
application/json DataSource (JSON)

Example

Request
GET /v2/datasource/gethint/{datasourcetype}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              

GET /v2/datasource/name/{name}

A GET request to the /datasource/name endpoint retrieves details of a specific data source.

Request Parameters
name type description
name path A data source name string.
Response Body
media type data type description
application/json DataSource (JSON) The DataSource object in JSON format.

Example

Request
GET /v2/datasource/name/{name}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "name" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "hdfsUri" : "...",
  "webHdfsUri" : "...",
  "hiveUrl" : "...",
  "jdbcUrl" : "...",
  "jdbcUsername" : "...",
  "jdbcPassword" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "rootResourceKey" : "...",
  "driverClass" : "...",
  "configured" : true,
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}
                
              

DELETE /v2/datasource/sync/{key}

Request Parameters
name type description
key path
Response Body
media type data type description
application/json object (JSON)

Example

Request
DELETE /v2/datasource/sync/{key}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 204 No Content
Content-Type: application/json

                
...
                
              

GET /v2/datasource/credentials/users/{user}

Request Parameters
name type description
user path
Response Body
media type data type description
application/json map of Credentials (JSON)

Example

Request
GET /v2/datasource/credentials/users/{user}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "..." : {
    "username" : "...",
    "password" : "..."
  }
}
                
              

DELETE /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name type description
key path
user path
Response Body
media type data type description
application/json Credential (JSON)

Example

Request
DELETE /v2/datasource/{key}/credentials/users/{user}
Content-Type: */*
Accept: application/json

                
...
                
              
Response
HTTP/1.1 204 No Content
Content-Type: application/json

                
{
  "principal" : {
    "name" : "..."
  },
  "secret" : { }
}
                
              

POST /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name type description
key path
user path
Request Body
media type data type
application/json Credentials (JSON)

Example

Request
POST /v2/datasource/{key}/credentials/users/{user}
Content-Type: application/json

                
{
  "username" : "...",
  "password" : "..."
}
                
              
Response
HTTP/1.1 201 Created

              

PUT /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name type description
key path
user path
Request Body
media type data type
application/json Credentials (JSON)

Example

Request
PUT /v2/datasource/{key}/credentials/users/{user}
Content-Type: application/json

                
{
  "username" : "...",
  "password" : "..."
}
                
              
Response
HTTP/1.1 204 No Content