Lumada Data Catalog REST API Reference: Data Sources

Endpoints under /datasource provide access to DataSource objects. A Waterline Data data source is the location of an HDFS directory, a Hive instance or specific database, a cloud storage location, or a relational database instance or specific database. A data source hosts data resources, in the form of folders, files, databases, and tables.
DataSource objects are uniquely identified by a "key" value, assigned on creation. In addition, data sources can be referenced by the "name" property, specified by the creator.
Data source calls return results based on the user credentials for the current authentication session.

GET /v2/datasource

A GET request to the /datasource endpoint retrieves a list of all data sources configured in the catalog. A DataSource object includes:

The data source "name", a string set by the creator to identify the data source.
The data source "uri", a string indicating the connection point of the data source. This URI may contain parameters to support authentication or may point to a symbolic host to support high-availability access. For example, "hdfs://acme01:8020" or "jdbc:hive2://acme04:10000"
The data source "type" (such as HDFS or Hive)
Connection credentials including a username and password used to connect to the data source instance.
A description of the data source visible to users in the Waterline Data UI.
A specific folder or database within the instance that this data source represents.

Response Body
media type	data type	description
application/json	array of DataSource (JSON)	A list of DataSource objects in JSON format.

Example

Request

GET /v2/datasource
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
[ {
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
} ]

POST /v2/datasource

A POST request to the /datasource endpoint creates a DataSource object.

Request Body
media type	data type	description
application/json	DataSource (JSON)	The data source content ("transfer object") in JSON format, including the "name" string, URI, and connection credentials.

Response Body
media type	data type	description
application/json	DataSource (JSON)	The new DataSource object, with its assigned unique key.

Example

Request

POST /v2/datasource
Content-Type: application/json
Accept: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

Response

HTTP/1.1 201 Created
Content-Type: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

GET /v2/datasource/list-configs

A GET request to the /datasource/list-configs endpoint retrieves a list of all data sources configured in the catalog. A DataSource object includes:

The data source "name", a string set by the creator to identify the data source.
The data source "uri", a string indicating the connection point of the data source. This URI may contain parameters to support authentication or may point to a symbolic host to support high-availability access. For example, "hdfs://acme01:8020" or "jdbc:hive2://acme04:10000"
The data source "type" (such as HDFS or Hive)
Connection credentials including a username and password used to connect to the data source instance.
A description of the data source visible to users in the Waterline Data UI.

Response Body
media type	data type	description
application/json	array of DataSource (JSON)	A list of DataSource objects in JSON format.

Example

Request

GET /v2/datasource/list-configs
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
[ {
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
} ]

POST /v2/datasource/validate

A POST request to the /datasource/validate endpoint tests the connection for a DataSource object.

Request Body
media type	data type	description
application/json	DataSource (JSON)	The data source content ("transfer object") in JSON format, including the "name" string, URI, and connection credentials.

Response Body
media type	data type	description
application/json	map of string (JSON)

Example

Request

POST /v2/datasource/validate
Content-Type: application/json
Accept: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

Response

HTTP/1.1 201 Created
Content-Type: application/json

                
...

DELETE /v2/datasource/{key}

Request Parameters
name	type	description
key	path

Example

Request

DELETE /v2/datasource/{key}
Content-Type: */*

                
...

Response

HTTP/1.1 204 No Content

GET /v2/datasource/{key}

A GET request to the /datasource/{key} endpoint retrieves details of a specific data source.

Request Parameters
name	type	description
key	path	The identifying key of a data source, as returned in the list from a call to listDataSources().

Response Body
media type	data type	description
application/json	DataSource (JSON)	The DataSource object in JSON format.

Example

Request

GET /v2/datasource/{key}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

PUT /v2/datasource/{key}

A PUT request to the /datasource/{key} endpoint updates properties of a DataSource object. Warning: Updating a data source after contents of the data source have been added to the Waterline Data catalog may cause unexpected results.

Request Parameters
name	type	description
key	path	The unique identifier of the DataSource object.

Request Body
media type	data type	description
application/json	DataSource (JSON)	The new data source content in JSON format. Properties that are not present are not changed.

Response Body
media type	data type	description
application/json	DataSource (JSON)	The updated DataResource object in JSON format.

Example

Request

PUT /v2/datasource/{key}
Content-Type: application/json
Accept: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

Response

HTTP/1.1 204 No Content
Content-Type: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

GET /v2/datasource/dependencies/{key}

Request Parameters
name	type	description
key	path

Response Body
media type	data type	description
application/json	DataSourceDependencies (JSON)

Example

Request

GET /v2/datasource/dependencies/{key}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "Association" : 12345,
  "Review" : 12345,
  "Virtual Folder" : 12345,
  "Dataset" : 12345,
  "Favorite" : 12345,
  "Resource" : 12345,
  "Role" : 12345
}

GET /v2/datasource/gethint/{datasourcetype}

Request Parameters
name	type	description
datasourcetype	path

Response Body
media type	data type	description
application/json	DataSource (JSON)

Example

Request

GET /v2/datasource/gethint/{datasourcetype}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

GET /v2/datasource/name/{name}

A GET request to the /datasource/name endpoint retrieves details of a specific data source.

Request Parameters
name	type	description
name	path	A data source name string.

Response Body
media type	data type	description
application/json	DataSource (JSON)	The DataSource object in JSON format.

Example

Request

GET /v2/datasource/name/{name}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "hiveUrl" : "...",
  "agentLocalUrl" : "...",
  "jdbcUrl" : "...",
  "sourceType" : "...",
  "useWebHdfs" : true,
  "agentName" : "...",
  "configured" : true,
  "webHdfsUri" : "...",
  "rootResourceKey" : "...",
  "permittedToRead" : true,
  "skipConnectionTest" : true,
  "hdfsUri" : "...",
  "driverClass" : "...",
  "name" : "...",
  "agentKey" : "...",
  "attributes" : {
    "property1" : { },
    "property2" : { }
  },
  "sourcePath" : "...",
  "jdbcPassword" : "...",
  "jdbcUsername" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "key" : "...",
  "description" : "...",
  "type" : "...",
  "restricted" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "richDescription" : "...",
  "score" : 12345.0,
  "customProperties" : [ {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  }, {
    "description" : "...",
    "value" : { },
    "facetAble" : true,
    "name" : "...",
    "type" : "...",
    "displayName" : "..."
  } ]
}

DELETE /v2/datasource/sync/{key}

Request Parameters
name	type	description
key	path

Response Body
media type	data type	description
application/json	object (JSON)

Example

Request

DELETE /v2/datasource/sync/{key}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 204 No Content
Content-Type: application/json

                
...

GET /v2/datasource/credentials/users/{user}

Request Parameters
name	type	description
user	path

Response Body
media type	data type	description
application/json	map of Credentials (JSON)

Example

Request

GET /v2/datasource/credentials/users/{user}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 200 OK
Content-Type: application/json

                
{
  "..." : {
    "username" : "...",
    "password" : "..."
  }
}

DELETE /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name	type	description
key	path
user	path

Response Body
media type	data type	description
application/json	Credential (JSON)

Example

Request

DELETE /v2/datasource/{key}/credentials/users/{user}
Content-Type: */*
Accept: application/json

                
...

Response

HTTP/1.1 204 No Content
Content-Type: application/json

                
{
  "principal" : {
    "name" : "..."
  },
  "secret" : { }
}

POST /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name	type	description
key	path
user	path

Request Body
media type	data type
application/json	Credentials (JSON)

Example

Request

POST /v2/datasource/{key}/credentials/users/{user}
Content-Type: application/json

                
{
  "username" : "...",
  "password" : "..."
}

Response

HTTP/1.1 201 Created

PUT /v2/datasource/{key}/credentials/users/{user}

Request Parameters
name	type	description
key	path
user	path

Request Body
media type	data type
application/json	Credentials (JSON)

Example

Request

PUT /v2/datasource/{key}/credentials/users/{user}
Content-Type: application/json

                
{
  "username" : "...",
  "password" : "..."
}

Response

HTTP/1.1 204 No Content

Data Sources Resource

GET /v2/datasource

Example

Request

Response

POST /v2/datasource

Example

Request

Response

GET /v2/datasource/list-configs

Example

Request

Response

POST /v2/datasource/validate

Example

Request

Response

DELETE /v2/datasource/{key}

Example

Request

Response

GET /v2/datasource/{key}

Example

Request

Response

PUT /v2/datasource/{key}

Example

Request

Response

GET /v2/datasource/dependencies/{key}

Example

Request

Response

GET /v2/datasource/gethint/{datasourcetype}

Example

Request

Response

GET /v2/datasource/name/{name}

Example

Request

Response

DELETE /v2/datasource/sync/{key}

Example

Request

Response

GET /v2/datasource/credentials/users/{user}

Example

Request

Response

DELETE /v2/datasource/{key}/credentials/users/{user}

Example

Request

Response

POST /v2/datasource/{key}/credentials/users/{user}

Example

Request

Response

PUT /v2/datasource/{key}/credentials/users/{user}

Example

Request

Response