Usage Guide

You can treat this as your starting point for using the Data Catalog Rest API. Jump to specific sections for details:

What's Available
Making Requests
Interpreting Responses
Sample Usage

What's available?

The REST API lets you:

Report metadata from catalog entries
Create and update Data Catalog objects
Search for catalog entries
Read and archive system metadata such as job execution and audit events

While you can use the REST API to insert metadata into the catalog, you cannot use the API to create catalog entries. The data resources referenced by the Data Catalog are actual data assets that are created by browsing or profiling a data source. There are many properties that are controlled such that they can only be inserted by the Data Catalog system. For example, the REST API does not allow updates or changes to the data resource properties that are provided by the Data Catalog processing engine, such as file sizes, field names, field density or value count.

This table summarizes the objects that the API can access and the actions that can be performed on those objects:

Object Type	Create	Update	View Properties For	Delete
Data sources	X	X	X	-
Data resources	-	-	X	-
Data sets	X	X	X	X
Virtual folders	X	X	X	X
Select system properties on data resources (such as Description)	-	X	X	-
Custom properties on data resources	X	X	X	-
Tag domains	X	X	X	X
Tags	X	X	X	X
Tag associations	X	X	X	X
Lineage relationships	-	-	X	-
User profiles	X	X	X	X
Roles	X	X	X	X
Favorites	X	X	X	X
Reviews	X	X	X	X
Configuration properties	-	X	X	-
Audit events	-	-	X	X

Making Requests

API Endpoint

The Data Catalog API endpoint is available on the same base URL as the Data Catalog application server under the /api/v2/ context. For e.g., your API endpoint would be of the form:

https://wld-host:4039/api/v2/...

Content Type

Data Catalog supports the JSON content type for requests and responses. All requests must be made with the Content-Type: application/json HTTP header

Authentication

First, authenticate as a user with the privileges you'll need to perform the actions you want to access through REST. The REST interface uses the same authentication as the UI: you pass username and password that will validate against the authentication scheme you have configured.

NOTE: At this time, the authentication requires a password; if your service user does not have a password, you cannot perform REST calls with that user.

The authentication call is a POST to /login:

curl -H "Content-Type:application/json"  -X POST -d '{ "username": "USER",  "password": "PASSWORD" }' \
      "http://localhost:8082/api/v2/login" -c cookie.txt

The call produces a session cookie that you reference in other calls.

Interpreting Responses

This API returns standard HTTP status message to indicate the success or failure of a call. Successful calls are always met with the HTTP code "200". Calls that succeed through the HTTP protocol but fail to produce the expected response from Data Catalog are marked with the HTTP code "500" and specific information about the failure is provided in the response payload. Other failure codes may be returned through HTTP before Data Catalog receives the request. For example:

{"message": "Existing lineage already present between the pair of data resources"}

The HTTP response codes you can expect are as follows:

Response Code	Description
200	OK. Request is successful.
401	UNAUTHORIZED. The user is not yet authenticated with the server. Call POST /login to request authentication and try again.
400	BAD REQUEST. Returned when parts of the parameters or request payload are not specified correctly or are missing.
5XX	SERVER ERROR. Returned when the server could not handle the request correctly or if there was an unexpected failure. Checking server logs should will provide more information.

Sample Usage

Note that in each of the calls below:

Replace {payloads} with actual values
Replace 'hostname' with the actual hostname of the Data Catalog application server
Replace 8082 with your actual port (4039 for HTTPs)

Authentication

Creates a login session cookie that is valid for the UI timeout duration

curl -X POST http://hostname:8082/api/v2/login
     -d '{"username":"username","password":"password"}'
     -H 'content-type: application/json'
     -c cookie.txt

Get Version of Data Catalog

curl -X GET http://hostname:8082/api/v2/version
     -H 'content-type: application/json'
     -b cookie.txt

Resource Details

Important - Provide ResourceKey for subsequent calls

curl -X GET http://hostname:8082/api/v2/dataresource/source/{sourceName}?resourcePath={completePath}
     -H 'content-type: application/json'
     -b cookie.txt

Create User

curl -X POST http://hostname:8082/api/v2/user
     -d '{"name":"{userName}","roles":[“{roleName}"],"attributes":{"firstName":"{first name}","lastName":"{last name}","email":"{email address}"}}'
     -H 'content-type: application/json'
     -b cookie.txt

Capture User Attributes

curl -X GET http://hostname:8082/api/v2/user?name={userName}
     -H 'content-type: application/json'
     -b cookie.txt

Delete User

Important - Capture userKey from above command

curl -X DELETE http://hostname:8082/api/v2/user/{userKey}
     -H 'content-type: application/json'
     -b cookie.txt

Create Role

curl -X POST http://hostname:8082/api/v2/user/role
     -d '{"name":"{role-name}","accessLevel":"{access Level},"description":"{description}"}'
     -H 'content-type: application/json'
     -b cookie.txt

Retrieve all available Roles

curl -X GET http://hostname:8082/api/v2/user/role
     -H 'content-type: application/json'
     -b cookie.txt

Retrieve a specific Role

curl -X GET http://hostname:8082/api/v2/user/role/byname?name={roleName}
     -H 'content-type: application/json'
     -b cookie.txt

Modify a specific Role

curl -X PUT http://hostname:8082/api/v2/user/role/{roleKey}"
     -d '{"name":"{role-name}","accessLevel":"{access Level}", "domainKeys":["TagDomainKey"]}'
     -H 'content-type: application/json'
     -b cookie.txt

Delete a specific Role

curl -X DELETE http://hostname:8082/api/v2/user/role/{roleKey}
     -H 'content-type: application/json'
     -b cookie.txt

Create a Tag Domain

curl -X POST http://hostname:8082/api/v2/tagdomain
     -d '{"name": "{Domain Name}","description":"{TagDomain Description}"}'
     -H 'content-type: application/json'
     -b cookie.txt

Delete a Tag Domain

curl -X DELETE http://hostname:8082/api/v2/tagdomain/{domainKey}
     -H 'content-type: application/json'
     -b cookie.txt

Update a Tag Domain

curl -X PUT http://hostname:8082/api/v2/tagdomain/{domainKey}
     -d '{"name": "{Domain Name}","description":"{TagDomain Description}"}'
     -H 'content-type: application/json'
     -b cookie.txt

Create Tag Domains in Bulk

NOTE:

When a URL ends with 'list' path element, it indicates a bulk operation.
Bulk API are not transactional and in worst case scenario, may be partially successful - creating/updating, say, two of the five create/update requested.
When performing bulk operations, Data Catalog API require the entire object to be specified not just the attributes that are being updated.

curl -X POST http://hostname:8082/api/v2/tagdomain/list
     -d '[{ "name": "{Domain Name 1}","description": "{TagDomain description 1}"},
          { "name": “{Domain Name 2}","description": "{TagDomain description 2}"},
          { "name": "{Domain Name 3}","description": "{TagDomain description 3}"}]'
     -H 'content-type: application/json'
     -b cookie.txt

Create a Tag

curl -X POST http://hostname:8082/api/v2/tag
     -d '{"name":"{name of the tag}","description":"{description of the tag}","domainKey":"{domainKey}", "tagState":"VALUE", "enable":true}'
     -H 'content-type: application/json'
     -b cookie.txt

Update a Tag

curl -X PUT http://hostname:8082/api/v2/tag/{TagKey}
     -d '{"name":"{name of the tag}","description":"{description of the tag}","domainKey":"{domainKey}", "enable":true,"learningEnabled":false}'
     -H 'content-type: application/json'
     -b cookie.txt

Update Tags in Bulk

NOTE:

When a URL ends with 'list' path element, it indicates a bulk operation.
Bulk API are not transactional and in worst case scenario, may be partially successful - creating/updating, say, two of the five create/update requested.
When performing bulk operations, Data Catalog API require the entire object to be specified not just the attributes that are being updated.

curl -X PUT http://hostname:8082/api/v2/tag/list
     -d '[ {"key":"{key1}","type":"tag","description":"{tag1Description}","domainKey":"{domainKey1}","parentKey":null,"name":"{tag1}","tagState":"VALUE","minScore":0.0,"regEx":null,"regexMinLen":0,"regexMaxLen":0,"regexFilterClass":null,"learningEnabled":true, "immutable":false,"enable":true,"fullName":"{tag1}"},
           {"key":"{key2}","type":"tag","description":"{tag2Description}","domainKey":"{domainKey2}","parentKey":null,"name":"{tag1}","tagState":"VALUE","minScore":0.0,"regEx":null,"regexMinLen":0,"regexMaxLen":0,"regexFilterClass":null,"learningEnabled":true, "immutable":false,"enable":true,"fullName":"{tag2}"}]'
     -H "content-type: application/json"
     -b cookie.txt

Retrieve a Tag object

Important - Specify Tag Domain and Tag Name

curl -X GET http://hostname:8082/api/v2/tag/byname?tagName={tagname}&domainName={domainName}
     -H 'content-type: application/json'
     -b cookie.txt

Create Multiple Tag objects

curl -POST http://hostname:8082/api/v2/tag/list
     -d '[{"name":"name","description":"{Description}","domainKey":"{domainKey}", "tagState":"VALUE", "enable":true},
          {"name":"name","description":"{Description}","domainKey":"{roleKey}", "tagState":"VALUE", "enable":true},
          {"name":"name","description":"{Description}","domainKey":"{roleKey}", "tagState":"VALUE", "enable":true},
          {"name":"name","description":"{Description}","domainKey":"{roleKey}", "tagState":"VALUE", "enable":true}]
     -H 'content-type: application/json'
     -b cookie.txt

Retrieve all subtags under a given Tag hierarchy

curl -X GET http://hostname:8082/api/v2/tag/children/{tagKey}
     -d '{"name": "{Tag name}","domainKey": "{domain key}"}'
     -H 'content-type: application/json'
     -b cookie.txt

Create a Tag association

curl -X POST http://hostname:8082/api/v2/tagassociation
     -d '{"resourceKey":"{resourceKey}","tagKey":"{TagKey}","field":"{fieldName}","state": "ACCEPTED","semantic":"OTHER","description":"auto "}'
     -H 'content-type: application/json'
     -b cookie.txt

Retrieve all Tag associations for a given data resource

curl -X GET http://hostname:8082/api/v2/tagassociation/byresource?sourceName={sourceName}&resourcePath={completePath}
     -H 'content-type: application/json'
     -b cookie.txt

Retrieve Tag associations for a field

curl -X GET http://hostname:8082/api/v2/tagassociation/byfield?sourceName={sourceName}&resourcePath={completePath}&fieldName={fieldName}
     -H 'content-type: application/json'
     -b cookie.txt

Delete Tag association

Important - To delete a tag associations, the association's state must be "ACCEPTED" or "REJECTED"

Use the PUT tagassociation/{key} to change its state before deleting the tag association

curl -X PUT http://hostname:8082/api/v2/tagassociation/tagassociation/{tagAssociationKey}
     -d '{"resourceKey": "{resourceKey}","tagKey":"{tagKey}","state": "REJECTED","semantic":"OTHER"}'
     -H 'content-type: application/json'
     -b cookie.txt

To delete Tag association

curl -X DELETE http://hostname:8082/api/v2/tagassociation/tagassociation/{tagAssociationKey}
     -H 'content-type: application/json'
     -b cookie.txt

Get DataResources in Bulk

NOTE:

The API provides 3 selectors to control the content it will return.
Initial scope must be specified in terms of either a Virtual Folder or a DataSet. The resources returned will be limited to this scope
Additional filters may be included to further limit the scope of resources returned. This is achieved by specifying one or more property-name and property-value pair through propertyFilter.
The request may also specify which contents are to be returned for each resource, thus fine-tuning the result payload. This can be specified in terms of specific properties, plus there are also convenient group selectors such as viewTags or viewFields.

Filter resources by given ids that belong to given virtualFolder and return only resource_path

curl -i "http://hostname:8082/api/v2/dataresource/metadata?virtualFolderKeys={vfKey}
&propertyFilter=id:{id1};{id2}&views=resource_path&pageSize=20"
-b ./cookie

Filter resources by properties (resource_type=resource_type_hds-file) that belong to given virtualFolder and return tags and fields

curl -i "http://hostname:8082/api/v2/dataresource/metadata?virtualFolderKeys={vfKey}
 &propertyFilter=resource_type:resource_type_hdfs_file&views=viewTags,viewFields&pageSize=20"
-b ./cookie

Filter resources that are modified between given time range.

curl -i "http://hostname.101:8082/api/v2/dataresource/metadata?virtualFolderKeys={vfKey}
&resourceModifiedDateFromL=1542057491055&&resourceModifiedDateToL=1542057491200&views=viewTags&pageSize=20"
-b ./cookie