Waterline Data REST API

Waterline Data provides a REST API to access metadata held in the catalog. The same API allows applications to insert metadata such as property values, tags, tag associations, and lineage relationships. The API provides access to the same operations available from the Waterline Data browser application.

The API uses JSON objects as request and response payloads. The HTTP call returns a general pass/fail status message; calls that fail at the Waterline Data server return an failure response message with an error code and more detailed message.

The API accepts the same user credentials as the Waterline Data browser application. Before sending API calls, an application sends an authentication request. The server responds to a successful request with a session cookie, which the application then uses in the header of API calls. The token is valid for the length of the session.

Resources

name path methods description
Audit Events
  • /v2/auditevent/list
  • /v2/auditevent/search
  • /v2/auditevent/{key}
  • GET
  • POST
  • GET
Endpoints under /auditevent provide access to audit events—Waterline Data system and user activities. You can use audit event characteristics such as user, date, or type of activity to filter the events returned.
Authentication
  • /v2/login
  • /v2/logout
  • POST
  • GET
The Authentication API accepts the same user credentials as the Waterline Data browser. These API calls control the creation and destruction of a session cookie to validate other API calls.
Browse: Field-level
  • /v2/browse/field/facet
  • POST
Endpoints under /browse/field manage the facets available when users view a list of fields for a given resource.
Browse: Resource-level
  • /v2/browse/facet
  • /v2/browse/new
  • /v2/browse/page
  • /v2/browse/pattern
  • POST
  • POST
  • POST
  • POST
Endpoints under /browse allow you to list and manage metadata for data resources based commonalities of location. Compare with the /search functionality, that lists data resources which share some metadata characteristics. Browse functionality depends on how data is stored and its location on a file system or database.
Configuration
  • /v2/config
  • /v2/config/{name}
  • GET
  • POST
Endpoints under /config provide access to the configuration options that control job and web server behavior.
Custom Header
  • /v2/customheader
  • GET
Data Objects
  • /v2/dataobject
  • /v2/dataobject/name
  • /v2/dataobject/recommendations
  • /v2/dataobject/searchresource
  • /v2/dataobject/{key}
  • /v2/dataobject/{key}/profile
  • /v2/dataobject/{key}/tag
  • /v2/dataobject/{key}/join/add
  • /v2/dataobject/{key}/join/refresh
  • /v2/dataobject/{key}/join/remove
  • /v2/dataobject/{key}/resource/add
  • /v2/dataobject/{key}/resource/remove
  • GET POST
  • GET
  • GET
  • GET
  • DELETE GET PUT
  • POST
  • POST
  • POST
  • POST
  • POST
  • POST
  • POST
Data Preview
  • /v2/content/{resourceKey}
  • GET
Endpoints under /datapreview provide access to previews of data resource content based on the authorization of the current user.
Data Resources
  • /v2/dataresource
  • /v2/dataresource/hivecolumns
  • /v2/dataresource/metadata
  • /v2/dataresource/savepatterns
  • /v2/dataresource/{key}
  • /v2/dataresource/browse/{resourceKey}
  • /v2/dataresource/schema/{resourceKey}
  • /v2/dataresource/source/{sourceName}
  • /v2/dataresource/table/{resourceKey}
  • /v2/dataresource/view/{resourceKey}
  • /v2/dataresource/virtualfolder/{virtualFolderName}
  • /v2/dataresource/{resourceKey}/partitions
  • PUT
  • GET
  • GET
  • POST
  • GET PUT
  • GET
  • PUT
  • GET
  • POST
  • POST
  • GET
  • GET
Endpoints under /dataresource provide access to DataResource objects in the Waterline Data repository. Waterline Data defines a data resource as an HDFS file or folder, a set of files that are organized in folders (known as a Waterline Data collection), as a Hive database or table, or as a relational database or table. DataResource objects are uniquely identified by a "key" value, assigned on creation.
Data Sets
  • /v2/dataset
  • /v2/dataset/name
  • /v2/dataset/{key}
  • /v2/dataset/members/add
  • /v2/dataset/members/remove
  • /v2/dataset/schema/{key}
  • /v2/dataset/schema/version/max
  • GET POST
  • GET
  • DELETE GET PUT
  • POST
  • POST
  • PUT
  • GET
Endpoints under /dataset provide access to Virtual Dataset objects. A Waterline Virtual Data Set allow users to create groups of resources having the same schema and spanning different folders in your data lake, into a single virtual unit for easier management. Virtual Data Sets are considered as user defined virtual collections that have the matching schema but may have different path specifications/hierarchy with respect to a data source.
Data Sources
  • /v2/datasource
  • /v2/datasource/list-configs
  • /v2/datasource/validate
  • /v2/datasource/{key}
  • /v2/datasource/dependencies/{key}
  • /v2/datasource/gethint/{datasourcetype}
  • /v2/datasource/name/{name}
  • /v2/datasource/sync/{key}
  • /v2/datasource/credentials/users/{user}
  • /v2/datasource/{key}/credentials/users/{user}
  • GET POST
  • GET
  • POST
  • DELETE GET PUT
  • GET
  • GET
  • GET
  • DELETE
  • GET
  • DELETE POST PUT
Endpoints under /datasource provide access to DataSource objects. A Waterline Data data source is the location of an HDFS directory, a Hive instance or specific database, a cloud storage location, or a relational database instance or specific database. A data source hosts data resources, in the form of folders, files, databases, and tables.
Entity Specifications
  • /v2/entityspec/all
  • /v2/entityspec/{name}
  • /v2/entityspec/{name}/propertyspec
  • GET
  • GET
  • POST PUT
An entity specification defines properties for Waterline Data objects such as tags, users, roles, data resources, and so on. Properties of entities are represented by PropertySpec objects. An EntitySpec object associates a set of properties with a specific entity type. Endpoints under /entityspec allow you to create, query, update, and delete properties associated with entities, including setting the access level for properties.
External Sources
  • /v2/externalsource
  • /v2/externalsource/{key}
  • /v2/externalsource/name/{name}
  • GET POST
  • DELETE GET
  • DELETE GET
External Sources are sources of metadata to be imported into the Waterline Data catalog. Endpoints under /externalsource provide a way to identify and connect to these outside applications. For example, the extensions created to import metadata from applications such as Apache Atlas or Cloudera Navigator use external sources to persist connection information for the application instance.
Favorites
  • /v2/favorites
  • /v2/favorites/{entityKey}
  • /v2/favorites/{user}
  • /v2/favorites/{entityKey}/{userName}
  • GET
  • DELETE POST
  • GET
  • POST
Endpoints under /favorites provide a way to manage user bookmarks on tags and resources.
JobManagement
  • /v2/job/execution
  • /v2/job/execution/terminate/{jobInstanceId}
  • /v2/job/execution/{key}/resources
  • POST
  • PUT
  • GET
JobManagement
  • /v2/job/instances
  • /v2/job/instances/currentuser
  • /v2/job/instances/{key}
  • /v2/job/instances/{key}/executions
  • GET
  • GET
  • GET
  • GET
JobManagement
  • /v2/job/sequences
  • /v2/job/sequences/{key}
  • GET
  • GET
JobManagement
  • /v2/job/templates
  • /v2/job/templates/bulkdelete
  • /v2/job/templates/{key}
  • /v2/job/templates/{key}/instances
  • GET POST
  • POST
  • DELETE GET PUT
  • GET
Lineage
  • /v2/lineage/searchresource
  • /v2/lineage/{key}
  • /v2/lineage/addparent/{key}
  • GET
  • GET
  • POST
Waterline Data defines a LineageRelations object that records and presents the lineage relationships between data resources. Endpoints under /lineage provide access to these objects.
Overlap tables
  • /v2/overlap/table
  • /v2/overlap/table/csv
  • GET POST
  • POST
Endpoints under /overlap/table allow you to pass tables information and receive a preliminary overlap report computed based on the table schema
Reviews
  • /v2/reviews
  • /v2/reviews/{key}
  • /v2/reviews/entity/{entityKey}
  • /v2/reviews/user/{user}
  • /v2/reviews/users/{user}
  • POST PUT
  • DELETE
  • GET
  • GET
  • POST
Endpoints under /reviews allow access to user reviews including deleting specific reviews.
Roles
  • /v2/role
  • /v2/role/byname
  • /v2/role/{key}
  • GET POST
  • GET
  • DELETE GET PUT
Waterline Data uses roles to indicate what data resources from the catalog are available to each user. User roles determine what actions each user can perform for metadata management inside Waterline Data. Endpoints under /role allow creation of new roles and modification of existing roles. To associate roles to users, use a PUT request to /user/{key}.
Rules
  • /v2/rule
  • /v2/rule/name
  • /v2/rule/{key}
  • /v2/rule/name/{name}
  • GET POST PUT
  • PUT
  • DELETE GET
  • DELETE GET
Endpoints under /rule allow you to list and manage the rules.
Search
  • /v2/search/facets
  • /v2/search/new
  • /v2/search/page
  • /v2/search/runFacet
  • GET
  • POST
  • POST
  • POST
Endpoints under /search allow you to collect data resources that share some metadata characteristics. Compare the browse functionality, that returns a set of data resources based on the resource's location. Searchable characteristics include facets and keywords.
SessionController
  • /session
  • GET
Tag Associations
  • /v2/tagassociation
  • /v2/tagassociation/byfield
  • /v2/tagassociation/byresource
  • /v2/tagassociation/list
  • /v2/tagassociation/{key}
  • /v2/tagassociation/list/byresource
  • /v2/tagassociation/resource/{key}
  • /v2/tagassociation/tag/{key}
  • /v2/tagassociation/counts/domainlevel/all
  • /v2/tagassociation/tag/container/{key}
  • /v2/tagassociation/counts/taglevel/domain/{key}
  • /v2/tagassociation/counts/taglevel/tag/{key}
  • POST
  • GET
  • GET
  • POST PUT
  • DELETE GET PUT
  • POST
  • GET
  • GET
  • GET
  • GET
  • GET
  • GET
Endpoints under /tagassociation provide access to TagAssociation objects in the Waterline Data repository. A tag association object defines the relationship of a tag to a specific data resource (folders, files, collections, and tables) or specific field in a data resource.
Tag Domains
  • /v2/tagdomain
  • /v2/tagdomain/all
  • /v2/tagdomain/alldomains
  • /v2/tagdomain/list
  • /v2/tagdomain/{key}
  • /v2/tagdomain/name/{name}
  • /v2/tagdomain/{domainKey}/alltags
  • POST
  • GET
  • GET
  • POST PUT
  • DELETE GET PUT
  • DELETE GET
  • GET
Waterline Data tags are maintained in a tag glossary that is organized into tag domains. Endpoints under /tagdomain provide access to TagDomain objects.
Tags
  • /v2/tag
  • /v2/tag/byname
  • /v2/tag/hierarchy
  • /v2/tag/list
  • /v2/tag/{key}
  • /v2/tag/children/{key}
  • /v2/tag/regex/validate
  • POST
  • GET
  • POST
  • POST PUT
  • DELETE GET PUT
  • GET
  • POST
Endpoints under /tag provide access to Tag objects in the Waterline Data repository. Tags are labels associated with data resources (folders, files, collections, or tables), or with specific fields in data resources.
User Profiles
  • /v2/user
  • /v2/user/_me
  • /v2/user/{key}
  • /v2/user/assign/role
  • /v2/user/name/{username}
  • /v2/user/permit/resource
  • /v2/user/permit/source
  • /v2/user/remove/role
  • GET POST
  • GET PUT
  • DELETE GET PUT
  • POST
  • GET
  • GET
  • GET
  • POST
Endpoints under /user allow administrators to manage user profiles, represented by the User object. The caller must have a roles that includes an Administrator access level to create, update, or delete user profiles.
Virtual Folders
  • /v2/virtualfolder
  • /v2/virtualfolder/databases
  • /v2/virtualfolder/{key}
  • /v2/virtualfolder/datasource/{datasourceKey}
  • /v2/virtualfolder/dependencies/{key}
  • /v2/virtualfolder/list/hive
  • /v2/virtualfolder/name/{name}
  • GET POST
  • GET
  • DELETE GET PUT
  • GET
  • GET
  • GET
  • GET
Endpoints under /virtualfolder provide access to VirtualFolder objects. Virtual ​folders ​allow ​users ​to ​create ​groups ​of ​resources ​belonging ​to ​a ​data ​source ​into ​smaller ​units ​for easier ​management. ​In ​addition, ​Virtual ​folders ​allow ​data resources ​to ​be ​part ​of ​multiple ​folders ​thus letting ​customers ​create ​folders ​ with ​overlapping ​sets ​of ​data ​resources.
Waterline Data Logs
  • /v2/logs
  • /v2/logs/download
  • /v2/logs/view
  • GET
  • GET
  • GET

Data Types

JSON

type description
AbstractField The complete metadata for each field (column) in a data file or table.
AccessLevel Access levels are a component of user roles that describe sets of Waterline Data behaviors available on the data sources and tag domains included in a role. The access levels are hierarchical: all behaviors available to the Guest level are available to the Analyst; all Analyst behaviors are available to the Steward level; all Steward behaviors are available to the Admin level.
AssociationCount Type that describes tag association counts by state for a given tag.
AtomicField The complete metadata for each field (column) in a data file or table.
AuditEvent Type that describes the Waterline Data activities that users (including the system user) perform. They include the action, the user who performed the action, and the Waterline Data objects that are affected by the action.
AuditEventListResponse Type that wraps a list of AuditEvent objects. It includes the pagination start index and page size for returning a fixed number of audit event descriptions.
AuditFilterCriteria
BrowseRequest The type that describes the data resource payload returned when viewing a list of resources based on their context in a file system or database. It includes the parent data resource (typically a folder or database) and pagination start and size. It can also include selections to filter the child resources based on facets.
BrowseResult Type that defines the fields to display a Waterline Data catalog entry ("data resource"), whether it is a container such as a folder or an individual file or table. Note that Waterline Data collections are described both as a container and as a single data set.
Category Enumeration that describes groupings for configuration properties. These categories are for organization and are not functional.
ConfigProperty Type that describes the configuration settings used to control Waterline Data behaviors. Properties include controls such as whether to profile files that failed previously. The information provided for each property includes the property name, the label used in the UI, a description of what the property controls, and attributes for who and how the property can be viewed or changed.
Credential
Credentials Type that describes the username and password for the current user's authenticated session. Credentials details are passed to the system, but cannot be returned.
CustomHeaderProperties
CustomProperty Type describing a user-defined property defined for a Waterline Data entity.
DataObjectRequest Base type for Waterline Data metadata objects.
DataPartition Type representing DataPartition
DataPreview
DataResource Type includes all the properties that describe a database, folder, table, schema, file, or Waterline Data collection. You can add properties to a data resource using a PUT call to /entityspec/\{key\/propertygroup}.
DataResourceFlexTO DataResourceFlexTO a Transfer object used to return data for resource/metadata api
DataResourceLite Type includes all the properties that describes a lite resource Used in children array of browse and search api response.
DataSource Type including the fields that describe a data source, including source type and connection information.
DataSourceDependencies
DataType Enumeration of data types for Waterline Data property values. These types are Java primitive types plus string, decimal, date, timestamp, array, object, and map.
DatasetMember
DatasetRequest Base type for Waterline Data metadata objects.
Entity Base type for Waterline Data metadata objects.
EntitySpecification Type that describes a Waterline Data object type.
ExternalSource Type that describes applications with which Waterline Data can exchange metadata. For example, connections to Cloudera Navigator or Apache Atlas are defined as ExternalSource objects.
FacetResult Type that describes a search facet, including facet category counts for the number of data resources or fields included in search results.
FacetSelection Type to indicate a filter on a search result.
Favorite Type that describes favorites, including the marked item and the time it was marked.
GroupIdentity
IterableOfExternalSource
IterableOfTableComparisionTO
IteratorOfString
JobDetailsTO
JobExecutionResultTO
JobExecutionTO
JobSequenceTO
JoinConditionTO
LineageRelationPath
LineageRelations Type that describes the connection between two data resources in a lineage graph, including the operation that identifies the connection.
LineageSearchPathTO
LogFileContent
LogFileElement
LogFolderElement
LoginResponse Type that provides a response to a login attempt. Successful attempts return a SUCCESS flag and message and echo the username. Failed attempts return a FAILURE flag and a message that describes the reason for failing.
LoginResult Enumeration of the possible outcomes of a login attempt, including "SUCCESS" and "FAILURE".
LogoutResponse
Member Type that describes payload for dataset add/remove member apis
Node Type that describes the operation and most recent operation execution that connects two resources in a lineage graph.
ObjectField Type that describes field-level metadata for data resources.
OpDetails Type that describes the operation executions that correspond to an operation.
OperationExecution
PaginatedResponse2OfDataResourceFlexTO Response Payload used in dataresource/metadta api
PaginatedResponseOfDataResource
PaginatedResponseOfJobDetailsTO
PaginatedResponseOfVirtualFolder
PagingCriteria Type describing the start and size of data resource or field information in search or browse results.
PathElement Type that maps a resource identifier (key) to the unique name of the resource, including its filesystem path or parent database.
PathInfo Type that describes the location of a resource based on the data source (indicated by its name and unique identifying key) and the fully qualified path of the resource.
PathSpecification Type describing PathSpecification of a virtual-folder
PatternTO
PatternValue
Principal
Properties
Property Type that describes metadata associated with an entity. You can create additional properties to enhance the metadata stored for data resources.
PropertySpecification Type describing the attributes of properties on Waterline Data entities.
PropertyType Enumeration of property value data types.
RegexTester Type that includes the regular expression used to identify field data to associate with a tag. This type includes a sample of field data to validate the regular expression against.
ReportedField Type representing ReportedField of data resource
ResourceField Type that describes a field in a data resource, including the field name, its data type, and the unique identifying key of the data resource the field is associated with.
Role Type describing the Waterline Data access model for users. Roles include an access level, one or more data sources, and one or more tag domains.
Rule Base type for Waterline Data metadata objects.
RuleAction
RuleScope
SearchCriteria Type that describes the criteria for searching on data resources or fields. Criteria can include a keyword and/or facets.
SearchResult Type that describes the data resources or fields returned in a search. The search criteria are also included.
SearchType
Semantic Enumeration of values that describe how the tag association contributes to tag association discovery.
Semantic Enumeration of values that describe how the domain was created. Should not be set from the REST API.
SessionData
Source Type that describes a parent in a lineage relationship, including the fully-qualified data resource name (path) and whether or not the data resource is a Waterline Data collection.
State Enumeration of values that describe the state of the tag association.
State
StatusDetail Type that describes the execution status of a job.
TableComparisionTO
Tag Type that describes a tag, including the tag name and properties that describe how the tag is used in tag association discovery.
TagAssociation Type that describes a tag association, including a data resource or resource and field, a tag, the state of the association, and whether or not the field data from this association is used as a reference for tag discovery (Semantic).
TagAssociationContainer Type that describes a list of tag associations.
TagAssociationRequest
TagAssociationView List of tag associations for this data resource. The content includes values designed to be human readable: it includes the names of the relevant items in addition to their unique keys.
TagDetail
TagDetails
TagDomain Type that describes a tag domain, including the name and description for the domain.
TagDomainContainer Type that describes a tag domain and all of the included tags.
TagRequest Type that describes the minimum properties needed to create a tag, including the tag domain (by key), the tag name, and the tag description.
TagState Enumeration of the types of tag propagation available in Waterline Data.
UpdatedUserReview Type that describes the new statistics associated with a rating on a data resource a user review for that resource has been updated. It includes the components of the updated review as well.
User Type that describes a user profile, including the user name and the roles associated with this user.
UserIdentity
UserInfo Type that provides information about the current user.
UserReview Type that includes the components of a review, including a title, comment, and rating. The UserReview also includes the username of the person who created the review.
VirtualFolder Base type for Waterline Data metadata objects.
VirtualFolderDependencies