Waterline Data defines a LineageRelations object that records and presents the lineage relationships between data
resources. Endpoints under /lineage
provide access to these objects.
media type | data type |
---|---|
application/json | LineageRelationPath (JSON) |
POST /v2/lineage/addsource
Content-Type: application/json
{
"operationExecutionKey" : "...",
"relatedKey" : "..."
}
HTTP/1.1 201 Created
name | type | description |
---|---|---|
path | query |
media type | data type | description |
---|---|---|
application/json | array of LineageSearchPathTO (JSON) |
GET /v2/lineage/searchresource
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
[ {
"sourceName" : "...",
"sourceKey" : "...",
"resourcePath" : "...",
"resourceKey" : "..."
} ]
name | type | description | constraints | multivalued |
---|---|---|---|---|
key | path | no | ||
downstream | query | required | no | |
field | query | no | ||
fieldLineage | query | required | no | |
lineageType | query | yes | ||
upstream | query | required | no |
media type | data type | description |
---|---|---|
application/json | LineageRelations (JSON) |
GET /v2/lineage/{key}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
{
"edges" : {
"property1" : [ {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}, {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
} ],
"property2" : [ {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}, {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
} ]
},
"resources" : {
"property1" : {
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
},
"property2" : {
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}
},
"rootId" : "...",
"rootPath" : "..."
}
A DELETE request to the /lineage/{operationExecutionKey}
endpoint to delete lineage relationship.
name | type | description |
---|---|---|
operationExecutionKey | path | The unique identifying key for the operation execution. |
DELETE /v2/lineage/{operationExecutionKey}
Content-Type: */*
...
HTTP/1.1 204 No Content
name | type | description |
---|---|---|
key | path | |
opKey | query |
media type | data type |
---|---|
application/json | LineageRelationPath (JSON) |
media type | data type | description |
---|---|---|
application/json | LineageRelations (JSON) |
POST /v2/lineage/addparent/{key}
Content-Type: application/json
Accept: application/json
{
"operationExecutionKey" : "...",
"relatedKey" : "..."
}
HTTP/1.1 201 Created
Content-Type: application/json
{
"edges" : {
"property1" : [ {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}, {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
} ],
"property2" : [ {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}, {
"opDetails" : {
"executions" : [ { }, { } ],
"op" : { }
},
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
} ]
},
"resources" : {
"property1" : {
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
},
"property2" : {
"resourcePath" : "...",
"resourceId" : "...",
"resourceType" : "...",
"hasParents" : true,
"collectionRoot" : true,
"hasChildren" : true
}
},
"rootId" : "...",
"rootPath" : "..."
}
A GET request to the field-mapping-info/{operationExecutionKey}
to get the field level lineage
mappings from the parent resource to child resource.
name | type | description |
---|---|---|
operationExecutionKey | path | Operation execution key for which field lineage mappings are required. |
media type | data type | description |
---|---|---|
application/json | map of string (JSON) | Field lineage mapping from parent to child. |
GET /v2/lineage/field-mapping-info/{operationExecutionKey}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
...
name | type | description | constraints |
---|---|---|---|
key | path | ||
fieldLineage | query | required | |
fieldPath | query | ||
lineageType | query |
media type | data type | description |
---|---|---|
application/json | LineageHopCountTO (JSON) |
GET /v2/lineage/hop-count/{key}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
{
"availableUpstreamHops" : 12345,
"availableDownstreamHops" : 12345
}
A GET request to the /lineage/{key}
endpoint retrieves a lineage relationship.
name | type | description | constraints |
---|---|---|---|
key | path | The unique identifying key for the lineage relation. This key is | |
downstream | query | Number of descendants levels | required |
field | query | Name of the fieldLineage name to which lineage relation graph is required. It returns fieldLineage level lineage
only if the fieldLineage |
|
fieldLineage | query | Boolean value indicates ON/OFF toggle for filed level lineage and resource level lineage | required |
lineageType | query | Lineages type allowable types are OperationExecution.State.REJECTED |
|
upstream | query | Number of ancestors levels | required |
media type | data type | description |
---|---|---|
application/json | LineageRelationGraph (JSON) | The LineageRelation object, in JSON format. |
GET /v2/lineage/multihop/{key}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
{
"resourceFields" : [ {
"dataType" : "TIMESTAMP",
"description" : "...",
"displayName" : "...",
"nullable" : true,
"name" : "...",
"comment" : "...",
"properties" : [ {
"name" : "...",
"value" : { },
"type" : "BOOLEAN"
}, {
"name" : "...",
"value" : { },
"type" : "BOOLEAN"
} ]
}, {
"dataType" : "INTEGER",
"description" : "...",
"displayName" : "...",
"nullable" : true,
"name" : "...",
"comment" : "...",
"properties" : [ {
"name" : "...",
"value" : { },
"type" : "STRING"
}, {
"name" : "...",
"value" : { },
"type" : "PROPERTIES"
} ]
} ],
"rootId" : "...",
"rootPath" : "...",
"edges" : [ {
"target" : "...",
"source" : "..."
}, {
"target" : "...",
"source" : "..."
} ],
"nodesLookup" : {
"property1" : {
"parent" : "...",
"key" : "...",
"lineageHopLevel" : 12345,
"label" : "...",
"nodeType" : "ResourceLineageNode"
},
"property2" : {
"parent" : "...",
"key" : "...",
"lineageHopLevel" : 12345,
"label" : "...",
"nodeType" : "ResourceLineageNode"
}
},
"nodes" : [ {
"parent" : "...",
"key" : "...",
"lineageHopLevel" : 12345,
"label" : "...",
"nodeType" : "OperationNode"
}, {
"parent" : "...",
"key" : "...",
"lineageHopLevel" : 12345,
"label" : "...",
"nodeType" : "ResourceLineageNode"
} ]
}
name | type | description | constraints |
---|---|---|---|
key | path | ||
hop | query | required | |
lineageType | query | ||
systemName | query |
media type | data type | description |
---|---|---|
application/json | array of TagAssociation (JSON) |
GET /v2/lineage/tagAssociations/{key}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
[ {
"weight" : 12345.0,
"semantic" : "DATES",
"state" : "ACCEPTED",
"field" : "...",
"tagKey" : "...",
"sourceKey" : "...",
"tagName" : "...",
"rules" : [ "...", "..." ],
"resourceKey" : "...",
"domainName" : "...",
"timeOfLastChange" : 12345,
"timeOfCreation" : 12345,
"key" : "...",
"description" : "...",
"type" : "...",
"restricted" : [ "...", "..." ],
"viewAble" : [ "...", "..." ],
"updateable" : [ "...", "..." ],
"richDescription" : "...",
"score" : 12345.0,
"customProperties" : [ {
"description" : "...",
"value" : { },
"facetAble" : true,
"name" : "...",
"type" : "...",
"displayName" : "..."
}, {
"description" : "...",
"value" : { },
"facetAble" : true,
"name" : "...",
"type" : "...",
"displayName" : "..."
} ]
} ]
name | type | description | constraints |
---|---|---|---|
key | path | ||
hop | query | required | |
lineageType | query | ||
systemName | query |
media type | data type | description |
---|---|---|
application/json | array of TagDomain (JSON) |
GET /v2/lineage/tagDomains/{key}
Content-Type: */*
Accept: application/json
...
HTTP/1.1 200 OK
Content-Type: application/json
[ {
"readTags" : true,
"semantic" : "DISCOVERY_CREATED",
"color" : "...",
"description" : "...",
"deleteTags" : true,
"createTags" : true,
"name" : "...",
"canUpdateTag" : true,
"timeOfLastChange" : 12345,
"timeOfCreation" : 12345,
"key" : "...",
"type" : "...",
"restricted" : [ "...", "..." ],
"viewAble" : [ "...", "..." ],
"updateable" : [ "...", "..." ],
"richDescription" : "...",
"score" : 12345.0,
"customProperties" : [ {
"description" : "...",
"value" : { },
"facetAble" : true,
"name" : "...",
"type" : "...",
"displayName" : "..."
}, {
"description" : "...",
"value" : { },
"facetAble" : true,
"name" : "...",
"type" : "...",
"displayName" : "..."
} ]
} ]
A DELETE request to the /lineage/{edge}/{key1}/{key2}/{edgeType}
to delete an edge in a lineage relationship.
Edges Type can be any of the following
1. RESOURCE_TO_OPERATION
2. OPERATION_TO_RESOURCE
3. RESOURCE_FIELD_TO_OPERATION
4. OPERATION_TO_RESOURCE_FIELD
In case 1, key1 refers to resource key and key2 refers to operation execution key
In case 2, key1 refers to operation execution key and key2 refers to resource key
In case 3: key1 refers to resource field key and key2 refers to resource key
In case 4: key1 refers to resource field key and key2 refers to resource key
name | type | description | constraints |
---|---|---|---|
edgeType | path | "OPERATION_TO_RESOURCE" or "OPERATION_TO_RESOURCE_FIELD" or "RESOURCE_FIELD_TO_OPERATION" or "RESOURCE_TO_OPERATION" | |
key1 | path | ||
key2 | path |
DELETE /v2/lineage/edge/{key1}/{key2}/{edgeType}
Content-Type: */*
...
HTTP/1.1 204 No Content