DataResource Data Type

Type includes all the properties that describe a database, folder, table, schema, file, or Waterline Data collection. You can add properties to a data resource using a PUT call to /entityspec/\{key\/propertygroup}.

Abstract Type
Properties
name data type description
permToReadMap map of boolean
dataSourceUri string
dataSourceType string
dataSourceName string
dataSetMember boolean
dataSetId string
dataSetSchemaVersion number
reportedFields array of ReportedField
dataSetPathSpecs map of array of PathSpecification
name string
dataSetRootResource string
dataSetRoot boolean
dataSetVirtualFolders array of string
detailedStatus array of StatusDetail Specific details of the jobs run against this data resource. One entry per job type.
status string An high-level indication of what Waterline Data processing has occurred on this data resource. Options include "Processed", "Profiled", "Issue", "Failed", and "Unprocessed". The detailedStatus maps the status values to the jobs run against the data resource.
errorMsg string
runId string
resourceType string Description of the data resource type including these components, separated by underscores:
  • The prefix "resource_type"
  • followed by the data source type including "hdfs", "hive", or "jdbc"
  • followed by "database", "table", "directory", or "file"
For example, the resource type for an HDFS file would be "resource_type_hdfs_file". Note that Waterline Data collections have resource type of "resource_type_hdfs_directory".
resourcePath string Fully qualified path of the data resource. For file system resources, the path begins with a forward slash (/) and uses standard UNIX nomenclature for path names. For database tables, the path begins with a forward slash, followed by the database name, followed by a dot (.), followed by a table name.
dataSourceKey string Unique identifier of the data source where this data resource resides.
timeOfLastProfile number Time this data resource was last processed by Waterline Data. (Epoch timestamp format)
dataSchemaName string
fileFormat string MIME type of the file as determined by Waterline Data format discovery.
fileFormatDisplay string Simplified version of the fileFormat.
resourceSize number Data resource size as calculated during Waterline Data's processing.
tableName string Table name. Null if the data resource is not a table.
databaseName string Database name. Null if the data resource is not a database or a table.
fields array of AbstractField List of fields included in the schema of this data resource as determined by Waterline Data schema discovery.
owner string Owner as reported by the data source application.
tagAssociationViewList array of TagAssociationView
timeOfResourceCreation number Time this data resource was added to the Waterline Data catalog. (Epoch timestamp format)
timeOfResourceAccess number Time this data resource was last accessed, as reported by the data source application at the time of last profile. (Epoch timestamp format)
timeOfResourceChange number Time this data resource was last changed, as reported by the data source application at the time of last profile. (Epoch timestamp format)
landing boolean Not used.
ratingCount number Number of user reviews associated with this data resource.
averageRating number User rating as calculated from all the user reviews for the data resource.
rowCount number Number of data rows at the top-level of the data resource. For tabular data, this value is typically the same as the number of values per field. For complex data, this value may not reflect the number of values for each field.
favorite boolean
canAssociateTag boolean
canDisassociateTag boolean
separator string
header boolean
headerRow number
xmlRootTag string
xmlRowTag string
collectionRoot boolean Indication of whether this data resource is the top-level of a Waterline Data collection.
parquetBinaryAsString boolean
origins array of string
partitionCount number
partitionDefinition array of string
tableType string
logicalFolders array of string
logicalFolderContext string
browseContextDataset string
datasetVirtualFolderName string
virtualFolderContextName string
lastPartitionProfile boolean
joinResources array of string
joinConditions array of JoinConditionTO
canRunJob boolean
canAddOrRemoveResourceFromDataset boolean
allowedTagDomainKeys array of string
Properties inherited from Entity
key string
type string
description string
timeOfLastChange number
timeOfCreation number
restricted array of string
updateable array of string
viewAble array of string
customProperties array of CustomProperty

Example

This data type is abstract. The example below may be incomplete. More accurate examples can be found in subtypes pages.
{
  "permToReadMap" : {
    "property1" : true,
    "property2" : true
  },
  "dataSourceUri" : "...",
  "dataSourceType" : "...",
  "dataSourceName" : "...",
  "dataSetMember" : true,
  "dataSetId" : "...",
  "dataSetSchemaVersion" : 12345,
  "reportedFields" : [ {
    "dataType" : "...",
    "name" : "...",
    "label" : "...",
    "description" : "..."
  }, {
    "dataType" : "...",
    "name" : "...",
    "label" : "...",
    "description" : "..."
  } ],
  "dataSetPathSpecs" : {
    "property1" : [ {
      "sourcePath" : "...",
      "includePattern" : "...",
      "excludePattern" : "..."
    }, {
      "sourcePath" : "...",
      "includePattern" : "...",
      "excludePattern" : "..."
    } ],
    "property2" : [ {
      "sourcePath" : "...",
      "includePattern" : "...",
      "excludePattern" : "..."
    }, {
      "sourcePath" : "...",
      "includePattern" : "...",
      "excludePattern" : "..."
    } ]
  },
  "name" : "...",
  "dataSetRootResource" : "...",
  "dataSetRoot" : true,
  "dataSetVirtualFolders" : [ "...", "..." ],
  "detailedStatus" : [ {
    "startTime" : 12345,
    "endTime" : 12345,
    "stageName" : "...",
    "status" : "...",
    "workflowId" : "...",
    "runId" : "...",
    "errorMsg" : "...",
    "numberSampledRows" : 12345,
    "lastPartitionProfile" : true
  }, {
    "startTime" : 12345,
    "endTime" : 12345,
    "stageName" : "...",
    "status" : "...",
    "workflowId" : "...",
    "runId" : "...",
    "errorMsg" : "...",
    "numberSampledRows" : 12345,
    "lastPartitionProfile" : true
  } ],
  "status" : "...",
  "errorMsg" : "...",
  "runId" : "...",
  "resourceType" : "...",
  "resourcePath" : "...",
  "dataSourceKey" : "...",
  "timeOfLastProfile" : 12345,
  "dataSchemaName" : "...",
  "fileFormat" : "...",
  "fileFormatDisplay" : "...",
  "resourceSize" : 12345,
  "tableName" : "...",
  "databaseName" : "...",
  "fields" : [ {
    "name" : "...",
    "nullable" : true,
    "properties" : [ {
      "name" : "...",
      "type" : "STRING",
      "value" : { }
    }, {
      "name" : "...",
      "type" : "PROPERTIES",
      "value" : { }
    } ],
    "dataType" : "DOUBLE",
    "comment" : "...",
    "displayName" : "...",
    "importedComment" : "..."
  }, {
    "name" : "...",
    "nullable" : true,
    "properties" : [ {
      "name" : "...",
      "type" : "STRING",
      "value" : { }
    }, {
      "name" : "...",
      "type" : "BOOLEAN",
      "value" : { }
    } ],
    "dataType" : "DECIMAL",
    "comment" : "...",
    "displayName" : "...",
    "importedComment" : "..."
  } ],
  "owner" : "...",
  "tagAssociationViewList" : [ {
    "timeOfCreation" : 12345,
    "timeOfLastChange" : 12345,
    "tagKey" : "...",
    "tagName" : "...",
    "domainKey" : "...",
    "domainName" : "...",
    "field" : "...",
    "semantic" : "...",
    "state" : "...",
    "resourceKey" : "...",
    "key" : "...",
    "weight" : 12345.0,
    "rules" : [ "...", "..." ]
  }, {
    "timeOfCreation" : 12345,
    "timeOfLastChange" : 12345,
    "tagKey" : "...",
    "tagName" : "...",
    "domainKey" : "...",
    "domainName" : "...",
    "field" : "...",
    "semantic" : "...",
    "state" : "...",
    "resourceKey" : "...",
    "key" : "...",
    "weight" : 12345.0,
    "rules" : [ "...", "..." ]
  } ],
  "timeOfResourceCreation" : 12345,
  "timeOfResourceAccess" : 12345,
  "timeOfResourceChange" : 12345,
  "landing" : true,
  "ratingCount" : 12345,
  "averageRating" : 12345.0,
  "rowCount" : 12345,
  "favorite" : true,
  "canAssociateTag" : true,
  "canDisassociateTag" : true,
  "separator" : "...",
  "header" : true,
  "headerRow" : 12345,
  "xmlRootTag" : "...",
  "xmlRowTag" : "...",
  "collectionRoot" : true,
  "parquetBinaryAsString" : true,
  "origins" : [ "...", "..." ],
  "partitionCount" : 12345,
  "partitionDefinition" : [ "...", "..." ],
  "tableType" : "...",
  "logicalFolders" : [ "...", "..." ],
  "logicalFolderContext" : "...",
  "browseContextDataset" : "...",
  "datasetVirtualFolderName" : "...",
  "virtualFolderContextName" : "...",
  "lastPartitionProfile" : true,
  "joinResources" : [ "...", "..." ],
  "joinConditions" : [ {
    "leftOwner" : "...",
    "leftTimeOfLastChange" : 12345,
    "leftTimeOfResourceChange" : 12345,
    "leftPath" : "...",
    "leftSize" : 12345,
    "leftFileFormat" : "...",
    "rightOwner" : "...",
    "rightTimeOfLastChange" : 12345,
    "rightTimeOfResourceChange" : 12345,
    "rightPath" : "...",
    "rightSize" : 12345,
    "rightFileFormat" : "...",
    "joinCardinality" : "...",
    "joinOrder" : "...",
    "joinStatState" : "...",
    "leftCols" : [ "...", "..." ],
    "rightCols" : [ "...", "..." ],
    "leftResourceId" : "...",
    "rightResourceId" : "...",
    "joinOp" : "...",
    "stats" : {
      "property1" : "...",
      "property2" : "..."
    },
    "leftResourceName" : "...",
    "rightResourceName" : "...",
    "leftSourceName" : "...",
    "rightSourceName" : "...",
    "leftSourceId" : "...",
    "rightSourceId" : "...",
    "tags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "ACCEPTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "leftTags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "ACCEPTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "REGEX",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "REJECTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "rightTags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "rightOrigin" : [ "...", "..." ],
    "leftOrigin" : [ "...", "..." ]
  }, {
    "leftOwner" : "...",
    "leftTimeOfLastChange" : 12345,
    "leftTimeOfResourceChange" : 12345,
    "leftPath" : "...",
    "leftSize" : 12345,
    "leftFileFormat" : "...",
    "rightOwner" : "...",
    "rightTimeOfLastChange" : 12345,
    "rightTimeOfResourceChange" : 12345,
    "rightPath" : "...",
    "rightSize" : 12345,
    "rightFileFormat" : "...",
    "joinCardinality" : "...",
    "joinOrder" : "...",
    "joinStatState" : "...",
    "leftCols" : [ "...", "..." ],
    "rightCols" : [ "...", "..." ],
    "leftResourceId" : "...",
    "rightResourceId" : "...",
    "joinOp" : "...",
    "stats" : {
      "property1" : "...",
      "property2" : "..."
    },
    "leftResourceName" : "...",
    "rightResourceName" : "...",
    "leftSourceName" : "...",
    "rightSourceName" : "...",
    "leftSourceId" : "...",
    "rightSourceId" : "...",
    "tags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "ACCEPTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "leftTags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "REGEX",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "ACCEPTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "VALUE",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "rightTags" : [ {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "REGEX",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    }, {
      "children" : [ { }, { } ],
      "domainKey" : "...",
      "parentKey" : "...",
      "name" : "...",
      "tagState" : "REGEX",
      "minScore" : 12345.0,
      "regEx" : "...",
      "regexMinLen" : 12345,
      "regexMaxLen" : 12345,
      "regexFilterClass" : "...",
      "learningEnabled" : true,
      "immutable" : true,
      "enable" : true,
      "fullName" : "...",
      "state" : "SUGGESTED",
      "externalId" : "...",
      "externalSourceName" : "...",
      "synonyms" : [ "...", "..." ],
      "key" : "...",
      "type" : "...",
      "description" : "...",
      "timeOfLastChange" : 12345,
      "timeOfCreation" : 12345,
      "restricted" : [ "...", "..." ],
      "updateable" : [ "...", "..." ],
      "viewAble" : [ "...", "..." ],
      "customProperties" : [ { }, { } ]
    } ],
    "rightOrigin" : [ "...", "..." ],
    "leftOrigin" : [ "...", "..." ]
  } ],
  "canRunJob" : true,
  "canAddOrRemoveResourceFromDataset" : true,
  "allowedTagDomainKeys" : [ "...", "..." ],
  "key" : "...",
  "type" : "...",
  "description" : "...",
  "timeOfLastChange" : 12345,
  "timeOfCreation" : 12345,
  "restricted" : [ "...", "..." ],
  "updateable" : [ "...", "..." ],
  "viewAble" : [ "...", "..." ],
  "customProperties" : [ {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  }, {
    "displayName" : "...",
    "value" : { },
    "description" : "...",
    "facetAble" : true,
    "name" : "...",
    "type" : "..."
  } ]
}