yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-06-27 05:03:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Page: Frontend API
Pages
Architecture Azkaban Execution Backend API Coding Guidelines Frontend API Frontend User Guide Getting Started Git Commit Metadata ETL Gobblin Kafka Integeration Gobblin Kafka Integration HDFS Dataset Ownership Metadata ETL HDFS Dataset Hive Dataset Home Integration Guide Kafka Integration Guide LDAP Directory Metadata ETL Lineage Multiproduct Metadata ETL Oozie Execution Oracle Dataset Quick Start with VM Set Up New Metadata ETL Jobs Teradata Dataset
Clone
3
Frontend API
Will Norvelle edited this page 2017-06-20 12:39:20 -04:00
Table of Contents

Table of Contents

  1. Datasets
  1. Metrics
  1. Flows
  1. Basic Search
  1. Advanced Search
## Datatsets ### Datasets By URN ##### Accepted Query Params | Name | Function | Default | Required | |:---|---|---|:---:| | size | Page Size | 10 | Y | | page | Page | 1 | Y | | urn | URN to search | | Y |
Example
curl $WHEREHOWS_URL/api/v1/datasets?size=1&page=1&urn=hdfs://
### Datasets By ID
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId
### Dataset Schema
Example
curl $WHEREHOWS/api/v1/datasets/:datasetId/columns
#### Get Dataset Column Comments
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/columns/:columnId/comments
#### Create Dataset Column Comments
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    url: '/api/v1/datasets/:datasetId/columns/:columnId/comments',
    method: 'POST',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      text: 'Blarg',
      csrfToken: token
    }
  })
})(jQuery)
#### Update Dataset Column Comments
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    url: '/api/v1/datasets/:datasetId/columns/:columnId/comments',
    method: 'POST',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      id: 35,
      text: 'Blarg',
      csrfToken: token
    }
  })
})(jQuery)
#### Delete Dataset Column Comments
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    url: '/api/v1/datasets/:datasetId/columns/:columnId/comments/:commentId',
    method: 'DELETE',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      csrfToken: token
    }
  })
})(jQuery)
#### Similar Dataset Column Comments The endpoint provides a list of comments that belong to columns of a similar name. This can be helpful in the re-use of comments. Allows Datasets that share a field (or Foreign Key) to pass comments along to that datasets column.
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/columns/:columnId/comments/similar
#### Promote Dataset Column Comments The endpoint provides means with which a user can promote a column comment to be the default comment for that column.
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    method: 'PATCH',
    url: '/api/v1/datasets/:datasetId/columns/columnId/comments',
    headers: {
      'Csrf-Token': token    
    }
    dataType: 'json',
    contentType: 'application/json',
    data: JSON.stringify({
      commentId: 3599,
      csrfToken: token
    })
  })
})(jQuery)
#### Dataset Column Comment Propagation The endpoint provides means with which a user can propagate a column comment to be the default column comment of multiple similar comments
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    method: 'PATCH',
    url: '/api/v1/datasets/:datasetId/columns/columnId/comments?csrfToken=' + token,
    headers: {
      'Csrf-Token': token    
    }
    dataType: 'json',
    contentType: 'application/json',
    data: JSON.stringify
    ( [ { commentId: 3599
        , datasetId: 531451
        , columnId: 6545345
        }
      , { commentId: 3599
        , datasetId: 53158
        , columnId: 548648
        }
      ]
    )
  })
})(jQuery)
### Dataset Properties
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/properties
### Dataset Sample Data
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/sample
### Dataset Impact Analysis
Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/impacts
### Dataset Comments #### Get Dataset Comments ##### Accepted Query Params for GET | Name | Function | Default | Required | |:--|---|---|:--:| | size | Size of response per page | 3 | N | | page | Page of response to return | 1 | Y |
GET Example
curl $WHEREHOWS_URL/api/v1/datasets/:datasetId/comments
#### Create Dataset Comment ##### Parameters | Name | Function | Required | |:--|---|:--:| | datasetId | ID of dataset for comment | Y | | text | Comment Text (text or markdown) | Y | | type | Comment Type | Comment | Y | | csrfToken | Csrf Token | Y |
POST Headers
Name Function Required
Csrf-Token CSRF Token Y
Example
(function($){
  var token = $("#csrfToken").val().replace('/','')
  $.ajax({
    url: '/api/v1/datatsets/:datasetId/comments',
    method: 'POST',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      datasetId: 1,
      text: "test test blargs",
      type: "Comment",
      csrfToken: token
    }   
  })
})(jQuery)
#### Update Dataset Comment ##### Parameters | Name | Function | Required | |:--|---|:--:| | datasetId | ID of dataset for comment | Y | | text | Comment Text (text or markdown) | Y | | type | Comment Type | comment | Y | | csrfToken | Csrf Token | Y | | id | Comment ID | Y |
Headers
Name Function Required
Csrf-Token CSRF Token Y
Example
(function($){
  var token = $("#csrfToken").val().replace('/','')
  $.ajax({
    url: '/api/v1/datatsets/:datasetId/comments',
    method: 'POST',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      datasetId: 1,
      id: 234304, // comment id
      text: "test test blargs",
      type: "Comment",
      csrfToken: token
    }   
  })
})(jQuery)
#### Delete Dataset Comment Note: Currently this actually deletes the record. Only the user that created the comment can delete it.
Headers
Name Function Required
Csrf-Token CSRF Token Y
Params
Name Function Required
Csrf-Token CSRF Token Y
Example
(function($){
  var token = $("#csrfToken").val().replace('/','')
  $.ajax({
    url: '/api/v1/datatsets/:datasetId/comments/:commentId',
    method: 'DELETE',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      csrfToken: token
    }   
  })
})(jQuery)
### Dataset Favoriting A shortcut system for datasets a user finds useful.
Favoriting
(function($){
  var token = $("#csrfToken").val().replace('/','')
  $.ajax({
    url: '/api/v1/datasets/:datasetId/favorite',
    method: 'POST',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      csrfToken: token
    }
  })
})(jQuery)
Un-Favoriting
(function($){
  var token = $("#csrfToken").val().replace('/','')
  $.ajax({
    url: '/api/v1/datasets/:datasetId/favorite',
    method: 'DELETE',
    headers: {
      'Csrf-Token': token
    },
    dataType: 'json',
    data: {
      csrfToken: token
    }
  })
})(jQuery)
### Dataset Watching Watch a dataset for changes. Allow for email notification of changes (Coming Soon) #### Watch A Dataset ##### Example ```javascript (function($){ var token = $("#csrfToken").val().replace('/','') $.ajax({ url: '/api/v1/datasets/:datasetId/watch', method: 'POST', headers: { 'Csrf-Token': token }, dataType: 'json', data: { csrfToken: token, item_type: 'dataset', notification_type: 'weekly' } }) })(jQuery) ``` #### Updating the Watch of a Dataset Currently there is no way to update the notification_type. This will likely come with the email feature. #### Un-watch A Dataset ##### Example ```javascript (function($){ var token = $("#csrfToken").val().replace('/','') $.ajax({ url: '/api/v1/datasets/:datasetId/watch/:watchId', method: 'DELETE', headers: { 'Csrf-Token': token }, dataType: 'json', data: { csrfToken: token } }) })(jQuery) ``` ## Metrics ### All Metrics
Accepted Query Parameters
Name Function Default Required
Size Size of response array 10 N
Page Page For Response 1 N
Example
curl $WHEREHOWS_URL/api/v1/metrics?size=4&page=1
### Metrics By Dashboard
Accepted Query Parameters
Name Function Default Required
Size Size of response array 10 N
Page Page For Response 1 N
Example
curl $WHEREHOWS_URL/api/v1/metrics/name/:dashboardName?size=4&page=1
### Metrics By Dashboard and Group
Accepted Query Parameters
Name Function Default Required
Size Size of response array 10 N
Page Page For Response 1 N
Example
curl $WHEREHOWS_URL/api/v1/metrics/name/:dashboardName/:groupName?size=4&page=1
### Metric By ID
Example
curl $WHEREHOWS_URL/api/v1/metrics/:id
### Updating Metrics

This API allows for full or partial update of fields within a specific metric.

Required Headers
Name Function
Csrf-Token CSRF-Token
Accepted Input
Name Function
id Metric ID
name Metric Name
dashboardName Associated Dashboard
group Associated Dashboard Group
category Business Category
description Metric Description
formula Metric Formula (SQL, PIG, Hive)
displayFactor
displayFactorSym
grain
productPageKeyGroupSK
refID
refIDType
schema
sourceType
urn Metric Urn
Example
(function($){
  var token = $("#csrfToken").val().replace('/', '')
  $.ajax({
    url: '/api/v1/metrics/:metricID/update',
    method: 'POST',
    headers: {
      'Csrf-Token': token    
    },
    dataType: 'json',
    data: {
      csrfToken: token,
      formula: "Pig: /r/n set mapred.min.spite.size 1073741824",
      id: 23
    }
  })
})(jQuery)
### Watching Metrics #### Watching Metrics ##### Example ```javascript (function($){ var token = $("#csrfToken").val().replace('/', '') $.ajax({ url: '/api/v1/metrics/:id/watch', method: 'POST', headers: { 'Csrf-Token': token }, dataType: 'json', data: { csrfToken: token, item_type: "metric", notification_type: "weekly" } }) })(jQuery) ``` #### Unwatch Metric ##### Example ```javascript (function($){ var token = $("#csrfToken").val().replace('/', '') $.ajax({ url: '/api/v1/metrics/:id/watch/:watchId', method: 'DELETE', headers: { 'Csrf-Token': token }, dataType: 'json', data: { csrfToken: token } }) })(jQuery) ``` ## Flows ### Flows by Application (cluster)
Accepted Query Parameters
name function default required
size size of response array 10 n
page page number 1 n
Example
curl $WHEREHOWS_URL/api/v1/flows/:application?page=1&size=5
### Flows By Application and Project
Accepted Query Parameters
name function default required
size size of response array 10 n
page page number 1 n
Example
curl $WHEREHOWS_URL/api/v1/flows/:application/:project?page=1&size=5
### Flows by Application, Project, and Flow
Accepted Query Parameters
name function default required
size size of response array 10 n
page page number 1 n
Example
curl $WHEREHOWS_URL/api/v1/flows/:application/:project/:flow?page=1&size=5
## Basic Search ### Search Auto-complete

Returns list of datasets, metrics, and flows for auto-complete

Example
curl $WHEREHOWS_URL/api/v1/autocomplete/search
### Search

Accepted Query Parameters

Name Function
page Page of results to return
size Number of results to return
category Category: [dataset, metric, comment]
source Source (Dataset Source ex. Oracle, Teradata, HDFS)
Example
curl $WHEREHOWS_URL/api/v1/search/:keyword?page=1&category=dataset&source=all
## Advanced Search ### Scopes
Example
curl $WHEREHOWS_URL/api/v1/advsearch/scopes
### Tables ##### Example ```bash curl $WHEREHOWS_URL/api/v1/advsearch/tables ``` ### Fields ##### Example ```bash curl $WHEREHOWS_URL/api/v1/advsearch/fields ``` ### Comments ##### Example ```bash curl $WHEREHOWS_URL/api/v1/advsearch/comments ``` ### Sources ##### Example ```bash curl $WHEREHOWS_URL/api/v1/advsearch/sources ``` ### Search
Accepted Query Parameters
Name Function
searchOpts stringified JSON object
Sample Search Opts
{ "scope":
  { "in": "ADDRESS_BOOK"
  , "not": ""
  }
, "table":
  { "in": "ADDRESS_BOOK_DATA"
  , "not": ""
  }
, "fields":
  { "any": "member_id"
  , "all": ""
  , "not": ""
  }
, "comments": ""
, "sources": "Teradata, Hdfs, Oracle"
}
Example
curl $WHEREHOWS_URL/api/v1/advsearch/search?searchOpts=$SEARCH_OPTS

User Guide

Developer Guide

Metadata ETL Types

A LinkedIn Product
Tech Blog | Jobs