Jyoti Wadhwani
7655ab25eb
Update townhalls.md
...
Update with the latest recording
2020-06-30 15:44:56 -07:00
Liangjun Jiang
5d078aa617
Implemented data process search feature ( #1706 )
...
* implement search feature
* add test for dataprocessIndexBuilder; refactor code based on feedback
* update based on PR feedback
* Update DataProcessDocument.pdl
fixed typo wording.
* add not null check for data process info
2020-06-29 10:20:22 -07:00
Kerem Sahin
2dc11a51f4
fix(py3): Bump ingestion Docker py dependency to 3.6 ( #1716 )
2020-06-29 08:22:50 -07:00
Mars Lan
982cc50cf9
Update README.md
2020-06-26 07:28:52 -07:00
Kerem Sahin
9501e9bd70
docs: Graph onboarding demo
2020-06-26 01:10:44 -07:00
Mars Lan
34d6f4ed09
Update README.md
2020-06-25 19:24:03 -07:00
Mars Lan
65bf623b8b
feat(ingest): add snowflake ETL script ( #1714 )
2020-06-25 19:05:38 -07:00
Mars Lan
682bb87a7e
feat(ingest): replace custom hive-etl with sql-based ETL ( #1713 )
...
This offloads most of the heavy lifting to SQLAlchemy.
Also add a docker file for testing
2020-06-25 19:04:56 -07:00
Mars Lan
5da55fe8d3
Update README.md
2020-06-25 16:32:22 -07:00
Mars Lan
52a54b9fda
feat(ingest): add PostgreSQL ETL script ( #1712 )
...
Also add a simple docker file for testing
2020-06-25 15:28:42 -07:00
Mars Lan
221c9af220
feature(ingest): add bigquery ETL script ( #1711 )
...
Also fix minor issues in the common script
2020-06-25 15:28:13 -07:00
Mars Lan
fa9fe5e110
refactor(py3): Refactor all ETL scripts to using Python 3 exclusively ( #1710 )
...
* refactor(py3): Refactor all ETL scripts to using Python 3 exclusively
Fix https://github.com/linkedin/datahub/issues/1688
* Update requirements.txt
2020-06-25 15:16:04 -07:00
Jyoti Wadhwani
60b7c63b26
[fix] load default picture link if not present ( #1709 )
2020-06-25 14:35:22 -07:00
Mars Lan
776d6b37af
Update README.md
2020-06-23 15:54:36 -07:00
Mars Lan
8e6665fc94
Update README.md
2020-06-22 21:26:38 -07:00
Mars Lan
4fea6083f8
feature(etl): add SQLAlchemy-based ingestion script ( #1708 )
...
This replaces the old incomplete rdbms ETL script.
2020-06-22 21:25:55 -07:00
Charlie Tran
3db9e1efc6
Support for volta in web client ( #1707 )
...
* Support for volta in web client
* Make sure yarn and volta version of yarn match
2020-06-22 13:14:08 -07:00
geosmart
3e0d658e62
fix-DatasetSearchConfig class ref ( #1704 )
2020-06-19 08:14:08 -07:00
Liangjun Jiang
40f08ecaf1
Implement data process graph feature ( #1695 )
...
* implement data process graph feature; update the document
* add unit test for data process graph
* removed auto generated avro files
* update image
2020-06-17 11:58:42 -07:00
Jyoti Wadhwani
7fc4d85ceb
Update townhalls.md
...
Update the agenda for next town hall
2020-06-15 15:55:39 -07:00
Kerem Sahin
ccd7c1f38a
Update townhalls.md
2020-06-15 15:33:14 -07:00
Kerem Sahin
2e2fb2b810
Add missing updates from recent internal push ( #1700 )
2020-06-12 12:55:50 -07:00
Javier Sotelo
aa65b46be7
fix: add missing neo4j.host helm var ( #1698 )
...
Co-authored-by: jsotelo <javier.sotelo@viasat.com>
2020-06-11 12:03:51 -07:00
Javier Sotelo
e7be6db936
fix: remove helm container command ( #1697 )
...
The current helm command is not compatible with the
docker image. Instead, rely on the docker image's CMD.
Co-authored-by: jsotelo <javier.sotelo@viasat.com>
2020-06-11 10:48:53 -07:00
Jyoti Wadhwani
ad6f1653e1
metadata-models 62.0.3 -> 72.0.8 ( #1693 )
2020-06-11 10:21:51 -07:00
Kerem Sahin
f79b2c958a
fix(ingestion): Fix sample MCE for data process
2020-06-11 01:04:52 -07:00
Jyoti Wadhwani
5fc80ae249
Update README.md
...
Align the downstream and upstream lineage APIs with other examples
2020-06-10 18:44:20 -07:00
Kerem Sahin
66271923a7
Update readme for mae-consumer-job
2020-06-09 18:10:29 -07:00
dependabot[bot]
70b2123e60
build(deps): bump websocket-extensions in /datahub-web ( #1694 )
...
Bumps [websocket-extensions](https://github.com/faye/websocket-extensions-node ) from 0.1.3 to 0.1.4.
- [Release notes](https://github.com/faye/websocket-extensions-node/releases )
- [Changelog](https://github.com/faye/websocket-extensions-node/blob/master/CHANGELOG.md )
- [Commits](https://github.com/faye/websocket-extensions-node/compare/0.1.3...0.1.4 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-06-09 16:51:16 -07:00
Liangjun Jiang
92c4a3689e
Data process entity ( #1680 )
...
* add job info as aspect of a dataset
* add job urn def., aspect and entity
* job entity with upstream and downstream lineage
* use job urn in upstream & downstream
* add Job entity rest APIs
* rest.li api, impl and factory for job entity
* code cleanup
* use pdl; onboard data process entity
* add es index json
* fix gradlew build ignored tasks
* add a comment about data process info field
* fix style warning issues
* update content based on PR
* checked in generated snapshot json
* updated based on PR feedback
* update data process data format
* updated based on code review feedback
* revert back gms & mce-job docker image
* delete temp files
* update based pr feedback
* file name and a typo
* format with linkedin style
Co-authored-by: Liangjun <liajiang@expediagroup.com>
2020-06-09 15:42:08 -07:00
Mars Lan
4f221f9a12
build(docker): refactor docker build scripts ( #1687 )
...
* build(docker): refactor docker build scripts
- add "build" option to docker-compose files to simplify rebuilding of images
- create "start.sh" script so it's easier to override "command" in the quickstart's docker-compose file
- use dockerize to wait for requisite services to start up
- add a dedicated Dockerfile for kafka-setup
This fixes https://github.com/linkedin/datahub/issues/1549 & https://github.com/linkedin/datahub/issues/1550
2020-06-08 13:37:14 -07:00
Mars Lan
94ffb300a9
build(docker): refactor ingestion docker build script ( #1690 )
...
- add "build" option to docker-compose file to simplify rebuilding of images
- move command from docker-compose.yml to Dockerfile
- add ingestion.sh script to simplify quickstart instruction and to reduce confusion
2020-06-05 14:39:20 -07:00
Jyoti Wadhwani
90e449d25e
upgrade the version of neo4j ( #1691 )
2020-06-04 22:14:57 -07:00
Mars Lan
867dbd0d36
fix: use tuple notations for union types
2020-06-03 15:36:07 -07:00
Mars Lan
b6589ab1d1
Update README.md
2020-06-03 13:52:56 -07:00
Jyoti Wadhwani
8a8dd3de77
move the gradle plugin version to top level build.gradle ( #1685 )
2020-06-01 19:00:19 -07:00
Jyoti Wadhwani
be75856597
Update townhalls.md
...
Change town hall details
2020-06-01 16:46:02 -07:00
Jyoti Wadhwani
d88217c769
Update README.md
...
Update next town hall meeting
2020-06-01 09:51:26 -07:00
Mars Lan
920a1774dc
docs: points to docker images hosted by linkedin org ( #1683 )
...
As we're now utilize GitHub Actions to build & publish docker images to docker hub under linkedin org
Also allow overriding image tags via DATAHUB_VERSION environment variable
2020-06-01 09:36:51 -07:00
Mars Lan
63943a1ca8
build: update workflows to build version-tagged docker images upon new release
2020-05-29 20:30:44 -07:00
Mars Lan
5cf27e7d08
Update docker-release.yml
2020-05-29 18:42:20 -07:00
Mars Lan
3c1825dab2
Update docker-release.yml
2020-05-29 18:36:07 -07:00
Mars Lan
57ba03b3b6
Update docker-release.yml
2020-05-29 18:12:55 -07:00
Mars Lan
387a207cc0
Update docker-release.yml
2020-05-29 18:11:56 -07:00
Mars Lan
875e3f078b
Update docker-release.yml
2020-05-29 18:11:09 -07:00
Mars Lan
28c9b2b53d
Update docker-release.yml
2020-05-29 18:09:53 -07:00
Mars Lan
9e18519ddd
Update docker-release.yml
2020-05-29 17:59:10 -07:00
Mars Lan
8ef7fae113
Rename docker-release to docker-release.yml
2020-05-29 17:50:47 -07:00
Mars Lan
c91534d315
build: release-tagged docker images
2020-05-29 17:49:58 -07:00
Mars Lan
2a2989609f
build: use shorter workflow names
...
To generate shorter badges: https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#adding-a-workflow-status-badge-to-your-repository
2020-05-29 16:53:13 -07:00
