12076 Commits

Author SHA1 Message Date
Harshal Sheth
c7892ada4c Codegen avro + datahub kafka sink (#3)
* Add codegen

* New architecture + setup file -> console pipeline

* Cleanup source loader

* Basic Kafka metadata source

* Kafka source and extractor

* Add kwargs construct interface

* Fix kafka source unit test

* start working on pipeline test

* kafka datahub sink

* Make myself a profile

* Ingest to datahub from kafka

* Update codegen

* Add restli transport

* Fix bug in restli conversion
2021-02-15 18:29:27 -08:00
Shirshanka Das
b59a62fa1c setting modest coverage targets 2021-02-15 18:29:27 -08:00
Shirshanka Das
35e9f28b56 dropping unit tests 2021-02-15 18:29:27 -08:00
Shirshanka Das
60d861b498 dropping hard failures 2021-02-15 18:29:27 -08:00
Shirshanka Das
5e589da514 adding python action 2021-02-15 18:29:27 -08:00
Shirshanka Das
6b5bbbdc5f workaround for docker exec, waiting for 5 more seconds 2021-02-15 18:29:27 -08:00
Harshal Sheth
4fb673925c Start using avro producer 2021-02-15 18:29:27 -08:00
Shirshanka Das
9e61220132 checking in testing fixtures. docker still not working 2021-02-15 18:29:27 -08:00
Shirshanka Das
1ddbdee60c Support for SQL databases (MySQL + MS-SQL) (#2)
* adding sql source + mysql

* adding sql support

* MSSQL support, basic integration test

* file sink and pipeline context
2021-02-15 18:29:27 -08:00
Shirshanka Das
faf472aa64 adding some TODOs 2021-02-15 18:29:27 -08:00
Shirshanka Das
128781942d Firstdrop of ingest (#1) 2021-02-15 18:29:27 -08:00
Shirshanka Das
90b635fb7c Initial commit 2021-02-15 18:29:27 -08:00
Harshal Sheth
082c86463e Move old metadata ingestion scripts out of the way 2021-02-15 18:29:27 -08:00
John Joyce
54346b2e09 Fix build args 2021-02-15 17:40:41 -08:00
John Joyce
ef58e3ca43 rename 2021-02-15 17:40:41 -08:00
John Joyce
b6b13e88c2 Adding path 2021-02-15 17:40:41 -08:00
John Joyce
11f030b118 Adding Github Action to publish Docker image 2021-02-15 17:40:41 -08:00
John Joyce
98bfb9ac8e Dockerization 2021-02-15 17:40:41 -08:00
John Joyce
ef04205a73 Serving React App via Docker 2021-02-15 17:40:41 -08:00
John Joyce
e3ac44cfd4 Fixes 2021-02-15 17:40:41 -08:00
John Joyce
8ef5e2a545 Deploy React via Docker 2021-02-15 17:40:41 -08:00
John Joyce
da271e06a1
refactor(React): Renaming components/ dir to app/ (#2104) 2021-02-12 13:57:38 -08:00
John Joyce
9af0e56a34
docs(React): React README (#2103)
* Adding React Documentation

* Add image registry

Co-authored-by: John Joyce <john@acryl.io>
2021-02-12 10:48:40 -08:00
John Joyce
98b14b84e2
fix(React): Fix Dataset Ownership Role Update Bug (#2102)
* Fix Ownership Update Bugs

Co-authored-by: John Joyce <john@acryl.io>
2021-02-12 10:30:51 -08:00
Gabe Lyons
a9069d4d94
feat(React): Powering lineage page through graphql endpoint (#2099) 2021-02-10 14:19:27 -08:00
Gabe Lyons
2e8ebcf109 feat(React): powering user page via graphql 2021-02-10 14:16:03 -08:00
Gabe Lyons
7f2ca12e36
feat(React): deprecation status in header (#2097) 2021-02-09 19:05:50 -08:00
Gabe Lyons
c2698b99f1
feat(React): Adding schema tab on dataset entity (#2096) 2021-02-09 18:14:10 -08:00
John Joyce
2645aa0b5f
feat(React): Adding Properties, Documentation tabs, user search, & supporting Dataset mutations (#2093)
* Adding Properties, Documentation tabs, & supporting mutations

Co-authored-by: John Joyce <john@acryl.io>
2021-02-09 14:30:23 -08:00
brendansun93
cbbe6a5c4b
fix (React): Fix broken profile link on ownership + header (#2095)
Co-authored-by: Brendan Sun <shihaosun@geotab.com>
2021-02-09 13:41:32 -08:00
Harshal Sheth
ad745a2a74
fix: Bump gma version (#2094) 2021-02-09 12:07:02 -08:00
John Joyce
da1258c335
feat(GraphQL Mutation): Introducing updateDataset mutation (#2091)
* Introducing updateDataset mutation

* WritableType -> MutableType

* Add createdAt input

Co-authored-by: John Joyce <john@acryl.io>
2021-02-09 10:08:28 -08:00
Arpan Majumder
b3dfa1e4ae
2056 | Change dataset lineage pendulum animation component (#2061) 2021-02-08 15:33:15 -08:00
Gabe Lyons
99e0a110d9
feat(React Incubation): adding lineage page with sample data (#2090) 2021-02-08 15:23:47 -08:00
mahao
6c04ba1ebc
fix(docs): update URN models and DatasetUrn links (#2087)
Signed-off-by: mahao <allenhaozi@gmail.com>
2021-02-07 11:33:54 -08:00
Gabe Lyons
dcefd96b4a
feat(React Incubation): User Profile (#2083)
* feat(react incubation): user profile w/ mock data

* removing spurious build-and-test change

* fixing CI issues
2021-02-05 14:21:04 -08:00
John Joyce
b0801de7ad
fix(GraphQL): Add schemaMetadata mapping & correctly form path string (#2084)
* Fixing DataHub frontend Browse bug

* Add schema mapping

Co-authored-by: John Joyce <john@acryl.io>
2021-02-05 12:56:50 -08:00
Gabe Lyons
b73723b98c
fix(react): including eslint in CI (#2086) 2021-02-05 12:56:07 -08:00
John Joyce
cb2f3a466d
fix(integ test): Bumping GMA to 0.2.40 (#2085)
Co-authored-by: John Joyce <john@acryl.io>
2021-02-05 11:51:56 -08:00
Harshal Sheth
b491e4ad3c
fix(SQL ingest): Bump confluent-kafka version (#2082)
This should help resolve #2079.
2021-02-03 18:06:35 -08:00
John Joyce
9d38ae4dd4
refactor(React Incubation): Entity Interface & EntityRegistry (#2077)
* React App: Entity Registry!  (#15)

Introducing the Entity Registry: an Entity-oriented Architecture

Co-authored-by: John Joyce <john@acryl.io>
2021-02-03 11:49:51 -08:00
John Joyce
4cdef602b4
Enable mutation only in frontend (#2081)
Co-authored-by: John Joyce <john@acryl.io>
2021-02-03 11:34:16 -08:00
John Joyce
1a49785add
feat(GraphQL Query): Supporting complete Dataset, CorpUser with Search & Browse in GraphQL API (#2076)
Co-authored-by: John Joyce <johnjoyce@johns-mbp.home>
2021-02-02 20:18:31 -08:00
Arun Vasudevan
84e952e138
feat (graphql): Datahub GMS Graphql Api Application for Querying Dataset (#2071) 2021-02-01 11:51:15 -08:00
Gabe Lyons
148c733673
refactor(React Incubation): Search routing refactor (#2075) 2021-02-01 11:50:10 -08:00
Shirshanka Das
ebfc3efe62
docs: Updating README with upcoming TownHall info (#2074)
* docs: Updating README with upcoming townhall info

* nit: whitespace
2021-01-29 15:01:48 -08:00
John Joyce
9e66aa0649
Fixing CorpUsers Search Bug (#2073)
Co-authored-by: John Joyce <john@acryl.io>
2021-01-29 12:08:50 -08:00
Mars Lan
7a786c185b
Drop obsolete info on mysql-etl (#2072) 2021-01-29 09:03:53 -08:00
John Plaisted
d89fa4a27b
feat: update GMA to 0.2.35 (#2067)
Required editing the search tests for the new search config tests.
2021-01-28 15:55:51 -08:00
Gabe Lyons
e2e3aca478
fix (react): fixing browse routing (#2069)
* fixing browse routing
* including react app in build
2021-01-27 10:52:46 -08:00