Harshal Sheth
3edbdcb3e9
ci(ingest): ensure datahub imports work ( #2502 )
2021-05-06 07:48:56 -07:00
Harshal Sheth
1facfbd5a3
feat(ingest): capture table properties if available ( #2497 )
2021-05-05 14:07:08 -07:00
Harshal Sheth
7f0656fd5e
fix(ingest): replace ImportError with ModuleNotFoundError ( #2498 )
...
Using the more specific exception will prevent us from accidentally
ignoring errors that should be handled.
2021-05-05 14:05:16 -07:00
Harshal Sheth
c32bf494d5
fix(ingest): support https connections with cookies in Hive ingestion ( #2489 )
...
Tested locally.
2021-05-04 13:10:52 -07:00
Harshal Sheth
6f1f0a4845
feat(ingest): support hive over http ( #2486 )
2021-05-03 22:11:50 -07:00
Harshal Sheth
9f4de4b20a
fix(ingest): remove datahub.metadata import shortcut ( #2449 )
2021-04-30 21:10:12 -07:00
Harshal Sheth
90108f4aba
docs(ingest): add gradle build step to developing setup as well ( #2479 )
2021-04-30 21:07:51 -07:00
Harshal Sheth
71933a9f31
test(ingest): rename TestSource -> FakeSource ( #2481 )
2021-04-30 20:54:07 -07:00
Harshal Sheth
201ffd4979
test: add smoke test ( #2464 )
2021-04-29 23:27:03 -07:00
Harshal Sheth
df9e7c594f
fix(ingest): guess hook type from name ( #2475 )
2021-04-29 23:23:19 -07:00
Harshal Sheth
e48a74b80a
test(ingest): add test names and IDs using pytest ( #2476 )
2021-04-29 23:18:55 -07:00
Harshal Sheth
5553dc820c
fix(ingest): use postgres data platform urn ( #2472 )
2021-04-28 11:34:45 -07:00
Rickard Cardell
48ea19a5f7
feat(react): add custom properties tab on dashboard profile page ( #2439 ) ( #2469 )
2021-04-27 11:26:09 -07:00
Brian
5894365d77
fix(bootstrap): update bootstrap data with more realistic nested schemas ( #2470 )
2021-04-27 11:19:40 -07:00
Gabe Lyons
3bf5ffab5d
docs(ingest): Update README.md to add superset source ( #2466 )
2021-04-26 23:10:27 -07:00
Harshal Sheth
50aee5c05a
fix(ingest): support Airflow 1.10.x style lineage in Airflow 2 ( #2455 )
2021-04-26 23:08:43 -07:00
Harshal Sheth
694c4ce8ec
test(ingest): fix mypy issue in schema util test ( #2465 )
2021-04-26 18:18:37 -07:00
Harshal Sheth
83fdc6417f
feat(ingest): capture default values in Avro schemas ( #2463 )
2021-04-26 17:07:29 -07:00
Harshal Sheth
d415234a8c
fix(ingest): fields with defaults should be optional ( #2461 )
2021-04-26 16:45:48 -07:00
Harshal Sheth
2da5e1fd10
feat(ingest): setup scaffolding for tox testing ( #2451 )
2021-04-26 16:44:36 -07:00
Harshal Sheth
e075acaec9
docs(ingest): clarify options field and fix bigquery sample config ( #2456 )
2021-04-26 16:30:05 -07:00
Dexter Lee
6554f15e67
fix(docker): Nuke ingestion containers when calling docker/nuke.sh ( #2459 )
2021-04-26 16:29:25 -07:00
Harshal Sheth
f6c7195ac5
fix(cli): check docker setup containers ( #2457 )
2021-04-26 16:28:49 -07:00
Harshal Sheth
c64196e8cf
fix(ingest): setup pyproject.toml ( #2446 )
2021-04-23 23:50:28 -07:00
Harshal Sheth
a857d3b9d8
fix(ingest): various updates to datahub rest sink ( #2445 )
2021-04-23 23:48:44 -07:00
Harshal Sheth
663dfe9a7c
fix(ingest): add snowflake warehouse and role to config ( #2444 )
2021-04-23 23:46:31 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
adriaanslechten
1295c44615
feat(ingest) LDAP groups ingestion ( #2434 )
2021-04-22 13:56:30 -07:00
Harshal Sheth
034c33a050
fix(ingest): use entrypoints lib instead of pkg_resources ( #2438 )
2021-04-22 00:13:47 -07:00
Gabe Lyons
c7b49de67b
feat(ingest): adding superset ingestion source ( #2425 )
2021-04-22 00:11:54 -07:00
Harshal Sheth
56b66ed328
fix(ingest): support custom snowflake types ( #2436 )
2021-04-21 15:14:59 -07:00
Harshal Sheth
ffe49f061a
fix(ingest): fix chart type enum serialization and add tests for rest emitter ( #2429 )
2021-04-21 11:34:24 -07:00
Harshal Sheth
79daec29b7
fix(ingest): ensure upstreams in airflow lineage emission are entities ( #2427 )
2021-04-20 20:44:38 -07:00
Harshal Sheth
7d1ec520e5
fix(ingest): include database info for snowflake ( #2426 )
2021-04-20 20:40:30 -07:00
Brian
95b96572c5
feat(react): schema visualization add support for nested structs ( #2423 )
2021-04-20 14:00:10 -07:00
Thomas Larsson
7869a8f142
feature(ingestion): Adding the concept of transformers ( #2411 )
...
Fixes : #2410
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-18 11:15:05 -07:00
Harshal Sheth
27a672e697
feat(ingest): add Kafka-based emitter example ( #2412 )
2021-04-16 17:54:50 -07:00
Harshal Sheth
9ac17c4ee0
fix(ingest): bump avro-gen3 ( #2403 )
...
Closes #2375 .
2021-04-16 11:59:05 -07:00
Harshal Sheth
91a2f69310
fix(ingest): properly handle fieldDiscriminator with restli ( #2408 )
2021-04-16 09:42:52 -07:00
Harshal Sheth
777c05973f
fix(ingest): add sqlalchemy extra ( #2409 )
2021-04-16 09:41:23 -07:00
Thomas Larsson
89fb538fa5
feature(ingestion): Make origin/fabric_type configurable ( #2405 )
...
Fixes : #2394
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-15 10:47:28 -07:00
Gabe Lyons
1e11851a90
feat(sample): adding sample mces for dataflows and datajobs ( #2396 )
2021-04-14 19:27:01 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods ( #2400 )
2021-04-14 19:25:57 -07:00
Harshal Sheth
9c2a30c3a1
fix(ingest): add db name to postgres URNs ( #2401 )
2021-04-14 15:04:13 -07:00
Harshal Sheth
2af4603e49
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators ( #2393 )
2021-04-14 13:40:24 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test ( #2392 )
2021-04-13 17:30:24 -07:00
Harshal Sheth
fb6f74b1da
feat(ingest): add generic sqlalchemy source ( #2389 )
2021-04-13 08:01:38 -07:00
Harshal Sheth
eeee8aa34e
fix(ingest): report correct version status in dev mode ( #2388 )
2021-04-12 19:35:40 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend ( #2368 )
2021-04-12 17:40:15 -07:00
Junqiu Zheng
cadcdd936c
fix(docs): fix config example for file sink ( #2380 )
2021-04-12 11:37:17 -07:00
