362 Commits

Author SHA1 Message Date
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift (#4344) 2022-03-14 08:50:29 -07:00
mayurinehate
e19b12030e
fix(GE): fix dependencies for GE DataHubValidationAction, logic for s… (#4347) 2022-03-08 11:20:09 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA (#4305) 2022-03-04 11:51:31 -08:00
Tamas Nemeth
3158e49171
fix(ingest): Removing python 3.9.9 restriction (#4312) 2022-03-04 08:21:33 -08:00
Aseem Bansal
d52638a252
fix(ingest): lib - revert positional arg change, create new method (#4266)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-01 22:06:37 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source (#4201) 2022-02-28 17:37:22 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling (#4131) 2022-02-24 23:26:06 -08:00
Ravindra Lanka
1ac20ee764
feat(platform): add timeline api for dataset entity (#4252) 2022-02-24 19:40:55 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file (#4116) 2022-02-24 17:02:38 -08:00
Kevin Hu
46701319dc
feat(ingest): switch telemetry endpoint to Mixpanel (#4238) 2022-02-24 12:35:48 -08:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 (#4205) 2022-02-22 22:13:50 -08:00
Swaroop Jagadish
c6cb549918
fix(ingest): superset - adding missing greenlet dep (#4203) 2022-02-21 09:07:44 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support (#4057)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing (#4142)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
Sunil Patil
be64092c89
fix(ingest): elasticsearch - pinning down elasticsearch dependency to less than 8.0.0 (#4182)
Co-authored-by: Sunil Patil <spatil@twilio.com>
2022-02-19 11:44:32 -08:00
Swaroop Jagadish
eaf7b02b2a
docs(model): auto-generated docs and hand-written docs for the metadata model (#4189) 2022-02-18 09:45:45 -08:00
Tamas Nemeth
8cb312c4be
fix(ci): pinning markup_safe (#4188) 2022-02-18 08:53:28 -08:00
Harshal Sheth
75d2ec2a39
ci(ingestion): fix airflow 1 deps for tox (#4083) 2022-02-17 00:33:28 -08:00
Harshal Sheth
6f37ddfee2
chore(ingestion): upgrade mypy (#4141) 2022-02-17 00:31:20 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets (#4123)
* Getting table properties for Athena datasets

* Isorting

* Fixing mypy error

* Addressing pr review comments
Adding tests

* Adding missing import

* black

* Fixing test run

* fixing flake8

* Adding athen to tox tests as well

* Not running athena tests on python < 3.7

* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging (#4077) 2022-02-10 20:02:23 -08:00
Dexter Lee
47db7b5e2c
fix(ingest): dependencies - fix for redshift-usage, mode, superset and others (#4103) 2022-02-09 23:12:41 -08:00
Ravindra Lanka
917720989a
fix(ingest): dependencies - Add psutil dependency for stateful ingestion reporting. (#4099) 2022-02-08 23:25:12 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] (#4063) 2022-02-08 14:26:44 -08:00
Kevin Hu
1a2f75b1e6
fix(ingest): data-lake - add aws dependencies (#4060) 2022-02-04 21:12:09 -08:00
Swaroop Jagadish
aab622793d
feat(ci): pin tox requirements to speed up ci runs, remove airflow-1 suite until we can pin it (#4055) 2022-02-04 07:32:38 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance (#4047) 2022-02-02 22:52:50 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting (#3807) 2022-02-02 13:19:15 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 (#3888) 2022-01-31 09:22:55 -08:00
Aseem Bansal
7180ae5ff6
fix(ci): fix linting lib version requirement (#3939) 2022-01-30 17:20:46 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms (#4009) 2022-01-30 11:47:53 -08:00
Tamas Nemeth
cec541d827
fix(ci): ingestion - Restrict python to <=3.9.9 due to upstream looker sdk issues (#3961)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-25 19:11:53 -08:00
Ravindra Lanka
7eda2bf144
fix(ingest): snowflake_usage - emit usage workunits when include_operational_stats is enabled. (#3949) 2022-01-21 17:16:03 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets (#3656) 2022-01-17 15:01:23 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source (#3893)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
varunbharill
cc52e3e759
fix(ingest): bigquery - update dependencies for module. (#3874)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-13 22:12:11 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources (#3845) 2022-01-07 17:27:09 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib (#3806) 2022-01-05 12:22:59 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression (#3757) 2021-12-21 12:13:13 -08:00
Aseem Bansal
77e3641a50
fix(ingest): fix compatibility with google composer (#3774) 2021-12-20 12:47:56 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Harshal Sheth
c6f3ddf077
build(ingest): restrict latest mypy version (#3756) 2021-12-16 08:48:15 -08:00
Sergio Gómez Villamor
c59c63e90d
feat: enables dbt metadata files to be loaded from URIs (#3739) 2021-12-15 09:11:39 -08:00
jawadqu
578590e795
feat(ingestion) : Add Metabase Source Connector (#3602)
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-13 23:02:47 -08:00
Gabe Lyons
8394fc62b0
feat(mode): add mode analytics ingestion source (#3710) 2021-12-09 16:10:08 -08:00
Gabe Lyons
0fdd3352bd
feat(ingestion): Add lineage support for Redshift source (#3697)
Co-authored-by: treff7es <treff7es@gmail.com>
2021-12-08 23:41:18 -08:00
mayurinehate
1d7ec8dba8
feat(ingest): add nifi source (#3681) 2021-12-08 14:56:31 -08:00
Swaroop Jagadish
a16c432a1b
feat(metadata-model): adding metadata model doc generation and upload… (#3667) 2021-12-05 12:22:17 -08:00
Harshal Sheth
7d5da17d88
fix(ingest): workaround great-expectations compatibility issue (#3634) 2021-11-29 17:14:19 -08:00
Swaroop Jagadish
2fd919cb4d
fix(ci): pin great expectations version (#3629) 2021-11-28 17:28:57 -08:00