3975 Commits

Author SHA1 Message Date
kr_Deepankar
94ce753bb0
fix(ingest/kafka): use SchemaReference properties instead of dict access (#8615)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-16 10:33:09 +05:30
skrydal
baae3d261d
fix(ingest/okta): fix event_loop RuntimeError with nested asyncio (#8637) 2023-08-16 10:32:57 +05:30
Andrew Sikowitz
526e626146
feat(ingest): Add DataHub source (#8561)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-15 17:49:20 -04:00
Tamas Nemeth
ec693d744a
feat(ingest/s3): Adding option to pass in any spark config property to s3 source (#8621) 2023-08-15 14:43:25 +02:00
Harshal Sheth
9eb432cda0
fix(ingest/airflow): make inlets work again (#8631) 2023-08-15 14:42:43 +02:00
Mayuri Nehate
ddcd5109dc
feat(ingest): allow relative start time config (#8562)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-14 17:48:06 -07:00
mohdsiddique
b3251577e1
docs(ingsetion/csv-enricher): fix sample csv mentioned in Docstrings (#8432)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-14 10:18:54 -07:00
alplatonov
11fdfcf956
Fix(ingestion/clickhouse) move to two tier sqlalchemy (#8300)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-11 16:11:40 -04:00
Mayuri Nehate
7bb4e7b90d
docs(ingest): update s3 and gcs doc with concept mapping (#8575) 2023-08-11 11:01:15 -07:00
Meenakshi Kamalaseshan Radha
93cc10ebfc
fix(ingest): Add client_certificate_path for rest client cert instead of ca_certif… (#8581)
Co-authored-by: Kamalaseshan Radha <mkamalas@LAMU02DN212MD6R.uhc.com>
2023-08-10 15:43:06 +05:30
alplatonov
b58f9bb396
Feat(ingest/ldap)fix list index out of range error (#8525)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-09 13:13:27 -04:00
Harshal Sheth
ebf42e2702
fix(ingest): use hive pure_sasl variant (#8570) 2023-08-09 13:04:36 -04:00
Harshal Sheth
fd9121737d
fix(ingest/file): remove entity_type_counts and aspect_counts (#8586) 2023-08-09 13:01:12 -04:00
Aseem Bansal
2b0952195b
doc(ingest/log): failure log at pipeline level document (#8591) 2023-08-09 10:18:14 +05:30
Ellie O'Neil
4b3e566be8
docs(lineage): Add description to make_lineage_mce (#8596) 2023-08-08 16:06:47 -07:00
Mayuri Nehate
b4e104f190
fix(ingest/snowflake): maintain qualified name casing, do not lowercase (#8574) 2023-08-04 10:43:22 -07:00
Harshal Sheth
ef15861d0d
fix(ingest): handle redaction of configs with int keys (#8545) 2023-08-04 13:02:04 -04:00
Aseem Bansal
dac89fb1fb
feat(ingest): allow lower freq profiling based on date of month/day of week (#8489) 2023-08-04 10:13:48 +05:30
mohdsiddique
6a36118b4f
feat(ingestion/snowflake): use user email-id in urn generation for top users stat (#8513)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-03 08:30:50 +05:30
mohdsiddique
05ef7db45e
fix(ingetion/mssql): convert dataset urns to lowercase (#8551)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-02 13:44:05 -07:00
Harshal Sheth
3428bcaaad
fix(ingest): add tableau sqlglot dep (#8552) 2023-08-02 15:18:06 -03:00
Aseem Bansal
d6263a6c0c
fix(ingest/s3): catch no such bucket exception instead of failing (#8549) 2023-08-02 19:21:39 +05:30
TusharM
e2919cfd06
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector (#8298) 2023-08-02 15:09:50 +05:30
Harshal Sheth
08a39c4b0d
fix(ingest/presto): fix presto on hive test failures (#8548) 2023-08-02 12:46:39 +05:30
Aseem Bansal
bb33f015ca
fix(ingest/s3): wrong sorting in case of multi-partition key (#8536) 2023-08-02 09:54:33 +05:30
Felix Lüdin
ea95e36c2b
fix(ingest): fix data platform urn in dataset_urn_to_key and dataset_key_to_urn (#8209) 2023-08-02 09:54:21 +05:30
Jarod Smilkstein
f51bd01a70
feat(ingest): add ability to read other method types than GET for OAS ingest recipes (#8303) 2023-08-02 09:54:09 +05:30
Kirill Popov
eec89a884a
feat(ingest): Add metabase database id to platform instance mapping (#8359) 2023-08-02 09:53:48 +05:30
Mayuri Nehate
e67f811034
feat(classification): allow parallelisation to reduce time (#8368) 2023-08-02 09:53:39 +05:30
zheyu001
bf47d65412
feat(ingest/presto-on-hive): enable partition key for presto-on-hive (#8380) 2023-08-02 09:53:27 +05:30
Gabe Lyons
843f82b943
feat(presto-on-hive): allow v1 fieldpaths in the presto-on-hive source (#8474) 2023-08-01 14:05:50 -07:00
Pedro Silva
a4a8182001
feat(cli): Adds ability to upload recipes to DataHub's UI (#8317)
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
2023-08-01 17:35:42 -03:00
Benjamin Dornel
2e2a6748ac
fix(ingest/json-schema): convert non-string enums to strings (#8479) 2023-08-01 19:35:40 +05:30
mohdsiddique
547e1f498e
feat(ingestion/tableau): support column level lineage for custom sql (#8466)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-01 19:35:18 +05:30
Harshal Sheth
66074341f7
test(ingest): test case statements with sql parser (#8437) 2023-08-01 19:34:48 +05:30
VISHAL KUMAR
ef3b9489aa
feat(ingest/vertica): performance improvement and bug fixes (#8328)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-01 19:34:35 +05:30
Aseem Bansal
09ef6e080c
fix(ingest): remove duplication of tags (#8532) 2023-08-01 19:31:26 +05:30
Harshal Sheth
d8b2397b93
fix(ingest): pin boto3-stubs in CI (#8527)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-31 19:48:05 -07:00
Tamas Nemeth
1a47a51f1b
fix(ingest/build): Fix sagemaker mypy and flake8 issues (#8530) 2023-07-31 16:13:07 +02:00
Harshal Sheth
89f23d3c36
chore(ingest): bump sqllineage and sqlparse (#8481) 2023-07-28 13:10:19 -07:00
Harshal Sheth
9718505fc7
fix(ingest): respect max_threads for ingestion reporter (#8521) 2023-07-28 13:09:32 -07:00
Harshal Sheth
99f1624ce7
docs(ingest/lookml): clarify connection map config (#8508) 2023-07-27 17:06:04 +05:30
Mayuri Nehate
b9060db515
chore(ingest/tableau): miscellaneous cleanup refractor (#8417)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-07-27 17:02:48 +05:30
Mayuri Nehate
5a3f91de53
docs(ingest/bigquery): add permissions to profile google drive backed… (#8490) 2023-07-27 17:01:39 +05:30
Harshal Sheth
2495b50f8c
ci: fix commandLine usage in build.gradle (#8510) 2023-07-26 11:12:22 +05:30
Harshal Sheth
c585a1bcc7
fix(ingest): hint at --update-golden-files option when tests fail (#8507) 2023-07-26 09:50:27 +05:30
Harshal Sheth
b12de099aa
docs: add docs on sqlglot lineage (#8482) 2023-07-25 10:48:37 -07:00
Harshal Sheth
eac003ccf4
feat(ingest): enable pipeline reporting by default (#8472) 2023-07-25 14:16:27 +05:30
Aseem Bansal
cc46729137
chore(ingest): add example of training metric/hyper parameters (#8491) 2023-07-25 13:53:16 +05:30
Andrew Sikowitz
bf9f380350
fix(ingest): Generate browse paths v2 for more sources; properly pass platform_instance (#8501) 2023-07-25 11:35:34 +05:30