3975 Commits

Author SHA1 Message Date
Andrew Sikowitz
301d3e6b1c
test(ingest/unity): Add Unity Catalog memory performance testing (#8932) 2023-10-04 10:23:13 -04:00
Andrew Sikowitz
d3346a04e4
feat(ingest/unity): Ingest notebooks and their lineage (#8940) 2023-10-04 10:22:45 -04:00
Upendra Rao Vedullapalli
13508a9d88
feat(bigquery): excluding projects without any datasets from ingestion (#8535)
Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-10-04 09:20:51 -04:00
Mayuri Nehate
e3780c2d75
feat(ingest/snowflake): initialize schema resolver from datahub for l… (#8903) 2023-10-04 16:23:31 +05:30
Harshal Sheth
a300b39f15
feat(ingest/airflow): airflow plugin v2 (#8853) 2023-10-04 16:23:15 +05:30
siddiquebagwan-gslab
c415d63dda
feat(ingestion/powerbi): column level lineage extraction for M-Query (#8796) 2023-10-04 16:22:51 +05:30
Hyejin Yoon
52156193b6
docs(lineage): Lineage docs refactoring (#8899) 2023-10-04 08:43:59 +00:00
Aseem Bansal
ad313ad282
feat(transfomer): add transformer to get ownership from tags (#8748) 2023-10-04 14:06:03 +05:30
Harshal Sheth
419b8a7cc2
fix(cli): make quickstart docker compose up command more robust (#8929) 2023-10-03 20:20:32 -07:00
Harshal Sheth
9deb7be3fc
fix(ingest): refactor test markers + fix disk space issues in CI (#8938) 2023-10-03 20:17:49 -07:00
Ellie O'Neil
0a5e7d176e
fix(client): use value for RelationshipDirection (#8912) 2023-10-03 11:53:05 -07:00
Pedro Silva
790011d40b
feat(docs): Corrects release version for custom ownership types. (#8847) 2023-10-02 20:04:55 -07:00
Andrew Sikowitz
833daa8efd
refactor(ingest/graph): Factor out filter logic (#8888) 2023-09-28 13:52:50 -04:00
Andrew Sikowitz
e738e16157
refactor(ingest/bigquery): Clarify table / view queries (#8913) 2023-09-28 13:52:35 -04:00
Andrew Sikowitz
2e2cd87d99
test(ingest/delta-lake): Fix minio test for new version of delta-lake (#8914) 2023-09-27 17:00:02 -04:00
Andrew Sikowitz
8396829dbc
build(ingest/databricks): Relax databricks-sdk pin (#8855) 2023-09-27 14:03:34 -04:00
hariishaa
622816dcb8
feat(metadata-ingestion): implement mlflow source (#7971)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-26 13:51:30 -04:00
Harshal Sheth
dc9141a6f4
fix(ingest): bump typing-extensions (#8897) 2023-09-26 10:34:36 -07:00
Tamas Nemeth
0a869dd6f8
fix(ingest/s3): Converting windows style path to posix one on local fs (#8757) 2023-09-26 10:28:03 +02:00
Harshal Sheth
53eaac5963
fix(ingest): bound types-requests (#8895) 2023-09-25 13:24:19 -07:00
Mayuri Nehate
874109f76e
feat(ingest/snowflake): allow shares config without platform instance (#8803) 2023-09-25 14:04:05 +05:30
Shubham Jagtap
501522d891
feat(ingest/kafka-connect): Lineage for Kafka Connect > Snowflake (#8811)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:12:48 -07:00
Mayuri Nehate
5c40390a92
feat(ingest/kafka): support metadata mapping from kafka avro schemas (#8825)
Co-authored-by: Daniel Messias <danielcmessias@gmail.com>
Co-authored-by: Deepankarkr <deepankar.kumar@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:11:42 -07:00
Harshal Sheth
5bb9f30895
docs(ingest/lookml): add guide on debugging lkml parse errors (#8890) 2023-09-22 16:55:15 -07:00
Sergio Gómez Villamor
e254a50b50
fix(report): too long report causes MSG_SIZE_TOO_LARGE in kafka (#8857) 2023-09-22 16:54:34 -07:00
Harshal Sheth
791e2e7bf5
feat(python): support custom models without forking (#8774) 2023-09-22 16:43:58 -07:00
Harshal Sheth
c946c01199
fix(ingest/bigquery): show report in output (#8867)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-22 13:01:38 -07:00
Mayuri Nehate
5481e19e0a
feat(ingest): bulk fetch schema info for schema resolver (#8865)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-21 18:05:26 -04:00
Harshal Sheth
2a0200b047
feat(ingest): bump acryl-sqlglot (#8882) 2023-09-21 14:28:51 -07:00
Mayuri Nehate
6ce35c9654
fix(ingest): fix mode lint error (#8875) 2023-09-21 09:05:58 -07:00
Harshal Sheth
6c6216aaa2
fix(airflow): fix provider loading exception (#8861) 2023-09-20 12:00:23 -07:00
Gabe Lyons
67af68284f
dcs(ml-models): enhancing ml model documentation (#8848) 2023-09-19 09:02:24 -07:00
Mayuri Nehate
99d7eb756c
feat(ingest/bigquery): support bigquery profiling with sampling (#8794) 2023-09-15 13:36:04 -07:00
Tony Ouyang
f4da93988e
feat(ingestion/dynamodb): Add DynamoDB as new metadata ingestion source (#8768)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-09-15 13:26:17 -07:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call (#8626) 2023-09-15 11:55:39 -07:00
Andrew Sikowitz
e75900b9a9
build(ingest): Remove constraint on jsonschema for Python >= 3.8 (#8842) 2023-09-14 12:25:41 -07:00
Hyejin Yoon
31abf383d1
ci: add markdown-link-check (#8771) 2023-09-14 11:34:21 +09:00
Andrew Sikowitz
493d31531a
feat(ingest/rest-emitter): Do not raise error on retry failure to get better error messages (#8837) 2023-09-13 14:00:58 -07:00
Andrew Sikowitz
1474ac01b1
build(ingest): Bump jsonschema for Python >= 3.8 (#8836) 2023-09-13 12:32:45 -07:00
Adriano Vega Llobell
3cc0f76d17
docs(transformer): fix names in sample code of 'pattern_add_dataset_domain' (#8755) 2023-09-12 14:34:24 -07:00
Pedro Silva
138f6c0f74
feat(cli): fix upload ingest cli endpoint (#8826) 2023-09-12 14:26:30 -07:00
Harshal Sheth
449cc9ba91
ci: make wheel builds more robust (#8815) 2023-09-12 13:15:05 -07:00
Harshal Sheth
f7fee743bf
fix(ingest): use epoch 1 for dev build versions (#8824) 2023-09-12 13:11:01 -07:00
Mayuri Nehate
303a2d0863
build(ingest): upgrade to sqlalchemy 1.4, drop 1.3 support (#8810)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-12 11:30:24 -07:00
cjm98332
a021053a72
fix(ingest/mssql): Add UNIQUEIDENTIFIER data type as String (#8642)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-12 19:23:39 +05:30
siddiquebagwan-gslab
95b2d437ca
feat(ingestion/looker): Add view file-path as option in view_naming_pattern config (#8713) 2023-09-11 16:55:17 +05:30
Harshal Sheth
eb4107a6e3
fix(ingest): drop wrap_aspect_as_workunit method (#8766) 2023-09-07 11:32:41 -07:00
Harshal Sheth
0e8000cf18
feat(ingest): drop sql_metadata parser (#8765) 2023-09-07 11:32:28 -07:00
Harshal Sheth
4ffad4d9b9
chore(ingest): upgrade sqlglot fork (#8775)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-09-06 12:49:44 -07:00
Mayuri Nehate
e680a97046
fix(ingest/bigquery): fix partition and median queries for profiling (#8778) 2023-09-06 12:48:11 -07:00