780 Commits

Author SHA1 Message Date
Mayuri Nehate
7b88de89d5
fix(ingest): snowflake - allow profiling to work with geography type (#6162) 2022-10-10 08:05:09 -07:00
Shirshanka Das
e9c4c823d8
fix(ingest): bigquery-beta - ensure that status aspect is emitted for… (#6154)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-10-08 16:00:45 -07:00
Harshal Sheth
30014072ad
feat(ingest): lookml - add support for local/remote dependencies (#6150) 2022-10-07 13:14:21 -07:00
Mayuri Nehate
b8513ae492
fix(ingest): fetch workbook tags in workbooks graphql query (#6102) 2022-10-07 15:04:03 +02:00
Tamas Nemeth
2f79b50c24
fix(ingest): presto-on-hive - not failing on Hive type parsing error (#6118)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-04 20:54:38 -07:00
Ravindra Lanka
055e4082da
fix(ingestion): fix percent change computation in stale_entity_removal (#6121) 2022-10-04 20:40:59 -07:00
Harshal Sheth
feb9d59b4d
fix(ingest): remove unused mysql golden file (#6106) 2022-10-04 20:26:51 -07:00
Harshal Sheth
3c0f63c50a
fix(ingest): hide deprecated path_spec option from config (#5944) 2022-10-04 12:14:00 -07:00
Tamas Nemeth
a6930562fd
fix(ingest): presto-on-hive - Setting display name properly (#6065) 2022-09-29 16:20:17 +02:00
Shirshanka Das
74d9fa25a6
feat(ingest): looker - add support for simple column level lineage (#6084) 2022-09-28 20:41:30 -07:00
Tamas Nemeth
3b9e9793a7
fix(ingest): bigquery-beta - handling complex types properly (#6062) 2022-09-27 21:31:24 +02:00
Harshal Sheth
59a2228d13
fix(ingest): correct tableau browse paths (#6064) 2022-09-27 12:30:52 -07:00
Harshal Sheth
3f1d47c069
feat(ingest): list referenced env variables in recipe (#6043) 2022-09-26 23:16:18 -07:00
mohdsiddique
c062bf2201
fix(ingestion): Tableau test case execution fix (#6005) 2022-09-26 18:14:42 -07:00
Alexey Kravtsov
3c3ab64954
feat(ingest): implement compression for CheckpointState (#6007) 2022-09-26 10:18:42 -07:00
John Joyce
78454888b5
refactor(ingest): Browse Paths Upgrade Tableau (#6008) 2022-09-23 13:02:34 -07:00
Harshal Sheth
27f28019de
refactor(ingest): move common host_port validation (#6009) 2022-09-22 16:32:07 -07:00
Ravindra Lanka
b8941ab190
feat(ingestion): Add fail-safe stale entity removal via configurable 'fail_safe_threshold' param. (#6027) 2022-09-22 16:09:22 -07:00
Harshal Sheth
68db859ca1
refactor(ingest): streamline two-tier db config validation (#5986) 2022-09-21 10:45:37 -07:00
mohdsiddique
44a67b52cf
feat(ingestion): Tableau dashboard & chart usage stats (#5999)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-20 17:41:56 -07:00
Mayuri Nehate
b195b6c123
fix(ingest): encode reserved characters when creating dataset urn (#5977)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-20 16:59:02 -07:00
Tamas Nemeth
46494bc0ad
feat(ingest): presto-on-hive - Description and hive view support (#5993)
* Presto on Hive Improvements like:
 - Capturing table/column descriptions
 - Adding Hive view support
 - Various bug fixes
2022-09-21 00:40:10 +02:00
Aezo
6c865d00c4
feat(ingest): stateful, catalog and pascalcase con (#5890) 2022-09-20 14:54:22 +02:00
Harshal Sheth
8ffa7054e5
feat(ingest): dbt column-level meta mappings + add_terms operation (#5970)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-19 15:26:12 -07:00
John Joyce
df3633e7f3
refactor(looker): Migrating to new browse path format (#5688) 2022-09-19 14:43:35 -07:00
Harshal Sheth
c8cd7d5040
refactor(ingest): use typed objects in dbt ingestion (#5929) 2022-09-16 15:07:59 -07:00
Mayuri Nehate
a14617b6a4
fix(ingest): continue validation of s3 path_specs even if platform is set (#5951) 2022-09-16 12:03:57 -07:00
Harshal Sheth
937ab192c0
feat(ingest): add support for aliases in plugin registry (#5958) 2022-09-16 07:19:32 -07:00
skrydal
a026c84691
feat: qualifiedName support + populating glue ARN (#5952) 2022-09-15 21:15:03 -07:00
Mayuri Nehate
2558129391
refactor(snowflake): move snowflake-beta to certified snowflake source (#5923)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-15 09:53:54 -07:00
Mugdha Hardikar
93681d7a71
feat(ingest): delta-lake - support delta tables from minio (#5758)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-15 09:51:14 -07:00
Harshal Sheth
9384a52b23
feat(ingest): skip ssh known_hosts verification for git clone (#5945) 2022-09-14 16:43:21 -07:00
skrydal
f61a040555
feat(ingestion) Add more info to glue entities (#5874)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-14 12:25:09 -07:00
Harshal Sheth
e23523a781
fix(ingest): fix type annotations on some pydantic fields (#5795) 2022-09-14 11:05:31 -07:00
Harshal Sheth
a1e1d2fd0a
feat(ingest): add ConfigEnum type (#5734) 2022-09-14 09:57:42 -07:00
Ravindra Lanka
ee68f09624
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. (#5766) 2022-09-14 09:30:42 -07:00
Shirshanka Das
15a33fadc8
feat(ingest): lookml - support for git checkout (#5924) 2022-09-13 16:40:59 -07:00
Mayuri Nehate
aedf1522fb
feat(ingest): snowflake-beta - minor changes, tests (#5910) 2022-09-12 10:42:52 -07:00
Harshal Sheth
e556bcb306
feat(ingest): add entity type inference to mcpw (#5880) 2022-09-10 20:36:10 -07:00
Harshal Sheth
220ae0b6c9
feat(ingest): make sink use type annotations (#5899) 2022-09-10 19:46:20 -07:00
Harshal Sheth
3cd8ea8316
test(ingest): use pytest parameterization for dbt integration tests (#5879) 2022-09-10 18:35:58 -07:00
Harshal Sheth
954397a078
fix(ingest): remove dbt disable_dbt_node_creation and load_schema options (#5877) 2022-09-09 14:07:55 -07:00
Marcin Szymański
57996049fb
fix(ingest): sql - add missing trino types (#5870)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-09 14:07:00 -07:00
Shirshanka Das
056add128d
fix(ingest): datahub-api - move instantiation to the right config class (#5878) 2022-09-09 13:34:21 -07:00
Harshal Sheth
6063484714
fix(ingest): avrogen handling for missing fields with default values (#5844) 2022-09-08 14:05:28 -07:00
Harshal Sheth
2115d5bf1d
fix(ingest): remove dbt delete_tests_as_datasets option (#5865)
This option was added for backwards compat and can be dropped now.
2022-09-08 11:22:25 -07:00
Harshal Sheth
08622f25ef
feat(ingest): add utility for converting MCEs to MCPs (#5812) 2022-09-06 15:25:48 -07:00
mohdsiddique
2f65e2f226
feat(transformers): Add semantics & transform_aspect support in transformers (#5514)
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-06 14:44:14 -07:00
Mayuri Nehate
a8c1397e5e
feat(ingest): tableau - improve lineage, workbooks query, fix pagination (#5756) 2022-09-05 23:19:22 -07:00
Shirshanka Das
9afda47085
feat(cli): add support for sampled reporting to keep logs manageable (#5800) 2022-09-01 14:47:28 -07:00