3089 Commits

Author SHA1 Message Date
Mayuri Nehate
b195b6c123
fix(ingest): encode reserved characters when creating dataset urn (#5977)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-20 16:59:02 -07:00
Tamas Nemeth
46494bc0ad
feat(ingest): presto-on-hive - Description and hive view support (#5993)
* Presto on Hive Improvements like:
 - Capturing table/column descriptions
 - Adding Hive view support
 - Various bug fixes
2022-09-21 00:40:10 +02:00
Shirshanka Das
8e8fb51ce3
feat(cli): extend put command to support platform creation (#5990) 2022-09-20 11:52:22 -07:00
Aezo
6c865d00c4
feat(ingest): stateful, catalog and pascalcase con (#5890) 2022-09-20 14:54:22 +02:00
Harshal Sheth
8ffa7054e5
feat(ingest): dbt column-level meta mappings + add_terms operation (#5970)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-19 15:26:12 -07:00
John Joyce
df3633e7f3
refactor(looker): Migrating to new browse path format (#5688) 2022-09-19 14:43:35 -07:00
Harshal Sheth
98ee621849
fix(ingest): handle uname missing on windows (#5981) 2022-09-19 10:36:16 -07:00
Tamas Nemeth
663a659f66
feat(ingest): bigquery-v2 - Add full profiling capability (#5953) 2022-09-17 09:46:45 +02:00
Harshal Sheth
c8cd7d5040
refactor(ingest): use typed objects in dbt ingestion (#5929) 2022-09-16 15:07:59 -07:00
Mayuri Nehate
a14617b6a4
fix(ingest): continue validation of s3 path_specs even if platform is set (#5951) 2022-09-16 12:03:57 -07:00
Harshal Sheth
937ab192c0
feat(ingest): add support for aliases in plugin registry (#5958) 2022-09-16 07:19:32 -07:00
skrydal
a026c84691
feat: qualifiedName support + populating glue ARN (#5952) 2022-09-15 21:15:03 -07:00
Mayuri Nehate
2558129391
refactor(snowflake): move snowflake-beta to certified snowflake source (#5923)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-15 09:53:54 -07:00
Mugdha Hardikar
93681d7a71
feat(ingest): delta-lake - support delta tables from minio (#5758)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-15 09:51:14 -07:00
Harshal Sheth
66985335d0
fix(ingest): add trailing newline to ssh keys (#5947) 2022-09-14 20:53:26 -07:00
Harshal Sheth
9384a52b23
feat(ingest): skip ssh known_hosts verification for git clone (#5945) 2022-09-14 16:43:21 -07:00
Ravindra Lanka
cf2f59f85f
fix(ingest): Fix snowflake usage stateful ingestion backward compatibility check. (#5943) 2022-09-14 13:25:50 -07:00
skrydal
f61a040555
feat(ingestion) Add more info to glue entities (#5874)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-14 12:25:09 -07:00
Harshal Sheth
e23523a781
fix(ingest): fix type annotations on some pydantic fields (#5795) 2022-09-14 11:05:31 -07:00
Mayuri Nehate
8ead62dae5
fix(transformer): fix invalid lastModified.actor entry in transformers (#5906) 2022-09-14 09:59:17 -07:00
Harshal Sheth
a1e1d2fd0a
feat(ingest): add ConfigEnum type (#5734) 2022-09-14 09:57:42 -07:00
Ravindra Lanka
ee68f09624
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. (#5766) 2022-09-14 09:30:42 -07:00
Harshal Sheth
8f5b773025
fix(airflow): prioritize conn_type over id-based type inference (#5911) 2022-09-14 18:30:03 +02:00
firas omrane
c606abdb40
fix(docker) Add platform to docker-compose command (#5683) 2022-09-14 17:56:42 +05:30
Harshal Sheth
68ea926770
fix(ingest): add dbt redshift type mappings (#5933) 2022-09-14 14:46:21 +05:30
Harshal Sheth
e3f8f29a31
feat(ingest): support version option in datahub client get_aspect_v2 (#5934) 2022-09-14 14:45:01 +05:30
Harshal Sheth
a8ac5f9e33
refactor(ingest): simplify DefaultSQLParser alias (#5935) 2022-09-14 14:44:22 +05:30
Shirshanka Das
15a33fadc8
feat(ingest): lookml - support for git checkout (#5924) 2022-09-13 16:40:59 -07:00
Harshal Sheth
dfeced8eee
fix(ingest): hide internal profiler.allow_deny_patterns from config (#5619) 2022-09-13 16:09:10 +05:30
Harshal Sheth
45979edd12
fix(ingest): handle lookml directory being in a symlink'd folder (#5916) 2022-09-13 13:42:44 +05:30
Harshal Sheth
b6d3fe88e5
fix(ingest): lookml - support relative paths in base_folder (#5914) 2022-09-12 13:59:48 -07:00
Harry Nguyen
46bc1fe367
feat(ingestion): schema inference for jsonlines in S3 (#5725)
Co-authored-by: Nguyen Tri Hieu <nhieu@fossil.com>
2022-09-12 10:49:46 -07:00
Mayuri Nehate
aedf1522fb
feat(ingest): snowflake-beta - minor changes, tests (#5910) 2022-09-12 10:42:52 -07:00
Ravindra Lanka
386719f15a
feat(platform): timeseries - Server & Client side changes to support timeseries aspect deletion & rollback. (#4756) 2022-09-11 11:27:46 -07:00
Harshal Sheth
e556bcb306
feat(ingest): add entity type inference to mcpw (#5880) 2022-09-10 20:36:10 -07:00
Harshal Sheth
220ae0b6c9
feat(ingest): make sink use type annotations (#5899) 2022-09-10 19:46:20 -07:00
Harshal Sheth
8f72de5d8b
refactor(ingest): move aspect maps to dedicated file (#5821) 2022-09-10 18:43:11 -07:00
Harshal Sheth
954397a078
fix(ingest): remove dbt disable_dbt_node_creation and load_schema options (#5877) 2022-09-09 14:07:55 -07:00
Marcin Szymański
57996049fb
fix(ingest): sql - add missing trino types (#5870)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-09-09 14:07:00 -07:00
Shirshanka Das
5074035c4d
feat(ingest): looker - improve defaults for usage extraction (#5893)
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-09-09 13:35:49 -07:00
Shirshanka Das
056add128d
fix(ingest): datahub-api - move instantiation to the right config class (#5878) 2022-09-09 13:34:21 -07:00
Pedro Silva
e7b33c595b
feat(cli): make docker compose quiet (#5869) 2022-09-08 14:55:03 -07:00
Harshal Sheth
da46dcdc22
refactor(ingest): add ALL_ENV_TYPES constant (#5866) 2022-09-08 14:07:22 -07:00
Harshal Sheth
2115d5bf1d
fix(ingest): remove dbt delete_tests_as_datasets option (#5865)
This option was added for backwards compat and can be dropped now.
2022-09-08 11:22:25 -07:00
Harshal Sheth
39fa9321c1
feat(ingest): support reading config file from stdin (#5847) 2022-09-08 11:04:20 -07:00
Harshal Sheth
eb57bebf48
fix(ingest): include urns in rest sink failure logs (#5848) 2022-09-08 10:29:40 -07:00
Shirshanka Das
d911b15da5
docs(ingest): improve doc gen, docs for snowflake, looker (#5867) 2022-09-08 00:38:10 -07:00
Harshal Sheth
1e8a1038d8
chore(ingest): remove outdated GE compatibility hack (#5862) 2022-09-07 17:06:02 -07:00
Tamas Nemeth
f9917ea5fb
Fixing sharded regexp pattern config (#5861) 2022-09-07 15:36:37 -07:00
Shirshanka Das
2fea3d26df
fix(ingest): reporting should work with timestamps (#5860) 2022-09-07 21:09:08 +02:00