60 Commits

Author SHA1 Message Date
Harshal Sheth
3d5735cbc5
chore(ingest): run pyupgrade for python 3.8 (#10513) 2024-05-15 22:31:05 -07:00
Harshal Sheth
a4a556a512
test(ingest/mssql): use non-ephemeral mapping port (#10104) 2024-03-22 08:40:33 +01:00
Harshal Sheth
1498c36875
chore(cli): drop support for python 3.7 (#9731) 2024-01-29 10:50:47 -08:00
sleeperdeep
720296ea46
fix(ingestion/mssql): use platform_instance for mssql dataflow/datajob (#9527)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: sleeperdeep <--global>
2024-01-22 20:25:01 -08:00
Tamas Nemeth
03590a1948
fix(ingest/snowflake) - Fixing snowflake url with default region (#9443) 2023-12-18 13:54:31 -05:00
Mayuri Nehate
08fb730676
fix(ingest): drop deprecated database_alias from sql sources (#9299)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-28 15:49:49 -05:00
terratrue-daniel
ff9876f5a2
fix(ingest/mssql): Add MONEY and SMALLMONEY data types as Number (#9313) 2023-11-28 14:43:52 -05:00
Harshal Sheth
73514ad9c5
fix(ingest): cleanup large images in CI (#9153) 2023-10-31 21:28:38 -07:00
Mayuri Nehate
5166d90433
fix(ingest/sql-server): update queries to use escaped procedure name (#9127) 2023-10-27 10:19:51 -07:00
cjm98332
a021053a72
fix(ingest/mssql): Add UNIQUEIDENTIFIER data type as String (#8642)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-09-12 19:23:39 +05:30
RChygir
43d48ddde4
feat(ingest/mssql): load jobs and stored procedures (#5363) 2023-08-24 14:48:03 +05:30
mohdsiddique
05ef7db45e
fix(ingetion/mssql): convert dataset urns to lowercase (#8551)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-02 13:44:05 -07:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation (#8342)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Andrew Sikowitz
afbd52bdf0
test(ingest/mysql): Configure sql_server tests for arm64 (#8360)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-07-05 13:41:24 -04:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties (#8027) 2023-05-22 12:07:16 -07:00
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize (#7437) 2023-02-28 13:11:07 -08:00
Harshal Sheth
29c6c43cb3
feat(ingest): detect old or missing docker compose (#6466) 2022-11-16 22:38:00 -08:00
Shirshanka Das
e9c4c823d8
fix(ingest): bigquery-beta - ensure that status aspect is emitted for… (#6154)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-10-08 16:00:45 -07:00
Mugdha Hardikar
a449e8ba7d
feat(mssql): add multi database ingest support (#5516)
* feat(mssql): add multi database ingest support

* Delete older golden file.

* Update s3.md

* fix test setup
2022-08-15 21:37:47 -07:00
Kevin Hu
08c34bfe15
feat(ingest): capture MSSQL table+column descriptions (#4579)
* feat(ingest): capture MSSQL table+column descriptions

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-12 17:49:56 -07:00
Kevin Hu
1bad3c7bc9
fix(ingest): mssql - support database_alias (#4523) 2022-03-29 20:47:43 -07:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Ravindra Lanka
7f4cb87c57
Revert "fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. (#4140)" (#4218)
This reverts commit 6c75185445bbb23974932ff64cb142ee6bf5b51b.
2022-02-22 16:21:40 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. (#4140) 2022-02-16 19:45:07 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion (#4051) 2022-02-07 09:51:49 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Gabe Lyons
cad9be6b1d
fix(ingest): fix whitespace in foreign key urns (#3486) 2021-10-28 23:14:42 -07:00
Swaroop Jagadish
df0b4f258b
feat(ingest): support for primary and foreign key extraction from sql sources (#3316) 2021-09-30 23:04:47 -07:00
Gabe Lyons
ff527f4bed
feat(foreign keys): add foreign key models (#3275) 2021-09-22 10:29:27 -07:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization (#3079) 2021-08-11 15:47:18 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983)
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Gabe Lyons
aa253f5b3b
feat(deletes): add run commands (list, show, rollback) to datahub ingest (#2960) 2021-07-29 20:04:40 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install (#2872) 2021-07-14 20:02:48 -07:00
Harshal Sheth
2f921d15e8
fix(ingest): avoid setting timestamps unless source system provides it (#2843) 2021-07-08 12:11:06 -07:00
Harshal Sheth
d66381451a
feat(ingest): refactor mce comparison and add pytest update golden files option (#2812) 2021-06-30 16:53:20 -07:00
Kevin Hu
554e1637c5
fix(ingest): types for dbt (#2716) 2021-06-22 10:37:08 -07:00
Harshal Sheth
31eae24300
fix(ingest): support mssql encryption via ODBC (#2657) 2021-06-04 18:19:11 -07:00
shakti-garg
8ed14a62e2
feat(business_glossary): add new entity business term and its relationship with dataset and its fields (#2228)
Co-authored-by: shubham.garg <shubham.garg@thoughtworks.com>
2021-05-10 13:20:23 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test (#2392) 2021-04-13 17:30:24 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime (#2327) 2021-04-01 12:15:05 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Harshal Sheth
4e15feebfd
fix(ingestion): nullable types and timestamp precision (#2212) 2021-03-10 23:29:24 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164) 2021-03-07 11:26:47 -08:00
Harshal Sheth
0310e97b5b
feat(ingest): capture table descriptions (#2157) 2021-03-02 11:49:37 -08:00
Harshal Sheth
9336cbc743
test(ingest): verify the output of mssql (#2120) 2021-02-18 14:47:49 -08:00
Harshal Sheth
0063c04460 gometa-ingest -> datahub ingest 2021-02-15 18:29:27 -08:00
Harshal Sheth
d0bc3c55db Setup CI 2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8 Use type + config everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
df3e3da45b More autofixes 2021-02-15 18:29:27 -08:00