81 Commits

Author SHA1 Message Date
Sergio Gómez Villamor
8cae980286
tests(ingestion): moving some tests so they are available for sdk users (#13540) 2025-05-19 08:39:53 +02:00
Harshal Sheth
7c791db087
feat(ingest/sql): column logic + join extraction (#13426) 2025-05-12 17:19:44 -07:00
sagar-salvi-apptware
b79857fd94
fix(ingest/sql-common): sql_common to use SqlParsingAggregator (#12220) 2024-12-29 18:52:05 +05:30
Sergio Gómez Villamor
31edb46dbc
feat(ingestion): adds env property in ContainerProperties (#11214)
Co-authored-by: siladitya2 <siladitya2@gmail.com>
2024-09-18 14:56:52 +05:30
Felix Lüdin
9619553e2d
fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (#10898)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-06 12:52:20 -07:00
Harshal Sheth
1dae37a8ed
fix(ingest/bigquery): remove last modified timestamp fallback (#10431) 2024-05-06 16:30:04 -07:00
Harshal Sheth
7d31420b69
feat(ingest): materialize terms produced by ingestion (#10249) 2024-04-18 10:48:16 -07:00
Mayuri Nehate
2de0e62ac4
feat(ingest): add classification for sql sources (#10013)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-03-12 09:23:20 -07:00
Tamas Nemeth
f561efee83
fix(ingest/mysql): Adding support for bit type (#9950) 2024-02-28 15:00:14 +01:00
Shubham Jagtap
1741c07d76
feat(ingestion): Add test_connection methods for important sources (#9334) 2023-12-14 12:31:51 -05:00
Andrew Sikowitz
27f23ecdd5
feat(ingest/unity): GE Profiling (#8951) 2023-12-06 13:59:23 -05:00
Mayuri Nehate
08fb730676
fix(ingest): drop deprecated database_alias from sql sources (#9299)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-11-28 15:49:49 -05:00
Mayuri Nehate
f402090c1e
feat(ingest): support view lineage for all sqlalchemy sources (#9039) 2023-10-26 09:14:32 -07:00
Aseem Bansal
8fb5912978
fix(mysql): getting table name correctly (#8476)
Co-authored-by: Ellie O'Neil <oneile729@gmail.com>
2023-07-23 17:10:54 +05:30
Ellie O'Neil
d734b2849e
feat(ingest/mysql): Add estimate row count for mysql (#8420) 2023-07-20 08:06:26 +05:30
Tamas Nemeth
e4736d7b46
fix(ingest/s3): Deequ import rearragement (#8389) 2023-07-10 14:40:06 -04:00
Andrew Sikowitz
1f84bf5b2b
fix(ingest/sql-common): Fix profile_table_level_only (#8331)
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-07 19:05:50 -04:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation (#8342)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Harshal Sheth
4e9c652707
feat(ingest): add env to container properties (#8027) 2023-05-22 12:07:16 -07:00
John Joyce
18f387c6ea
fix(cli): Adding exit code to correctly return failure or success (#7520)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-03-13 13:32:40 -07:00
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize (#7437) 2023-02-28 13:11:07 -08:00
Mayuri Nehate
69a2347db1
feat(ingest): update profiling to fetch configurable number of sample values (#6859) 2022-12-27 08:57:26 +01:00
Mayuri Nehate
5c99f20b7d
fix(ingest): mysql - fix mysql ingestion issue with non-lowercase database (#6713) 2022-12-12 10:48:01 +01:00
Harshal Sheth
bf307a4bcf
feat(ingest): run profiler in more cardinality cases (#6397) 2022-12-07 12:20:06 -05:00
Mayuri Nehate
a1b3929b38
fix(ingest): fix typo in unique count profiling (#6517) 2022-11-22 14:23:01 -05:00
Mayuri Nehate
443d15f346
fix(ingest): mysql - fix native data type for mysql set type (#6407) 2022-11-14 14:33:26 +01:00
Harshal Sheth
ae2ea524a4
fix(ingest): use profiler config settings correctly (#6354)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-09 19:03:22 -08:00
Shirshanka Das
e9c4c823d8
fix(ingest): bigquery-beta - ensure that status aspect is emitted for… (#6154)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-10-08 16:00:45 -07:00
Harshal Sheth
feb9d59b4d
fix(ingest): remove unused mysql golden file (#6106) 2022-10-04 20:26:51 -07:00
Mugdha Hardikar
e448bb8832
feat(ingest): mysql - support multiple database in single recipe (#5684) 2022-08-26 19:47:49 +02:00
Shirshanka Das
860d475c5e
feat(ingest): improve domain ingestion usability (#5366) 2022-07-11 09:37:38 -07:00
Shirshanka Das
558a65a3c3
fix(ci): fix mysql test and attempt kafka-connect ingestion (#5352) 2022-07-07 08:28:34 -07:00
David Haglund
df9e07fda2
fix: replace direct and indirect references to linkedin with datahub-project (#4557)
* Update links for github-related links to use datahub-project:
  - https://github.com
  - https://img.shields.io/github/...
  - https://raw.githubusercontent.com/...
* Also replace references for github repo linkedin/datahub with
  datahub-project/datahub.
2022-04-04 14:39:30 -05:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality (#4319)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-09 09:29:58 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion (#4051) 2022-02-07 09:51:49 -08:00
ksrinath
4c24f386a6
feat(model): data quality model (#3787)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
Tamas Nemeth
646dd6d5f0
fix(ingest): profiler - enable profiling for low cardinality number columns (#3990) 2022-01-30 17:36:18 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets (#3656) 2022-01-17 15:01:23 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) (#3900) 2022-01-16 12:03:38 -08:00
Tamas Nemeth
599edd22ae
fix(ingest): profiling - disable expensive profilers by default (#3759) 2021-12-17 17:17:25 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
Harshal Sheth
1afcb661bd
feat(ingest): add --strict-warnings option (#3665) 2021-12-03 14:07:07 -08:00
Harshal Sheth
b0b275ee34
test(profiler): make profiling tests more comprehensive (#3513) 2021-11-07 18:53:53 -08:00
Gabe Lyons
cad9be6b1d
fix(ingest): fix whitespace in foreign key urns (#3486) 2021-10-28 23:14:42 -07:00
Swaroop Jagadish
df0b4f258b
feat(ingest): support for primary and foreign key extraction from sql sources (#3316) 2021-09-30 23:04:47 -07:00
John Joyce
33f4d2ede0
fix(upgrade): Improving NoCodeUpgrade logic to account for Bootstrap logic (#3301) 2021-09-28 16:30:49 -07:00
Gabe Lyons
ff527f4bed
feat(foreign keys): add foreign key models (#3275) 2021-09-22 10:29:27 -07:00
Swaroop Jagadish
d1b57927a9
fix(frontend): replacing broken link for default avatar (#3156) 2021-08-24 21:40:39 -07:00