Sergio Gómez Villamor
8cae980286
tests(ingestion): moving some tests so they are available for sdk users ( #13540 )
2025-05-19 08:39:53 +02:00
Sergio Gómez Villamor
a8637abfe2
tests(kafka): fixing flaky tests ( #13171 )
2025-04-22 12:58:47 +02:00
Mayuri Nehate
01a2c0c779
fix(ingest/kafka): update dependency, tests ( #12159 )
2024-12-18 19:02:44 +05:30
skrydal
b091e4615d
feat(ingest/kafka): Flag for optional schemas ingestion ( #12077 )
2024-12-11 16:02:31 +00:00
Mayuri Nehate
8d15df0c11
feat(ingest/kafka): additional validation for oauth_db signature ( #11996 )
2024-12-05 12:41:43 +05:30
Mayuri Nehate
ecba2244f0
fix(ingest/kafka):add poll for admin client for oauth_cb ( #11985 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-11-28 19:23:30 +05:30
sid-acryl
86b8175627
fix(ingestion/kafka): OAuth callback execution ( #11900 )
2024-11-22 13:08:23 +05:30
Harshal Sheth
fa2ab1bcee
fix(ingest): add status aspect to dataProcessInstance ( #10757 )
2024-06-27 12:07:28 -07:00
aabharti-visa
8a905774f7
feat(ingestion/kafka)-Add support for ingesting schemas from schema registry ( #10612 )
2024-06-11 14:00:12 +02:00
Harshal Sheth
3d5735cbc5
chore(ingest): run pyupgrade for python 3.8 ( #10513 )
2024-05-15 22:31:05 -07:00
Harshal Sheth
c9613043c8
fix(ingest): improve kafka-connect test stability ( #9519 )
2024-01-03 11:28:22 -08:00
Shubham Jagtap
1741c07d76
feat(ingestion): Add test_connection methods for important sources ( #9334 )
2023-12-14 12:31:51 -05:00
Mayuri Nehate
5c40390a92
feat(ingest/kafka): support metadata mapping from kafka avro schemas ( #8825 )
...
Co-authored-by: Daniel Messias <danielcmessias@gmail.com>
Co-authored-by: Deepankarkr <deepankar.kumar@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-09-22 17:11:42 -07:00
Andrew Sikowitz
d92cbe99e0
test(ingest/kafka): Configure kafka connect tests for arm64 ( #8362 )
2023-07-10 12:27:29 -04:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation ( #8342 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability ( #8085 )
2023-05-25 15:40:56 -07:00
Harshal Sheth
b0f8c3de1e
refactor(ingest): simplify stateful ingestion provider interface ( #8104 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-05-23 12:57:57 -07:00
Shubham Jagtap
8cc6606e68
feat(ingestion/kafka): add description in dataset properties ( #7974 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: mohdsiddique <mohdsiddiquebagwan@gmail.com>
2023-05-17 11:03:08 -07:00
Harshal Sheth
8394dcb538
chore(ingest): change kafka connect mapped ports ( #7728 )
2023-04-04 18:38:30 +05:30
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize ( #7437 )
2023-02-28 13:11:07 -08:00
Andrew Sikowitz
e82e284982
fix(ingest/kafka): Remove topic from kafka browse path ( #7398 )
2023-02-22 18:38:08 -05:00
Mayuri Nehate
a05c5c4069
feat(ingest): extract kafka topic config properties as customProperties ( #6783 )
2022-12-22 09:34:55 +01:00
Harshal Sheth
47be95689e
refactor(ingest/stateful): remove most remaining state classes ( #6791 )
2022-12-19 13:40:48 -05:00
Tamas Nemeth
e41b455e14
fix(ingest): bigquery - sharded table support improvements ( #6789 )
2022-12-19 18:57:37 +01:00
Harshal Sheth
8a537b0559
feat(ingest): add datahub state inspect command ( #6763 )
2022-12-15 18:55:36 -05:00
Harshal Sheth
2f95719dba
feat(ingest): remove source config from DatahubIngestionCheckpoint ( #6722 )
2022-12-14 12:39:21 -05:00
Harshal Sheth
521e61d3bf
fix(ingest): Closeable as a context manager ( #6067 )
2022-10-13 12:49:50 -07:00
Ravindra Lanka
b8941ab190
feat(ingestion): Add fail-safe stale entity removal via configurable 'fail_safe_threshold' param. ( #6027 )
2022-09-22 16:09:22 -07:00
Ravindra Lanka
ee68f09624
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. ( #5766 )
2022-09-14 09:30:42 -07:00
Ravindra Lanka
228f3b50ea
feat(ingestion): send reports of ingestion runs to datahub ( #5639 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-19 09:08:17 -07:00
Shirshanka Das
f387fa6149
fix(ingest): ensure upgrade checks run async ( #5383 )
2022-07-16 17:38:33 -07:00
Shirshanka Das
860d475c5e
feat(ingest): improve domain ingestion usability ( #5366 )
2022-07-11 09:37:38 -07:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently ( #4496 )
2022-03-27 18:49:26 -05:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion ( #4051 )
2022-02-07 09:51:49 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method ( #3746 )
2021-12-16 20:07:38 -08:00
mayurinehate
f5d37c0621
feat(ingest): add complex types support in hive and trino source ( #3375 )
2021-10-18 22:53:51 -07:00
Swaroop Jagadish
146e79b9c3
fix(ingest): fixing support for kafka key schemas when only key schemas are present ( #3379 )
2021-10-12 18:50:39 -07:00
rslanka
8844240328
feat: Adding support for nested schemas in ingestion and visualization ( #3079 )
2021-08-11 15:47:18 -07:00
