65 Commits

Author SHA1 Message Date
Harshal Sheth
29e09185eb
fix(ingest): warn when API tracing is unexpectedly inactive (#13007) 2025-03-27 22:21:33 -07:00
Harshal Sheth
dce21bac1e
feat(sdk): add/remove methods + other small improvements (#12646) 2025-02-17 18:27:46 -08:00
Harshal Sheth
4448fc24b7
feat(sdk): add scaffolding for sdk v2 (#12554) 2025-02-06 11:17:21 -08:00
Aseem Bansal
2226820ad1
dev(ingest): use ruff instead of flake8 (#12359) 2025-01-16 08:19:07 +05:30
Harshal Sheth
7dbb3e60cb
chore(ingest): start using explicit exports (#11899) 2024-11-20 13:33:30 -08:00
Harshal Sheth
9589adebb3
feat(ingest): enable query usage stats by default (#11281) 2024-09-06 10:02:40 -07:00
Harshal Sheth
047bfcc412
fix(ingest): avoid using _inner_dict in urn iterator (#10492) 2024-05-15 22:08:27 +02:00
Harshal Sheth
0d780e5f8f
feat(ingest): sql parsing aggregator (#9786) 2024-02-09 16:27:45 -05:00
Harshal Sheth
89dff8f7bd
chore(ingest): start working on pydantic v2 support (#9220) 2023-11-10 09:34:08 -08:00
Harshal Sheth
9deb7be3fc
fix(ingest): refactor test markers + fix disk space issues in CI (#8938) 2023-10-03 20:17:49 -07:00
Mayuri Nehate
e867dbc3da
ci: separate airflow build and test (#8688)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-30 14:08:42 -07:00
Harshal Sheth
2d442161c4
ci(ingest/kafka): improve kafka integration test reliability (#8085) 2023-05-25 15:40:56 -07:00
Andrew Sikowitz
1ac1ccf26e
perf(ingest/bigquery): Improve bigquery usage disk usage and speed (#7825) 2023-04-14 18:09:43 -07:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting (#7763) 2023-04-10 14:14:42 -07:00
Andrew Sikowitz
54a372795b
test(ingest/bigquery): Add performance testing framework for bigquery usage (#7690)
- Creates metadata-ingestion/tests/performance directory
- Excludes metadata-ingestion/tests from docs generation
- Updates bigquery reporting around project state
2023-03-29 14:13:43 -07:00
Harshal Sheth
b82afa89f1
feat(ingest): enable joins across FileBackedDicts + add FileBackedList (#7506) 2023-03-09 15:22:03 -08:00
Harshal Sheth
44cfd21a65
chore(ingest): bump and pin mypy (#6584) 2022-12-02 19:53:28 +01:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ (#6204)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00
Harshal Sheth
bfb682ace7
build(ingest): pin mypy version (#6391) 2022-11-09 11:38:42 +01:00
Shirshanka Das
941770f3bc
feat(cli): quickstart - experimental support for backup restore (#5418) 2022-07-24 23:52:25 -07:00
Aseem Bansal
c942350e3c
feat(ci): split integration tests to reduce run time (#5205) 2022-06-20 21:08:37 +05:30
Aseem Bansal
4541379024
feat(build): changes to decrease build time, cancel runs in case of multiple commits (#5187) 2022-06-17 18:05:10 +05:30
buggythepirate
92338c7912
feat(ingest): Added new ingestion source SAP HANA (#4376)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-26 03:42:50 -07:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging (#4077) 2022-02-10 20:02:23 -08:00
mayurinehate
1d7ec8dba8
feat(ingest): add nifi source (#3681) 2021-12-08 14:56:31 -08:00
Harshal Sheth
cb2a3e6927
feat(profiler): dynamically combine queries (#3572) 2021-11-24 10:19:03 -08:00
Kevin Hu
40ecd96c7a
feat(ingest): replace and warn against relative imports (#3033) 2021-08-06 10:25:30 -07:00
Harshal Sheth
328b098d01
text(ingestion): test multiple python versions in CI (#2952) 2021-07-26 13:25:58 -07:00
Harshal Sheth
62d5306a28
fix(ingestion): add more mypy annotations (#2948) 2021-07-25 21:19:25 -07:00
Harshal Sheth
90e05df791
fix(ingest): refactor + fix recursion in lookml file loading logic (#2913) 2021-07-22 15:54:01 -07:00
Harshal Sheth
ad30f2b8ec
feat(ingestion): support multiple project IDs in bigquery usage stats (#2920) 2021-07-21 12:42:06 -07:00
Harshal Sheth
416f2a95df
feat(ingest): add support for Oracle spatial types (#2909) 2021-07-20 19:13:49 -07:00
Harshal Sheth
45e931dbac
feat(ingest): add can_add_aspect method for MCEs (#2905) 2021-07-17 20:02:07 -07:00
Harshal Sheth
283d131546
test(ingest): update tox test configurations and test airflow 2.x by default (#2906) 2021-07-17 20:00:50 -07:00
Harshal Sheth
be39037b10
build(ingest): reduce dependencies for dev install (#2872) 2021-07-14 20:02:48 -07:00
John Joyce
7591c8994a
feat(datahub cli): DataHub CLI Quickstart (#2689) 2021-06-14 17:15:24 -07:00
Harshal Sheth
6d875b8241
test(ingest): ensure transformer registry works for aliases (#2572) 2021-05-17 15:08:49 -07:00
Harshal Sheth
f590f11ff3
fix(ingest): check mypy types for test helpers (#2561) 2021-05-17 11:42:12 -07:00
Harshal Sheth
3edbdcb3e9
ci(ingest): ensure datahub imports work (#2502) 2021-05-06 07:48:56 -07:00
Harshal Sheth
2da5e1fd10
feat(ingest): setup scaffolding for tox testing (#2451) 2021-04-26 16:44:36 -07:00
Harshal Sheth
c64196e8cf
fix(ingest): setup pyproject.toml (#2446) 2021-04-23 23:50:28 -07:00
Harshal Sheth
2af4603e49
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators (#2393) 2021-04-14 13:40:24 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend (#2368) 2021-04-12 17:40:15 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime (#2327) 2021-04-01 12:15:05 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
07f4cb1199
feat: datahub check local-docker (#2295) 2021-03-26 10:03:51 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source (#2289) 2021-03-23 20:15:44 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Pedro Silva
6a0c402a58
feat(ingest): Add support for druid (#2235) 2021-03-17 20:06:48 -07:00