Harshal Sheth
7d63399d00
fix(ingest): fix serde for empty dicts in unions with null ( #6745 )
...
The code changes in https://github.com/acryldata/avro_gen/pull/16 , but tests are written here.
2022-12-13 08:17:24 +01:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ ( #6204 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00
Harshal Sheth
0ca3383d30
feat(ingest): support reserved keywords in model codegen ( #6351 )
2022-11-02 22:26:56 -07:00
Harshal Sheth
b4687ffceb
feat(ingest): drop plugin support for airflow 1.x ( #6331 )
2022-11-01 21:12:34 -07:00
Ravindra Lanka
b638bcf442
feat(ingestion): Documentation on adding stateful ingestion use-cases to new sources ( #5985 )
2022-09-21 09:02:50 -07:00
Shirshanka Das
f70bb201ce
fix(build): fix preflight script for m1 for lib postgresql ( #5957 )
2022-09-15 13:21:02 -07:00
Harshal Sheth
e23523a781
fix(ingest): fix type annotations on some pydantic fields ( #5795 )
2022-09-14 11:05:31 -07:00
Harshal Sheth
a1e1d2fd0a
feat(ingest): add ConfigEnum type ( #5734 )
2022-09-14 09:57:42 -07:00
Harshal Sheth
8f72de5d8b
refactor(ingest): move aspect maps to dedicated file ( #5821 )
2022-09-10 18:43:11 -07:00
Shirshanka Das
b9068ffd2e
fix(m1): preflight works with brew postgres@14 ( #5868 )
2022-09-08 15:02:30 +05:30
Shirshanka Das
d911b15da5
docs(ingest): improve doc gen, docs for snowflake, looker ( #5867 )
2022-09-08 00:38:10 -07:00
Harshal Sheth
abddc01877
fix(ingest): fix doc generation import ordering issue with postgres ( #5846 )
...
Relying on the correct import directly, rather than going through
SQLAlchemy's import wrapper (in their dialect.py) allows us to bypass
this strange error in doc generation.
2022-09-07 21:44:33 +05:30
Harshal Sheth
6d83eb9449
refactor(ingest): prefer as imports instead of pegasus2avro files ( #5820 )
2022-09-05 21:09:07 -07:00
Shirshanka Das
fe52df6bab
fix(docs): fixes issue with auto-generated ingestion doc ( #5733 )
2022-08-26 09:31:53 -07:00
Harshal Sheth
41e8f7b203
feat(ingest): generate a list of aspects in codegen ( #5633 )
2022-08-15 17:21:40 -07:00
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP ( #5566 )
2022-08-07 07:52:58 -07:00
Piotr Sierkin
828a711684
feat(ingest): dbt - control over emitting test_results, test_definitions, etc. ( #5328 )
...
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-06 21:42:53 -07:00
Harshal Sheth
3c2a9c585b
fix(ingest): activate mypy support for ParamSpec typing annotation ( #5551 )
2022-08-04 17:05:52 +05:30
Tamas Nemeth
d08a8f79ab
feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed ( #5435 )
2022-07-22 14:20:12 +05:30
Vincent Koc
d60f789519
chore: Refactor Python Codebase ( #5113 )
2022-07-06 16:09:27 +05:30
Shirshanka Das
64fae6121a
feat(docs): ingest - add tab for config json schema ( #5086 )
2022-06-03 14:57:36 -07:00
Shirshanka Das
b2d957ddbf
feat(docs): make config section for ingestion sources auto-open by default ( #5075 )
2022-06-02 18:27:13 +02:00
Shirshanka Das
83d9f7d6e6
fix(ingest): fix build breakage due to traitlets 5.2.2 bug ( #5051 )
2022-05-31 14:42:56 -07:00
Shirshanka Das
3bb8a41c2a
fix(build): docgen should fail if plugin is not loadable ( #5038 )
2022-05-29 08:09:02 -07:00
Aseem Bansal
21f8d4be03
fix(redash): improve logging for debugging, add validation for dataset urn, some refactoring ( #4997 )
2022-05-26 16:24:54 +05:30
Aseem Bansal
b77f381429
doc(ingestion): default boolean fix, broken bigquery docgen ( #4984 )
2022-05-24 23:17:27 +05:30
Shirshanka Das
8b235b1706
fix(docs): ingest - sort modules, fix small typos ( #4880 )
2022-05-09 13:01:44 -07:00
Shirshanka Das
d152ca367c
fix(docgen): fix failure count incrementing during doc generation ( #4806 )
2022-05-03 22:56:35 +02:00
Shirshanka Das
a9ad138172
feat(ingest): docs - overhaul source connector docs to make it code driven ( #4798 )
...
Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
2022-05-02 00:18:15 -07:00
Aseem Bansal
f099aeb550
fix(m1): tweak m1 preflight ( #4771 )
2022-04-29 10:29:47 +02:00
John Joyce
325c9b0f08
Adding new models to python codegen ( #4726 )
2022-04-22 10:12:58 -07:00
John Joyce
c69310522b
feat(metadata service): Introducing Platform Events ( #4477 )
2022-03-29 18:32:04 -07:00
Kevin Neville
d8e6f890a9
fix: Replace old repository link with new link ( #4446 )
2022-03-18 14:12:19 -07:00
Swaroop Jagadish
eaf7b02b2a
docs(model): auto-generated docs and hand-written docs for the metadata model ( #4189 )
2022-02-18 09:45:45 -08:00
Harshal Sheth
75d2ec2a39
ci(ingestion): fix airflow 1 deps for tox ( #4083 )
2022-02-17 00:33:28 -08:00
Swaroop Jagadish
d1a14abb53
fix(docs): fixing metadata model doc generation script and updating png ( #4120 )
2022-02-10 22:56:28 -08:00
Aseem Bansal
d3b7cece7a
fix(build): m1 - harden pre-flight script for M1 ( #3958 )
2022-01-27 23:01:18 -08:00
John Joyce
271784c9c1
feat(ui): UI-based ingestion (as featured in Dec Townhall) ( #3975 )
2022-01-27 10:33:12 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) ( #3900 )
2022-01-16 12:03:38 -08:00
Gabe Lyons
b7a732d96b
fix(build): correcting m1 preflight check ( #3677 )
2021-12-06 21:49:08 -08:00
Swaroop Jagadish
a16c432a1b
feat(metadata-model): adding metadata model doc generation and upload… ( #3667 )
2021-12-05 12:22:17 -08:00
Tamas Nemeth
f043b79f4c
feat(build): Preflight script for metadata ingestion on m1 ( #3652 )
2021-12-01 11:16:32 -08:00
Swaroop Jagadish
806b28e697
fix(ci): SKIP_RELEASE_UPLOAD flag was not being respected by python release script ( #3588 )
2021-11-16 15:11:25 -08:00
Swaroop Jagadish
9fdcbdabb0
feat(ci): adding support for env variables in python release script ( #3587 )
2021-11-16 12:15:52 -08:00
Harshal Sheth
6e5d1fe42f
fix(ingest): switch to avro from deprecated avro-python3 ( #3412 )
2021-10-18 15:35:27 -07:00
Swaroop Jagadish
9dd7303bad
feat(build): adding support for python codegen for all aspects, not just the snapshot ones ( #3299 )
2021-09-26 17:22:58 -07:00
rslanka
c418bc845c
feat(Analytics): Support for Timeseries Aggregated Statistics ( #3207 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Dexter Lee <dexter@acryl.io>
2021-09-14 18:35:10 -07:00
John Joyce
352a0abf8d
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect ( #2983 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-07-30 17:41:03 -07:00
Harshal Sheth
6135eed757
test(ingestion): fix flaky package discovery test ( #2949 )
2021-07-25 21:15:22 -07:00
Harshal Sheth
f01bb8bcd0
build(ingestion): add version prompt to release script ( #2866 )
2021-07-13 15:02:14 -07:00
