Alberto Miorin
4aa8f9b8e2
Fix #2414 : AirflowPipeline is storing the service in the entity_relationship and in the json column of airflow_pipeline_entity ( #2415 )
2022-01-25 11:56:32 +01:00
Matt
c092b03171
Add RBAC for UpdateLineage MetadataOperation ( #2406 )
2022-01-25 02:11:37 -08:00
Sriharsha Chintalapani
b47f49e418
Fix #2403 : Add support for deleted entities to be published into Slack Events ( #2404 )
...
* Fix #2403 : Add support for deleted entities to be published into Slack Events
* Fix #2403 : Add support for deleted entities to be published into Slack Events
2022-01-25 14:39:12 +05:30
Sriharsha Chintalapani
0e736012a9
Fix #1843 : Add Delta Lake Connector ( #1844 )
...
* Fix #1843 : Add Delta Lake Connector
* Fix #1843 : Add Delta Lake Connector
2022-01-25 14:35:44 +05:30
Ayush Shah
917ccd7147
Bigquery and Helpers fix - Ingestion ( #2408 )
...
* Bigquery Ingestion fix - error handling and keyerror
* Update helpers.py and column_type_parser
* Update helpers.py
2022-01-25 14:30:09 +05:30
Vivek Ratnavel Subramanian
47b9afa8c4
Fix #1877 : Editing services description does not save ( #2397 )
...
* Fix #1877 : Editing services description does not save
2022-01-25 00:15:59 -08:00
Pere Miquel Brull
ce5df0eb88
Use path str properly ( #2405 )
2022-01-25 08:55:44 +01:00
darth-coder00
e75e9d0780
UI added support all deleted entities ( #2388 )
...
* Added deleted functionality for explore page
* Added UI support for deleted topics
* Added UI support for deleted Dashboards and Pipelines
2022-01-25 13:20:52 +05:30
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency ( #2398 )
2022-01-25 11:58:10 +05:30
parthp2107
f199f2fc39
added roles page tests ( #2376 )
...
* added roles page tests
* addressing style-check
2022-01-25 07:47:07 +05:30
darth-coder00
af0fb11c23
Fix #2389 : Revamp Service details with new Ingestion tab
2022-01-24 14:44:36 -08:00
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency ( #2385 )
...
* Added pyarrow dependency
* Fix #2378
* Fix Looker
* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing ( #2387 )
...
* Check for start_date
* Fix date keys and remove repr
* Add date management tests
2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call ( #2382 )
2022-01-24 12:29:03 -08:00
Matt
71e78ab460
Add Documentation for backend debugging ( #2375 )
2022-01-24 09:28:42 -08:00
Akash Jain
23d3b8648d
Added env for airflow and elasticsearch ( #2381 )
2022-01-24 21:44:57 +05:30
Alberto Miorin
7a6e9a8060
Fix #2377 : Need versions support for deleted entities ( #2380 )
2022-01-24 19:26:32 +05:30
Akash Jain
71a2b0145c
bump version ingestion-core ( #2374 )
2022-01-24 14:10:31 +05:30
Akash Jain
a850628593
update make core_generate to run on ubuntu as well ( #2373 )
2022-01-24 14:04:27 +05:30
Sriharsha Chintalapani
429735f503
Fix #2368 : Support soft deleted entities update in ElasticSearch ( #2369 )
2022-01-23 23:03:30 -08:00
Akash Jain
c3ba6d6efd
Update _version.py ( #2371 )
2022-01-24 12:26:54 +05:30
Akash Jain
657edfeb9a
use venv instead ( #2370 )
2022-01-24 12:21:35 +05:30
Suresh Srinivas
8bb50d1a24
Fixes #2364 Use terms entityType and entityName consistently ( #2365 )
2022-01-23 15:23:19 -08:00
Suresh Srinivas
eeecde3524
Fixes #2361 - Move DELETE related and DELETE + PUT tests to common class ( #2362 )
2022-01-23 11:48:53 -08:00
Suresh Srinivas
6e7cf40a33
Fixes #1142 - Add entityDeleted event for DELETE APIs ( #2360 )
2022-01-23 10:39:47 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify ( #2359 )
...
* Add installing instructions
* Unify makefile and automate bump version setup
* Update submake call
* Mark targets as phony
* Add help to recipes
* Update core publish
* Do not trigger install from echo
* Run make from root
* Prepare py_format from isort and black
* Run py_format
* Check black and isort formatting
* Update comment msg
2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config ( #2160 )
...
* Bigquery support for env without passing any config
* Bigquery File formatted
2022-01-22 22:25:48 -08:00
parthp2107
a784bc217d
added periods to json schema descriptions ( #2334 )
2022-01-22 22:25:15 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service ( #2357 )
...
* Fix #2340 : Ingestion pipeline deployment refactor - update ingestion
* Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service
* Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
darth-coder00
73a657ef3f
UI service revamp ( #2350 )
...
* Added UI changes to support service with ingestion
* Added new interfaces from schema
* Integration of new databaseService and airflowPipeline APIs
2022-01-23 03:12:50 +05:30
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI ( #2349 )
...
* Prepare cli module
* Check call
* Prepare backup
* Run pytest on schema changes
* Remove traces
* Add backup plugin
* run generate during CI
* Install after generate
* Address logger hotspot
2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id ( #2335 )
...
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id
* Update bigquery.py
* Update bigquery.py
* Update profiler_runner.py
* Update profiler.py
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser ( #2271 )
...
* Fix 2270: Column Type Parser
* Fix 2270: Column Type Parser
* Added checks to allow arrayDataType and dataTypeDisplay
* Modified - arrayDataType and dataTypeDisplay
* Update sql_source.py
* Update sql_source.py
* file formatted
* Modified according to column_type_parser.py
* modified / refactored / deleted the files
* Added Tests, modified sql_source
* file formatted
* Added missing datatypes
* Added Tests
* Added Tests
* Added Tests - refactored expected output into a json file
* file formatted
* Sample Data Updated
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:32:58 +05:30
darth-coder00
e2602b81fb
Deleted dataset UI support ( #2333 )
...
* Deleted dataset UI support
* Added API integration to get deleted tables
* Added deleted badge on search results
* Addressing comments
* Addressing comment
* Restrict Versions button for deleted entities
2022-01-22 22:43:39 +05:30
Sachin Chaurasiya
590a0ab624
UI: Adding Policy UI. ( #2259 )
...
* [WIP]UI: Adding Policy UI.
* Integrating API to get Roles and Policy.
* Adding API integration to create and update role
* Adding form for Rule
* changing types
* removing suggested rules
* Changing rule form modal layout.
* Adding support for adding rules
* Adding support for deleting rule
* Adding support for editing rules
* Adding validation for rules
* Adding isAdminOnly check for roles page
* Adding license to new file
* Minor Fix
* replacing break-all with break-words class.
* Addressing review comments
2022-01-22 20:46:45 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Matt
8e5a9c1f0e
Set Access Control Rule default priority to 25000 ( #2346 )
2022-01-22 02:14:56 -08:00
Matt
319fde9207
Throw error if multiple rules within policy have same operation ( #2344 )
2022-01-22 01:09:51 -08:00
Sriharsha Chintalapani
10e71a7e09
Fix #2340 : Ingestion pipeline deployment refactor ( #2341 )
...
* Fix #2340 : Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
Sachin Chaurasiya
69e7096650
Adding some Misc UI Changes ( #2327 )
2022-01-22 10:24:03 +05:30
Matt
2d44e11f8a
Update py-ingestion-core version control and publish workflows ( #2330 )
2022-01-21 08:58:24 -08:00
parthp2107
91767ee4cf
added tests for tags and description ( #2332 )
2022-01-21 21:37:25 +05:30
darth-coder00
9da9266c3a
Scrollable description ( #2266 )
...
* Scrollable description
* Scroll enhancement
2022-01-21 18:00:39 +05:30
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated ( #2041 )
...
* bigquery-credentials-in-bigquery-json
* ui-changes-reverted
* credentials-moved-to-options
* Update bigquery.json
* Update bigquery.py
* delete-temp-code-added
* delete-temp-code-added
* removed-init-from-config
* Update datatypes_test.py
* Update column_helpers.py
* code-formatted
Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Sachin Chaurasiya
cd65037ac7
Fix: Fixing #2306 Links in team description is considered as a page. ( #2329 )
...
* Fix: Fixing #2306 Links in team description is considered as a page.
* Adding isValidURl util
* Checking falsy condition first
2022-01-21 14:06:17 +05:30
Matt
845b62d48d
Add Access Control Policy for DataConsumer role ( #2328 )
2022-01-21 00:26:06 -08:00
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup ( #2307 )
...
* Add logic to initialize relationships from seed data during application startup
* Remove ingestion related code for access control policies
* Move PolicyEvaluator init to PolicyResource
2022-01-20 22:18:03 -08:00
Pere Miquel Brull
979bad18d9
Update python checkstyle ( #2315 )
2022-01-20 16:18:18 -08:00
Sriharsha Chintalapani
7312aff270
Fix #2320 : Fix DB migration scripts to upgrade to 0.8 from 0.7
2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
18f5c288fe
Fix #2320 : Fix DB migration scripts to upgrade to 0.8 from 0.7
2022-01-20 15:08:03 -08:00
