Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency ( #2385 )
...
* Added pyarrow dependency
* Fix #2378
* Fix Looker
* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
05c8d1ebb6
Fix #2153 - Lineage date parsing ( #2387 )
...
* Check for start_date
* Fix date keys and remove repr
* Add date management tests
2022-01-24 12:29:16 -08:00
Pere Miquel Brull
6a8fc9e0fb
import on call ( #2382 )
2022-01-24 12:29:03 -08:00
Matt
71e78ab460
Add Documentation for backend debugging ( #2375 )
2022-01-24 09:28:42 -08:00
Akash Jain
23d3b8648d
Added env for airflow and elasticsearch ( #2381 )
2022-01-24 21:44:57 +05:30
Alberto Miorin
7a6e9a8060
Fix #2377 : Need versions support for deleted entities ( #2380 )
2022-01-24 19:26:32 +05:30
Akash Jain
71a2b0145c
bump version ingestion-core ( #2374 )
2022-01-24 14:10:31 +05:30
Akash Jain
a850628593
update make core_generate to run on ubuntu as well ( #2373 )
2022-01-24 14:04:27 +05:30
Sriharsha Chintalapani
429735f503
Fix #2368 : Support soft deleted entities update in ElasticSearch ( #2369 )
2022-01-23 23:03:30 -08:00
Akash Jain
c3ba6d6efd
Update _version.py ( #2371 )
2022-01-24 12:26:54 +05:30
Akash Jain
657edfeb9a
use venv instead ( #2370 )
2022-01-24 12:21:35 +05:30
Suresh Srinivas
8bb50d1a24
Fixes #2364 Use terms entityType and entityName consistently ( #2365 )
2022-01-23 15:23:19 -08:00
Suresh Srinivas
eeecde3524
Fixes #2361 - Move DELETE related and DELETE + PUT tests to common class ( #2362 )
2022-01-23 11:48:53 -08:00
Suresh Srinivas
6e7cf40a33
Fixes #1142 - Add entityDeleted event for DELETE APIs ( #2360 )
2022-01-23 10:39:47 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify ( #2359 )
...
* Add installing instructions
* Unify makefile and automate bump version setup
* Update submake call
* Mark targets as phony
* Add help to recipes
* Update core publish
* Do not trigger install from echo
* Run make from root
* Prepare py_format from isort and black
* Run py_format
* Check black and isort formatting
* Update comment msg
2022-01-23 09:47:06 -08:00
Ayush Shah
6f5a4eafe7
Bigquery support for env without passing any config ( #2160 )
...
* Bigquery support for env without passing any config
* Bigquery File formatted
2022-01-22 22:25:48 -08:00
parthp2107
a784bc217d
added periods to json schema descriptions ( #2334 )
2022-01-22 22:25:15 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service ( #2357 )
...
* Fix #2340 : Ingestion pipeline deployment refactor - update ingestion
* Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service
* Fix #2355 : Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353 : Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
darth-coder00
73a657ef3f
UI service revamp ( #2350 )
...
* Added UI changes to support service with ingestion
* Added new interfaces from schema
* Integration of new databaseService and airflowPipeline APIs
2022-01-23 03:12:50 +05:30
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI ( #2349 )
...
* Prepare cli module
* Check call
* Prepare backup
* Run pytest on schema changes
* Remove traces
* Add backup plugin
* run generate during CI
* Install after generate
* Address logger hotspot
2022-01-22 12:08:14 -08:00
ulixius9
8fca53ec52
ISSUE-2048: Bigquery data profiler uses schema name instead of project id ( #2335 )
...
* ISSUE-2048: Bigquery data profiler uses schema name instead of project id
* Update bigquery.py
* Update bigquery.py
* Update profiler_runner.py
* Update profiler.py
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:33:15 +05:30
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser ( #2271 )
...
* Fix 2270: Column Type Parser
* Fix 2270: Column Type Parser
* Added checks to allow arrayDataType and dataTypeDisplay
* Modified - arrayDataType and dataTypeDisplay
* Update sql_source.py
* Update sql_source.py
* file formatted
* Modified according to column_type_parser.py
* modified / refactored / deleted the files
* Added Tests, modified sql_source
* file formatted
* Added missing datatypes
* Added Tests
* Added Tests
* Added Tests - refactored expected output into a json file
* file formatted
* Sample Data Updated
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:32:58 +05:30
darth-coder00
e2602b81fb
Deleted dataset UI support ( #2333 )
...
* Deleted dataset UI support
* Added API integration to get deleted tables
* Added deleted badge on search results
* Addressing comments
* Addressing comment
* Restrict Versions button for deleted entities
2022-01-22 22:43:39 +05:30
Sachin Chaurasiya
590a0ab624
UI: Adding Policy UI. ( #2259 )
...
* [WIP]UI: Adding Policy UI.
* Integrating API to get Roles and Policy.
* Adding API integration to create and update role
* Adding form for Rule
* changing types
* removing suggested rules
* Changing rule form modal layout.
* Adding support for adding rules
* Adding support for deleting rule
* Adding support for editing rules
* Adding validation for rules
* Adding isAdminOnly check for roles page
* Adding license to new file
* Minor Fix
* replacing break-all with break-words class.
* Addressing review comments
2022-01-22 20:46:45 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Matt
8e5a9c1f0e
Set Access Control Rule default priority to 25000 ( #2346 )
2022-01-22 02:14:56 -08:00
Matt
319fde9207
Throw error if multiple rules within policy have same operation ( #2344 )
2022-01-22 01:09:51 -08:00
Sriharsha Chintalapani
10e71a7e09
Fix #2340 : Ingestion pipeline deployment refactor ( #2341 )
...
* Fix #2340 : Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
Sachin Chaurasiya
69e7096650
Adding some Misc UI Changes ( #2327 )
2022-01-22 10:24:03 +05:30
Matt
2d44e11f8a
Update py-ingestion-core version control and publish workflows ( #2330 )
2022-01-21 08:58:24 -08:00
parthp2107
91767ee4cf
added tests for tags and description ( #2332 )
2022-01-21 21:37:25 +05:30
darth-coder00
9da9266c3a
Scrollable description ( #2266 )
...
* Scrollable description
* Scroll enhancement
2022-01-21 18:00:39 +05:30
codingwithabhi
ddf4ae261b
Bigquery credential configuration updated ( #2041 )
...
* bigquery-credentials-in-bigquery-json
* ui-changes-reverted
* credentials-moved-to-options
* Update bigquery.json
* Update bigquery.py
* delete-temp-code-added
* delete-temp-code-added
* removed-init-from-config
* Update datatypes_test.py
* Update column_helpers.py
* code-formatted
Co-authored-by: Abhishek <abhishek@Abhisheks-MacBook-Air.local>
Co-authored-by: = <=>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-21 17:23:02 +05:30
Sachin Chaurasiya
cd65037ac7
Fix: Fixing #2306 Links in team description is considered as a page. ( #2329 )
...
* Fix: Fixing #2306 Links in team description is considered as a page.
* Adding isValidURl util
* Checking falsy condition first
2022-01-21 14:06:17 +05:30
Matt
845b62d48d
Add Access Control Policy for DataConsumer role ( #2328 )
2022-01-21 00:26:06 -08:00
Matt
2a80caa8fe
Add logic to initialize relationships from seed data during application startup ( #2307 )
...
* Add logic to initialize relationships from seed data during application startup
* Remove ingestion related code for access control policies
* Move PolicyEvaluator init to PolicyResource
2022-01-20 22:18:03 -08:00
Pere Miquel Brull
979bad18d9
Update python checkstyle ( #2315 )
2022-01-20 16:18:18 -08:00
Sriharsha Chintalapani
7312aff270
Fix #2320 : Fix DB migration scripts to upgrade to 0.8 from 0.7
2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
18f5c288fe
Fix #2320 : Fix DB migration scripts to upgrade to 0.8 from 0.7
2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
138fd0747d
Fix #2320 : Fix DB migration scripts to upgrade to 0.8 from 0.7
2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86
fix upgrade
2022-01-20 15:08:03 -08:00
Alberto Miorin
b6fee6cb1c
Fix #2209 : Path error: bootstrap_storage.sh not able to find main class ( #2319 )
2022-01-20 21:28:26 +01:00
Sachin Chaurasiya
a41dd01aaa
Fix : Fixing #2299 Able to add the tag itself as it's associated tag and #2311 ( #2310 )
...
* Fix : Fixing #2299 Able to add the tag itself as it's associated tag.
* Fixing #2311 UI struct text in dataTypeDisplay looks like a link
2022-01-20 23:15:23 +05:30
Shannon Bradshaw
b0beac243b
GitBook: [ #89 ] Add docker-compose to list of requirements.
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
2e4e8c823c
GitBook: [ #88 ] No subject
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
43f1b8404d
GitBook: [ #87 ] Fix broken links in glue connector docs
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
c8acf54818
GitBook: [ #86 ] No subject
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
ab14355936
GitBook: [ #85 ] No subject
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
ccc2f045b3
GitBook: [ #84 ] Update Glue with greater detail for service settings.
2022-01-20 09:14:01 -08:00
Shannon Bradshaw
072dec0aaf
GitBook: [ #83 ] Clarified when to use different authentication methods.
2022-01-20 09:14:01 -08:00
