* fix: removed sqlparse dependency for system metrics
* fix: update sample query
* fix: move system test os retrieval to `.get()`
* fix: move os.environ to `get`
* feat: updated GE integration to match new test workflow
* feat: updated unit tests to match new signature
* feat: added GE integration tests
* feat: ran python linting
* feat: updated dbt ingestion to match new TestSuite workflow
* feat: ran python linting
* feat: remove testSuite from Elasticsearch event test case update
* feat: ran java linting
* fixed issue for lineage description
* fixed issue while ingesting
* fixed issue while ingesting
* added test case for Lingeage with description
* addressing comments .. enhancement
* addressing comments .. enhancement
* modified py test case and removed description from addLineage as we are not using it.
* add support for topic entity and description in lineage details
* fix pylint & test
* pytest fix
* fix column lineage null issue
---------
Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
* feat: fixed update to
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.
* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity
* feat: add status object to test case results
* feat: ran python linting
* feat: Add threshold and strategy logic on the custom SQL object test
* feat: ran python linting
* feat: added safety checks for custom sql query
* feat: ran python linting
* feat: refactor ingestion flow logic
* feat: ran python linting
* feat: update tests to match new workflow
* feat: ran python linting
* feat: update sample data test suite name
* feat: Added backend logic to support logical and executable test suites
* feat: clean up java and json code
* feat: added sample data for logical and executable test suites
* feat: remove executable from CreateTestSuite
* feat: ran python and java linting
* feat: added README info for data quality structure
* skipping cypress to keep main green
* fixed typescript type issue
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* feat: fetch metrics from system tables
* feat: add permission doc for fetching metrics from system tables
* feat: fix E2E tests to reflect full table row count after table metric update
* feat: ran linting
* feat: fix doc string engine name + function typing
* feat: ran python linting
* fix: extracted profiler object from workflow and implemented factory to allow service base logic
* fix: ran python linting
* fix: renamed `base` to `base_profiler_source`
* fix: add logic to set correct database for BQ multi project ID connections
* fix: ran python linting
* fix: optimize system metrics retrieval for memory
* fix: ran python linting
* fix: logic to retrieve unique system metrics operations
* fix: added logic to clean up query before parsing it
* fix: added E2E tests for rds, bq, snflk system metrics
* fix: ran python linting
* fix: fix postgres query + add default byte size to env var
* fix: ran python linting
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.
* added the ordinalPosition data point into the Column constructor.
* renamed variable to better describe its usage.
* updated profile errors.
Hive connections now comment columns by default.
* removed print statements
* Cleaned up code by pulling check into its own function
* Updated median function to return null when it is being used for first and third quartiles.
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.
* added the ordinalPosition data point into the Column constructor.
* renamed variable to better describe its usage.
* updated profile errors.
Hive connections now comment columns by default.
* removed print statements
* Cleaned up code by pulling check into its own function
* Updated median function to return null when it is being used for first and third quartiles.
* removed print statements and ran make py_format
* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine
* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala
* added cast to decimal for column when running average in mean.py
* fixed lint error
* fixed ui ordering of precision and scale.
Precision should be ordred in front of scale since the precision is set first in decimal data types
* Fixed overflow error when converting large numbers to bigint
Fixed error for CHAR datatype missing.
* Fixed NaN issues with Impala Profile
* py formatting
* Fixed warnings from SqlAlchemy
The GenericFunction 'max' is already registered and is going to be overridden.
The GenericFunction 'min' is already registered and is going to be overridden.
Updated Min/Max to handle strings by getting they length.
* Updated profiler to handle strings by using the string length as the parameter to compute the profile
* py_format updates
* fix: ran linting
* fix: Mysql hardcoded table alias
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
