yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-01 19:18:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixes 10959: MSSQL profile has error when averaging large integer column (#10960)

Browse Source 
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error
This commit is contained in:
Keith Sirmons 2023-04-06 23:48:18 -05:00 committed by GitHub
parent 7c6b9b3c56
commit 27984c25f3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 10 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
ingestion/src/metadata/profiler/metrics/static/mean.py
View File

@ -41,6 +41,16 @@ def _(element, compiler, **kw):
return f"if(isNaN(avg({proc})), null, avg({proc}))"
@compiles(avg, Dialects.MSSQL)
def _(element, compiler, **kw):
"""
Cast to decimal to get around potential integer overflow error -
Error 8115: Arithmetic overflow error converting expression to data type int.
"""
proc = compiler.process(element.clauses, **kw)
return f"avg(cast({proc} as decimal))"
class Mean(StaticMetric):
"""
AVG Metric