Harshal Sheth
eaac303610
Reduce mypy errors
2021-02-15 18:29:27 -08:00
Harshal Sheth
40b8e5e158
Fixes in pipeline
2021-02-15 18:29:27 -08:00
Harshal Sheth
db1ff07e6d
Add reporter everywhere
2021-02-15 18:29:27 -08:00
Harshal Sheth
fd9bc09e67
Start adding reporting
2021-02-15 18:29:27 -08:00
Harshal Sheth
0929c7cb77
Add test to check that classes are not abstract
2021-02-15 18:29:27 -08:00
Harshal Sheth
9ea03820c4
Enable bare pytest
2021-02-15 18:29:27 -08:00
Shirshanka Das
063f513997
Update README.md
2021-02-15 18:29:27 -08:00
Shirshanka Das
4e17a24dd4
Adding mysql integration test + source
2021-02-15 18:29:27 -08:00
Shirshanka Das
e03a9e25f8
Update README.md
2021-02-15 18:29:27 -08:00
Harshal Sheth
22a4dbb45f
Cleanup kafka sink schema setup
2021-02-15 18:29:27 -08:00
Shirshanka Das
d5f89f8170
adding commented out example to yaml file
2021-02-15 18:29:27 -08:00
Harshal Sheth
d68648670a
Add back line that I accidentally removed when testing
2021-02-15 18:29:27 -08:00
Harshal Sheth
4b83fc6591
adding allow deny patterns to sql config
2021-02-15 18:29:27 -08:00
Harshal Sheth
62bb7f012f
Quick readme updates
2021-02-15 18:29:27 -08:00
Harshal Sheth
9d741815c5
Also run unit tests in CI
2021-02-15 18:29:27 -08:00
Harshal Sheth
e09528909f
python 3.7 compat
2021-02-15 18:29:27 -08:00
Shirshanka Das
9c6ea3b9e9
Adding 3.9 to setup.py classifiers
2021-02-15 18:29:27 -08:00
Shirshanka Das
024745bcc4
removing py 3.6, adding 3.9 for ci
2021-02-15 18:29:27 -08:00
Shirshanka Das
7b12fc9827
support for multiple mce-s in a file
2021-02-15 18:29:27 -08:00
Harshal Sheth
58554725aa
Add bootstrap mce
2021-02-15 18:29:27 -08:00
Harshal Sheth
4dc8936f27
Add mypy type checks
2021-02-15 18:29:27 -08:00
Shirshanka Das
5a8bb3cfac
adding docker commands
2021-02-15 18:29:27 -08:00
Shirshanka Das
863cc97ea3
simplify docker action
2021-02-15 18:29:27 -08:00
Shirshanka Das
1c05c1b23a
simplify docker action
2021-02-15 18:29:27 -08:00
Shirshanka Das
e80ebf180c
simplify docker action
2021-02-15 18:29:27 -08:00
Shirshanka Das
76a9564a6f
Adding github actions to build and push docker image
2021-02-15 18:29:27 -08:00
Shirshanka Das
8d74f97713
adding a file-to-file recipe
2021-02-15 18:29:27 -08:00
Shirshanka Das
327705fd0d
MS-SQL integ test works
2021-02-15 18:29:27 -08:00
Shirshanka Das
d73d8ff4c2
fixing unit tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
c7892ada4c
Codegen avro + datahub kafka sink ( #3 )
...
* Add codegen
* New architecture + setup file -> console pipeline
* Cleanup source loader
* Basic Kafka metadata source
* Kafka source and extractor
* Add kwargs construct interface
* Fix kafka source unit test
* start working on pipeline test
* kafka datahub sink
* Make myself a profile
* Ingest to datahub from kafka
* Update codegen
* Add restli transport
* Fix bug in restli conversion
2021-02-15 18:29:27 -08:00
Shirshanka Das
b59a62fa1c
setting modest coverage targets
2021-02-15 18:29:27 -08:00
Shirshanka Das
35e9f28b56
dropping unit tests
2021-02-15 18:29:27 -08:00
Shirshanka Das
60d861b498
dropping hard failures
2021-02-15 18:29:27 -08:00
Shirshanka Das
5e589da514
adding python action
2021-02-15 18:29:27 -08:00
Shirshanka Das
6b5bbbdc5f
workaround for docker exec, waiting for 5 more seconds
2021-02-15 18:29:27 -08:00
Harshal Sheth
4fb673925c
Start using avro producer
2021-02-15 18:29:27 -08:00
Shirshanka Das
9e61220132
checking in testing fixtures. docker still not working
2021-02-15 18:29:27 -08:00
Shirshanka Das
1ddbdee60c
Support for SQL databases (MySQL + MS-SQL) ( #2 )
...
* adding sql source + mysql
* adding sql support
* MSSQL support, basic integration test
* file sink and pipeline context
2021-02-15 18:29:27 -08:00
Shirshanka Das
faf472aa64
adding some TODOs
2021-02-15 18:29:27 -08:00
Shirshanka Das
128781942d
Firstdrop of ingest ( #1 )
2021-02-15 18:29:27 -08:00
Shirshanka Das
90b635fb7c
Initial commit
2021-02-15 18:29:27 -08:00
Harshal Sheth
082c86463e
Move old metadata ingestion scripts out of the way
2021-02-15 18:29:27 -08:00
Harshal Sheth
b491e4ad3c
fix(SQL ingest): Bump confluent-kafka version ( #2082 )
...
This should help resolve #2079 .
2021-02-03 18:06:35 -08:00
Mars Lan
7a786c185b
Drop obsolete info on mysql-etl ( #2072 )
2021-01-29 09:03:53 -08:00
Kerem Sahin
4d8320e4a0
feat(dashboard): Dashboards backend implementation ( #1884 )
2020-11-23 09:25:58 -08:00
Grant Nicholas
fa58c2d161
fix(metadata-ingestion): Fix auditStamp unix timestamp format in sql etl ingestion ( #1918 )
...
Datahub was expecting this timestamp to be in milliseconds since epoch, not seconds. This change makes the lastModified timestamp render correctly in the UI when it is converted to a date time string.
2020-10-06 11:13:02 -07:00
John Plaisted
821bce7d69
feat: Port mce-cli to Java. ( #1871 )
...
Port mce-cli to Java.
Also moved off the avro format event file to json instead. Much nicer to use :)
2020-09-25 14:05:29 -07:00
Charlie Tran
57fdc5c00c
Adds ability for midtier to serve custom dataset properties from aspect ( #1881 )
2020-09-20 11:04:51 -07:00
John Plaisted
6ece2d6469
Start adding java ETL examples, starting with kafka etl. ( #1805 )
...
Start adding java ETL examples, starting with kafka etl.
We've had a few requests to start providing Java examples rather than Python due to type safety.
I've also started to add these to metadata-ingestion-examples to make it clearer these are *examples*. They can be used directly or as a basis for other things.
As we port to Java we'll move examples to contrib.
2020-09-11 13:04:21 -07:00
John Plaisted
23ad0e9c8b
Small fixes to mce_cli ( #1868 )
...
- default argument value should be None not "None"
- Test data should have corpuser, not corpUser (case sensitive)
fixes https://github.com/linkedin/datahub/issues/1867
fixes https://github.com/linkedin/datahub/issues/1865
2020-09-10 19:30:47 -07:00
