Enrico Minack
b0435fd808
test(CI): show test results on GitHub ( #3362 )
2021-10-25 22:21:56 -07:00
Gabe Lyons
aeba0e9afe
Revert "revert "test(ElasticSearch): Retry ES requests" ( #3385 )" ( #3392 )
...
This reverts commit b16b090f2bf78e0a751490310b95db3dfc142bbf.
2021-10-13 18:53:47 -07:00
Gabe Lyons
b16b090f2b
revert "test(ElasticSearch): Retry ES requests" ( #3385 )
2021-10-13 14:37:51 -07:00
Swaroop Jagadish
146e79b9c3
fix(ingest): fixing support for kafka key schemas when only key schemas are present ( #3379 )
2021-10-12 18:50:39 -07:00
Enrico Minack
104b331a2e
test(ci): timeout build job ( #3364 )
2021-10-11 22:50:27 -07:00
Dexter Lee
4c038d7cfe
fix(git-ignore): Git ignore generated python and avro artifacts ( #3320 )
2021-10-06 11:54:30 -07:00
Enrico Minack
25e1bb1b5a
ci: upload test reports ( #3201 )
2021-09-07 09:30:13 -07:00
John Joyce
2c5edd88ab
feat(graphql): migrating GraphQL API to metadata-service (nee GMS) ( #3131 )
2021-08-20 10:58:07 -07:00
Dexter Lee
801e39bc40
feat(workflow): Add mysql/postgres setup workflow ( #3090 )
2021-08-12 15:56:48 -07:00
Harshal Sheth
328b098d01
text(ingestion): test multiple python versions in CI ( #2952 )
2021-07-26 13:25:58 -07:00
Harshal Sheth
d7248fd08d
build(ingest): separate metadata-ingestion build workflow fully ( #2862 )
2021-07-13 09:13:13 -07:00
Harshal Sheth
1b4788537d
feat(ci): separate metadata-ingestion into a separate workflow ( #2828 )
2021-07-02 20:26:00 -07:00
John Joyce
7591c8994a
feat(datahub cli): DataHub CLI Quickstart ( #2689 )
2021-06-14 17:15:24 -07:00
Kevin Hu
ebdaa0e359
feat(ingest): Feast ingestion integration ( #2605 )
...
* Add feast testing setup
* Init Feast test script
* Add feast to dependencies
* Update feast descriptors
* Sort integrations
* Working feast pytest
* Clean up feast docker-compose file
* Expand Feast tests
* Setup feast classes
* Add continuous and bytes data to feature types
* Update field type mapping
* Add PDLs
* Add MLFeatureSetUrn.java
* Comment out feast setup
* Add snapshot file and update inits
* Init Feast golden files generation
* Clean up Feast ingest
* Feast testing comments
* Yield Feature snapshots
* Fix Feature URN naming
* Update feast MCE
* Update Feature URN prefix
* Add MLEntity
* Update golden files with entities
* Specify feast sources
* Add feast source configs
* Working feast docker ingestion
* List entities and features before adding tables
* Add featureset names
* Remove unused
* Rename feast image
* Update README
* Add env to feast URNs
* Fix URN naming
* Remove redundant URN names
* Fix enum backcompatibility
* Move feast testing to docker
* Move URN generators to mce_builder
* Add source for features
* Switch TypeClass -> enum_type
* Rename source -> sourceDataset
* Add local Feast ingest image builds
* Rename Entity -> MLPrimaryKey
* Restore features and keys for each featureset
* Do not json encode source configs
* Remove old source properties from feature sets
* Regenerate golden file
* Fix race condition with Feast tests
* Exclude unknown source
* Update feature datatype enum
* Update README and fix typos
* Fix Entity typo
* Fix path to local docker image
* Specify feast config and version
* Fix feast env variables
* PR fixes
* Refactor feast ingest constants
* Make feature sources optional for back-compatibility
* Remove unused GCP files
* adding docker publish workflow
* Simplify name+namespace in PrimaryKeys
* adding docker publish workflow
* debug
* final attempt
* final final attempt
* final final final commit
* Switch to published ingestion image
* Update name and namespace in java files
* Rename FeatureSet -> FeatureTable
* Regenerate codegen
* Fix initial generation errors
* Update snapshot jsons
* Regenerated schemas
* Fix URN formats
* Revise builds
* Clean up feast URN builders
* Fix naming typos
* Fix Feature Set -> Feature Table
* Fix comments
* PR fixes
* All you need is Urn
* Regenerate snapshots and update validation
* Add UNKNOWN data type
* URNs for source types
* Add note on docker requirement
* Fix typo
* Reorder aspect unions
* Refactor feast ingest functions
* Update snapshot jsons
* Rebuild
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-09 15:07:04 -07:00
Harshal Sheth
2f7b4f9def
build(docker): test docker builds in pull request CI ( #2661 )
2021-06-08 09:47:32 -07:00
John Joyce
97e9660037
feat: No Code Metadata Modeling ( #2629 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-06-03 13:24:33 -07:00
Dexter Lee
9bff4e1cc4
fix(ci): update trigger to always generate docker images ( #2636 )
2021-06-03 11:02:59 -07:00
Harshal Sheth
a8ced6525a
ci(docker): disable GitHub Docker registry ( #2547 )
2021-05-12 17:53:20 -07:00
Harshal Sheth
d0ca3191c9
build(ingest): add metadata-ingestion to gradle build ( #2510 )
2021-05-06 22:10:49 -07:00
Harshal Sheth
5a9dbd7aa9
build: ensure files are not changed in CI ( #2480 )
2021-04-30 21:01:34 -07:00
Harshal Sheth
201ffd4979
test: add smoke test ( #2464 )
2021-04-29 23:27:03 -07:00
Dexter Lee
1b3a1459e3
fix(kafka-setup): Add the correct context to the git workflow for pushing kafka-setup image ( #2414 )
2021-04-16 17:51:43 -07:00
Harshal Sheth
478d232d2f
build: remove deprecated ember app from build ( #2328 )
2021-04-01 12:16:47 -07:00
John Joyce
728a742528
refactor(react): React as the default UI ( #2252 )
...
Co-authored-by: John Joyce <john@acryl.io>
2021-03-18 09:55:05 -07:00
Harshal Sheth
c015cf7ca9
feat(docs): use gradle for building docs ( #2239 )
2021-03-15 16:13:07 -07:00
Dexter Lee
fd183919cf
fix(ci): rename file to match git workflow needs ( #2222 )
2021-03-11 11:38:25 -08:00
Dexter Lee
07c9a33335
fix(ci): remove paths_ignore from workflow files ( #2220 )
2021-03-11 11:31:21 -08:00
Dexter Lee
e62386f0cd
fix(es-setup): Add git workflows to upload docker for elasticsearch and kafka setup ( #2216 )
2021-03-11 10:30:03 -08:00
Dexter Lee
ce075236a4
fix(ci): Fix github package path ( #2205 )
2021-03-10 10:48:59 -08:00
Dexter Lee
b02039c3da
feat(ci): Add SHA based tagging before pushing to docker registries ( #2204 )
2021-03-10 10:33:27 -08:00
Harshal Sheth
b8b023e131
build(docs): only deploy docs on main repo ( #2175 )
2021-03-05 00:54:20 -08:00
Harshal Sheth
20bf794ec4
docs: hosted documentation website ( #2174 )
2021-03-05 00:12:12 -08:00
Harshal Sheth
9e73794022
ci(ingest): setup docker container for metadata ingestion ( #2150 )
2021-03-01 17:36:38 -08:00
John Joyce
1aede64465
bug(docker react): Fix react docker image build ( #2118 )
...
Co-authored-by: John Joyce <john@acryl.io>
2021-02-17 23:11:32 -08:00
Harshal Sheth
347148b79b
Update python workflow
2021-02-15 18:29:27 -08:00
Harshal Sheth
d0bc3c55db
Setup CI
2021-02-15 18:29:27 -08:00
John Joyce
54346b2e09
Fix build args
2021-02-15 17:40:41 -08:00
John Joyce
ef58e3ca43
rename
2021-02-15 17:40:41 -08:00
John Joyce
b6b13e88c2
Adding path
2021-02-15 17:40:41 -08:00
John Joyce
11f030b118
Adding Github Action to publish Docker image
2021-02-15 17:40:41 -08:00
John Joyce
98bfb9ac8e
Dockerization
2021-02-15 17:40:41 -08:00
Mars Lan
a897f7a3bf
build: add failure notification on push ( #1873 )
...
* build: add failure notification on push
* build: fix build (#1872 )
* build: fix build
`kafkaSerializers` is already included transitively via `kafkaAvroSerde`
* Update build.gradle
2020-09-21 10:29:24 -07:00
John Plaisted
5d394c6062
build: fix docker actions. ( #1787 )
...
* bug: Fix docker actions.
We renamed directories in docker/ which broke the actions.
Also try to refactor the action files a little so that we can run (but not publish) these images on pull requests that change the docker/ dir as an extra check. Note this only seems to be supported by the dockerhub plugin; the github plugin doesn't support this (so that will be an issue when we move to it only).
* Drop extra pipes
2020-08-07 19:15:09 -07:00
John Plaisted
b8e18b0b5d
refactor(docker): make docker files easier to use during development. ( #1777 )
...
* Make docker files easier to use during development.
During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support.
Changes made to docker files:
- Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides.
- Remove redundant README files that provided little information.
- Rename docker/<dir> to match the service name in the docker-compose file for clarity.
- Move environment variables to .env files. We only provide dev / the default environment for quickstart.
- Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead.
- Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image).
- Added docs/docker documentation for this.
2020-08-06 16:38:53 -07:00
Mars Lan
03f1d78ef8
build(docker): build & publish GitHub Package ( #1771 )
...
* build(docker): build & publish docker images to GitHub Packages
Will kepp publishing to Docker Hub meanwhile until all Dockerfiles have been updated to point to GitHub.
Fixes https://github.com/linkedin/datahub/issues/1548
* Rebase & fix dockerfile locations
2020-08-03 13:08:05 -07:00
John Plaisted
a60d650787
build: remove travis (we're using GitHub actions). ( #1770 )
...
Remove travis (we're using GitHub actions).
Also ignore markdown in our current workflows.
Also update the README.md badge.
2020-08-03 12:00:28 -07:00
John Plaisted
c56936b9ff
Add workflow to check build and tests on PRs + releases. ( #1769 )
...
PRs are setup to skip docs.
Also, only run docker actions on linkedin/datahub (i.e. disable on forks; makes forks nicer since you don't have failing actions).
2020-08-01 14:54:32 -07:00
Mars Lan
63943a1ca8
build: update workflows to build version-tagged docker images upon new release
2020-05-29 20:30:44 -07:00
Mars Lan
5cf27e7d08
Update docker-release.yml
2020-05-29 18:42:20 -07:00
Mars Lan
3c1825dab2
Update docker-release.yml
2020-05-29 18:36:07 -07:00
