Mars Lan
15de751c38
Drop the job name from stdout & stderr files as they're now placed in job-execution-specific folder ( #883 )
2017-11-28 14:30:18 -08:00
Shridhar Sattur
f85bc8a081
Updated jobs documentation to include apache common changes. ( #865 )
...
* Updated jobs documentation to include apache common changes.
* Update README.md
2017-11-15 15:57:43 -08:00
Na Zhang
04664a8fec
fix a type in application conf to allow run java
2017-11-09 15:42:05 -08:00
Mars Lan
bf16a411a8
Make the ETL java command configurable ( #845 )
2017-11-07 15:54:05 -08:00
na zhang
54237ff5ca
support elasticsearch auto re-index with zero downtime via alias switch
2017-10-25 15:50:03 -07:00
Shridhar Sattur
4c3aaf5305
Updating ETL config to incorporate unify logging. ( #805 )
...
* Updating ETL config to incorporate unify logging.
* Update EtlJobActor.java
* Update EtlJobActor.java
2017-10-20 10:22:30 -07:00
Mars Lan
98a3a0ea4f
Move various constants into Constant class ( #780 )
2017-10-03 14:52:31 -07:00
Yi (Alan) Wang
3a16b6e280
Refactor BaseDao to add generic functions ( #749 )
2017-09-14 15:41:44 -07:00
Mars Lan
9f7341f542
Clean up the log4j vs slf4j mess and consolidate the dependency resolution into a gradle script ( #745 )
2017-09-13 15:37:29 -07:00
Mars Lan
5127b90ded
Remove the unused backend templates ( #742 )
2017-09-13 11:16:45 -07:00
Yi (Alan) Wang
f015d66fd2
Remove obsolete kafka code in wherehows-common ( #739 )
2017-09-11 18:13:31 -07:00
Mars Lan
082e882268
Remove now-obsolete Kafka-related code from wherehows-backend ( #738 )
2017-09-11 17:49:19 -07:00
Yi (Alan) Wang
e0e2acf6bf
Move JobsUtil to WH common, fix some tests ( #730 )
2017-09-06 11:45:52 -07:00
Yi (Alan) Wang
cd5f72b7ee
Rename DAO and model into table and view sub-package ( #722 )
2017-09-05 09:59:35 -07:00
Yi (Alan) Wang
8084e35303
Modify HIVE extract, disable schema fetching from HDFS, add DB reconnect, refactor code ( #718 )
2017-08-30 17:34:31 -07:00
hzhang2
2ca851753a
remove backend changes for this PR
2017-08-22 22:35:27 -07:00
Mars Lan
03ba2bc85e
Move license header to wherehows-docs & update documentation ( #698 )
...
Fix license checking for Play-based projects and fix violated files
2017-08-22 21:32:53 -07:00
Mars Lan
bf5448d561
Replace db.id & app.id property keys with the existing job.ref.id key ( #695 )
2017-08-22 17:39:10 -07:00
wenhuaOpenx
f7ec09e19a
Add etl job to extract/load druid metadata ( #680 )
...
* test
* test
* add druid etl code
* remove comments
* remove comments
* remove test doc"
* add job template for DRUID_METADATA_ETL
* add druid metadata etl configs to local_test.properties.template
* refactor logger
* remove comments
* fix typos
* add unit test for druid metadata etl job
* refactor unit test code
* import testgn package
* import new package
* reformat the druid etl code based on LinkedIn code style
* add README for druid metadata etl
* add README for druid metadata etl
* add README for druid metadata etl
2017-08-21 16:34:49 -07:00
Yi (Alan) Wang
28b83b8e7b
Add BaseJob in wherehows-common, make ETLjob extends from it ( #681 )
2017-08-16 21:24:38 -07:00
Na Zhang
ca34cd920d
ump metric ETL
2017-08-16 16:43:46 -07:00
hzhang2
4eabd773d2
skip empty classify result DB insertion ( #668 )
2017-08-14 15:09:38 -07:00
Yi (Alan) Wang
3479180ca3
Fix DAO Factory config name error, refactor some code ( #658 )
2017-08-10 18:13:59 -07:00
Mars Lan
4b4cae2148
Add base class for all pure jython ETLs ( #663 )
2017-08-10 17:40:19 -07:00
hzhang2
2215c32e19
correct syntax in application.conf and correct default value for mysql5dialect ( #644 )
2017-08-08 10:56:28 -07:00
hzhang2
7e3d4c220b
fix hibernate connection pool only support mysql issue ( #641 )
2017-08-07 17:26:37 -07:00
Shridhar Sattur
aa28eba78a
Added an end point in wherehows-frontend to display all the libs from container directory ( #639 )
2017-08-07 13:55:05 -07:00
Andrew Park
8ec3113a33
Changed service port number for frontend and backend ( #638 )
2017-08-03 15:40:34 -07:00
Mars Lan
6447cb4538
Update LDAP_USER_ETL.job
2017-08-03 15:20:26 -07:00
hzhang2
c8dcd2ef83
submit hibernate cfg ( #636 )
2017-08-03 15:04:56 -07:00
Andrew Park
830ad7d537
Updated a readme instruction step and synced up all the readmes ( #635 )
2017-08-03 10:53:37 -07:00
Andrew Park
70aba2df45
Add application environment variables & husky error on Mac ( #626 )
...
* Fixed Tracking and run cli
- added piwik directives
- added application.env for frontend specific
* adding frontend env file
* added LDAP config variables in frontend env
* revert back web package
* added readme troubleshooting
* fixed PR review feedbacks
2017-07-28 14:51:50 -07:00
Yi (Alan) Wang
d52edaf3e1
Allow ETL jobs to be disabled at run time ( #622 )
2017-07-26 21:13:56 -07:00
Mars Lan
2967489f18
Update ETL docs.
2017-07-26 16:32:44 -07:00
Mars Lan
fd8b0eac4c
Update README.md
2017-07-26 14:46:27 -07:00
hangzhang925
5a6603596e
update getting-started link in README doc ( #609 )
2017-07-21 16:09:50 -07:00
Yi (Alan) Wang
6a8fe66ef9
Add active, deprecated flag to dataset, modify ETL to update is_active ( #604 )
2017-07-19 17:07:28 -07:00
Mars Lan
c025ba8494
Add template job config files. ( #597 )
2017-07-17 17:58:17 -07:00
Mars Lan
ec0b5ac309
Fix and significatly simplify the docker build process. ( #593 )
2017-07-13 09:39:59 -07:00
Mars Lan
bb5f483be9
Clean up comments and DDL for obsolete tables. ( #586 )
2017-07-10 13:44:35 -07:00
Mars Lan
deb98480a3
Completely retire wh_property table and associated codes. ( #583 )
2017-07-10 13:44:35 -07:00
Mars Lan
0e5e5685d7
Do not automatically kick off all jobs that yet have an entry in wh_etl_job_schedule table. ( #582 )
...
* Do not automatically kick off all jobs that yet have an entry in wh_etl_job_schedule table.
Instead simply schedule an entry and wait for its time to come.
* Make the ETL actor pool size configurable and set the default to something more conservative.
2017-07-10 13:44:35 -07:00
Mars Lan
770ac152e9
Refactor tree jobs to use job files. ( #576 )
2017-07-10 13:44:35 -07:00
Mars Lan
3f85b26d81
Drop the duplicated DB-related settings from wh_property table. ( #575 )
2017-07-10 13:44:35 -07:00
Mars Lan
75f57c6ac0
Remove some of the LI-speicifc ETLs. ( #559 )
2017-07-10 13:44:33 -07:00
Yi (Alan) Wang
5808179017
Increase play http parser buffer size to 10MB ( #555 )
2017-07-10 13:44:33 -07:00
Mars Lan
53a30d5a77
Major refactoring of ETL scheduling & configuration ( #542 )
...
* Major refactoring
- Move job-spcific properties from wh_etl_job_property table to .job files
- Use the job file name instead of numeric IDs to identify ETL jobs
- Use reflection to create ETL job class at run time instead of relying hard-coded enums
- Drop ETL job-related APIs as they're no longer needed
- Drop wh_etl_job, wh_etl_job_property, wh_etl_job_execution tables
- Add wh_etl_schedule & wh_etl_history tables
2017-07-10 13:44:33 -07:00
Mars Lan
30f9bf28b2
Filter out HDFS datasets with "temporary" in path name. ( #537 )
...
Treat "_" as a word boundary.
2017-07-10 13:43:00 -07:00
Mars Lan
8411e220a4
Actually write properties to dict_dataset table. ( #529 )
...
Temporariliy disable the broken updateKafkaDatasetOwner call.
Add extra blacklist pattern.
2017-07-10 13:43:00 -07:00
Mars Lan
b486f7ee38
Extract and set dataset name correctly from Gobblin metadata audit event. ( #526 )
2017-07-10 13:42:59 -07:00
