mirror of
https://github.com/WeiYe-Jing/datax-web.git
synced 2026-06-30 21:17:37 +08:00
fantasticKe
GitHub
commit
b97c003e8e
64
README.md
64
README.md
@ -34,7 +34,14 @@ DataX阿里的开源的时候并未提供任何可视化界面，我们在使用
|
||||
- 14、用户管理：支持在线管理系统用户，存在管理员、普通用户两种角色；
|
||||
- 15、任务依赖：支持配置子任务依赖，当父任务执行结束且执行成功后将会主动触发一次子任务的执行, 多个子任务用逗号分隔；
|
||||
- 16、运行报表：支持实时查看运行数据，以及调度报表，如调度日期分布图，调度成功分布图等；
|
||||
|
||||
- 17、指定增量字段，配置定时任务自动获取每次的数据区间，任务失败重试，保证数据安全；
|
||||
- 18、页面可配置DataX启动JVM参数；
|
||||
- 19、数据源配置成功后添加手动测试功能；
|
||||
- 20、可以对常用任务进行配置模板，在构建完JSON之后可选择关联模板创建任务；
|
||||
- 21、jdbc添加hive数据源支持，可在构建JSON页面选择数据源生成column信息并简化配置；
|
||||
- 22、优先通过环境变量获取DataX文件目录，集群部署时不用指定JSON及日志目录；
|
||||
- 23、通过动态参数配置指定hive分区，也可以配合增量实现增量数据动态插入分区；
|
||||
- 24、任务类型由原来DataX任务扩展到Shell任务、Python任务、PowerShell任务；
|
||||
## Quick Start
|
||||
|
||||
### 1. 下载datax打包之后的文件或者github拉取datax代码打包
|
||||
@ -74,19 +81,30 @@ http://localhost:8080/index.html#/dashboard
|
||||
|
||||
|
||||
### 8. 构建JSON脚本
|
||||
JSON构建目前支持的数据源有hive,mysql,oracle,postgresql,sqlserver,其它数据源的JSON构建正在开发中,暂时需要手动编写。
|
||||
|
||||
|
||||
### 9. 创建任务
|
||||
|
||||
|
||||
#### DataX任务
|
||||
|
||||
#### Shell任务
|
||||
|
||||
#### Python任务
|
||||
|
||||
#### PowerShell任务
|
||||
|
||||
|
||||
- 任务类型：目前支持DataX任务、Shell任务、Python任务、PowerShell任务；
|
||||
- 阻塞处理策略：调度过于密集执行器来不及处理时的处理策略；
|
||||
- 单机串行：调度请求进入单机执行器后，调度请求进入FIFO队列并以串行方式运行；
|
||||
- 丢弃后续调度：调度请求进入单机执行器后，发现执行器存在运行的调度任务，本次请求将会被丢弃并标记为失败；
|
||||
- 覆盖之前调度：调度请求进入单机执行器后，发现执行器存在运行的调度任务，将会终止运行中的调度任务并清空队列，然后运行本地调度任务；
|
||||
- 增量增新建议将阻塞策略设置为丢弃后续调度或者单机串行
|
||||
- 设置单机串行时应该注意合理设置重试次数(失败重试的次数*每次执行时间<任务的调度周期)，重试的次数如果设置的过多会导致数据重复，例如任务30秒执行一次，每次执行时间需要20秒，设置重试三次，如果任务失败了，第一个重试的时间段为1577755680-1577756680，重试任务没结束，新任务又开启，那新任务的时间段会是1577755680-1577758680
|
||||
- [增量参数设置](https://github.com/WeiYe-Jing/datax-web/blob/master/doc/datax-web/%E5%8A%A8%E6%80%81%E5%8F%82%E6%95%B0%E5%AE%8C%E6%88%90%E5%A2%9E%E9%87%8F%E6%8A%BD%E5%8F%96.md
|
||||
)
|
||||
- [增量参数设置](https://github.com/WeiYe-Jing/datax-web/blob/master/doc/datax-web/%E5%8A%A8%E6%80%81%E5%8F%82%E6%95%B0%E5%AE%8C%E6%88%90%E5%A2%9E%E9%87%8F%E6%8A%BD%E5%8F%96.md)
|
||||
- [分区参数设置](https://github.com/WeiYe-Jing/datax-web/blob/master/doc/datax-web/%E5%88%86%E5%8C%BA%E5%8A%A8%E6%80%81%E4%BC%A0%E5%8F%82%E4%BD%BF%E7%94%A8.md)
|
||||
|
||||
### 10. 任务列表
|
||||
|
||||
|
||||
@ -106,26 +124,26 @@ Quick Start操作完前四步之后
|
||||
## UI
|
||||
[前端github地址](https://github.com/WeiYe-Jing/datax-vue-admin.git)
|
||||
|
||||
## 完成功能
|
||||
- 修复前端页面自适应问题
|
||||
- 修复添加任务时json转换失败的问题
|
||||
- 修复构建json时数据库连接增加问题
|
||||
- 修复增量更新失败重试成功后增量更新时间跳跃的问题
|
||||
- 修复失败重试时序列化失败问题
|
||||
- 1、指定增量字段，自动获取每次的数据区间
|
||||
- 2、页面配置datax启动参数
|
||||
- 3、数据源连接错误提醒功能
|
||||
- 4、任务模板创建
|
||||
- 5、构建JSON之后选择任务模板创建任务
|
||||
- 6、jdbc添加hive数据源支持
|
||||
- 7、json构建模块代码重构
|
||||
- 8、json构建支持hive
|
||||
- 9、添加数据源测试功能
|
||||
- 10、优先通过环境变量获取DataX文件目录
|
||||
- 11、字段映射优化
|
||||
### Contributing
|
||||
Contributions are welcome! Open a pull request to fix a bug, or open an Issue to discuss a new feature or change.
|
||||
|
||||
欢迎参与项目贡献！比如提交PR修复一个bug，或者新建 Issue 讨论新特性或者变更。
|
||||
|
||||
## Copyright and License
|
||||
This product is open source and free, and will continue to provide free community technical support. Individual or enterprise users are free to access and use.
|
||||
|
||||
- Licensed under the GNU General Public License (GPL) v3.
|
||||
- Copyright (c) 2015-present, xuxueli.
|
||||
|
||||
产品开源免费，并且将持续提供免费的社区技术支持。个人或企业内部可自由的接入和使用。
|
||||
|
||||
> 欢迎在 [登记地址](https://github.com/WeiYe-Jing/datax-web/issues/14 ) 登记，登记仅仅为了产品推广和提升社区开发的动力。
|
||||
>
|
||||
## TODO List
|
||||
- 1、从源表到目标端表的自动创建
|
||||
- 2、任务批量导入功能
|
||||
- 1、对接DataX支持的数据源，简化json构建
|
||||
- 2、从源表到目标端表的自动创建
|
||||
- 3、任务批量导入功能
|
||||
|
||||
|
||||
|
||||
## Contact us
|
||||
|
||||
@ -5,7 +5,7 @@
|
||||
<parent>
|
||||
<groupId>com.wugui</groupId>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
<artifactId>datax-admin</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
@ -2,11 +2,13 @@ package com.wugui.datax.admin.controller;
|
||||
|
||||
import com.baomidou.mybatisplus.extension.api.ApiController;
|
||||
import com.baomidou.mybatisplus.extension.api.R;
|
||||
import com.wugui.datax.admin.core.util.I18nUtil;
|
||||
import com.wugui.datax.admin.dto.DataxJsonDto;
|
||||
import com.wugui.datax.admin.service.DataxJsonService;
|
||||
import io.swagger.annotations.Api;
|
||||
import io.swagger.annotations.ApiOperation;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.util.CollectionUtils;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
@ -32,6 +34,18 @@ public class DataxJsonController extends ApiController {
|
||||
@PostMapping("/buildJson")
|
||||
@ApiOperation("JSON构建")
|
||||
public R<String> buildJobJson(@RequestBody DataxJsonDto dataxJsonDto) {
|
||||
if (dataxJsonDto.getReaderDatasourceId()==null) {
|
||||
return failed(I18nUtil.getString("system_please_choose")+I18nUtil.getString("jobinfo_field_readerDataSource"));
|
||||
}
|
||||
if (dataxJsonDto.getWriterDatasourceId()==null) {
|
||||
return failed(I18nUtil.getString("system_please_choose")+I18nUtil.getString("jobinfo_field_writerDataSource"));
|
||||
}
|
||||
if (CollectionUtils.isEmpty(dataxJsonDto.getReaderColumns())) {
|
||||
return failed(I18nUtil.getString("system_please_choose")+I18nUtil.getString("jobinfo_field_readerColumns"));
|
||||
}
|
||||
if (CollectionUtils.isEmpty(dataxJsonDto.getWriterColumns())) {
|
||||
return failed(I18nUtil.getString("system_please_choose")+I18nUtil.getString("jobinfo_field_writerColumns"));
|
||||
}
|
||||
return success(dataxJsonService.buildJobJson(dataxJsonDto));
|
||||
}
|
||||
|
||||
|
||||
@ -54,15 +54,15 @@ public class JobCodeController {
|
||||
jobInfoMapper.update(exists_jobInfo);
|
||||
|
||||
// log old code
|
||||
JobLogGlue xxlJobLogGlue = new JobLogGlue();
|
||||
xxlJobLogGlue.setJobId(exists_jobInfo.getId());
|
||||
xxlJobLogGlue.setGlueType(exists_jobInfo.getGlueType());
|
||||
xxlJobLogGlue.setGlueSource(glueSource);
|
||||
xxlJobLogGlue.setGlueRemark(glueRemark);
|
||||
JobLogGlue jobLogGlue = new JobLogGlue();
|
||||
jobLogGlue.setJobId(exists_jobInfo.getId());
|
||||
jobLogGlue.setGlueType(exists_jobInfo.getGlueType());
|
||||
jobLogGlue.setGlueSource(glueSource);
|
||||
jobLogGlue.setGlueRemark(glueRemark);
|
||||
|
||||
xxlJobLogGlue.setAddTime(new Date());
|
||||
xxlJobLogGlue.setUpdateTime(new Date());
|
||||
jobLogGlueMapper.save(xxlJobLogGlue);
|
||||
jobLogGlue.setAddTime(new Date());
|
||||
jobLogGlue.setUpdateTime(new Date());
|
||||
jobLogGlueMapper.save(jobLogGlue);
|
||||
|
||||
// remove code backup more than 30
|
||||
jobLogGlueMapper.removeOld(exists_jobInfo.getId(), 30);
|
||||
|
||||
@ -39,23 +39,23 @@ public class JobGroupController {
|
||||
|
||||
@PostMapping("/save")
|
||||
@ApiOperation("新建执行器")
|
||||
public ReturnT<String> save(@RequestBody JobGroup xxlJobGroup){
|
||||
public ReturnT<String> save(@RequestBody JobGroup jobGroup){
|
||||
|
||||
// valid
|
||||
if (xxlJobGroup.getAppName()==null || xxlJobGroup.getAppName().trim().length()==0) {
|
||||
if (jobGroup.getAppName()==null || jobGroup.getAppName().trim().length()==0) {
|
||||
return new ReturnT<String>(500, (I18nUtil.getString("system_please_input")+"AppName") );
|
||||
}
|
||||
if (xxlJobGroup.getAppName().length()<4 || xxlJobGroup.getAppName().length()>64) {
|
||||
if (jobGroup.getAppName().length()<4 || jobGroup.getAppName().length()>64) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_appName_length") );
|
||||
}
|
||||
if (xxlJobGroup.getTitle()==null || xxlJobGroup.getTitle().trim().length()==0) {
|
||||
if (jobGroup.getTitle()==null || jobGroup.getTitle().trim().length()==0) {
|
||||
return new ReturnT<String>(500, (I18nUtil.getString("system_please_input") + I18nUtil.getString("jobgroup_field_title")) );
|
||||
}
|
||||
if (xxlJobGroup.getAddressType()!=0) {
|
||||
if (xxlJobGroup.getAddressList()==null || xxlJobGroup.getAddressList().trim().length()==0) {
|
||||
if (jobGroup.getAddressType()!=0) {
|
||||
if (jobGroup.getAddressList()==null || jobGroup.getAddressList().trim().length()==0) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_addressType_limit") );
|
||||
}
|
||||
String[] addresss = xxlJobGroup.getAddressList().split(",");
|
||||
String[] addresss = jobGroup.getAddressList().split(",");
|
||||
for (String item: addresss) {
|
||||
if (item==null || item.trim().length()==0) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_registryList_unvalid") );
|
||||
@ -63,26 +63,26 @@ public class JobGroupController {
|
||||
}
|
||||
}
|
||||
|
||||
int ret = jobGroupMapper.save(xxlJobGroup);
|
||||
int ret = jobGroupMapper.save(jobGroup);
|
||||
return (ret>0)?ReturnT.SUCCESS:ReturnT.FAIL;
|
||||
}
|
||||
|
||||
@PostMapping("/update")
|
||||
@ApiOperation("更新执行器")
|
||||
public ReturnT<String> update(@RequestBody JobGroup xxlJobGroup){
|
||||
public ReturnT<String> update(@RequestBody JobGroup jobGroup){
|
||||
// valid
|
||||
if (xxlJobGroup.getAppName()==null || xxlJobGroup.getAppName().trim().length()==0) {
|
||||
if (jobGroup.getAppName()==null || jobGroup.getAppName().trim().length()==0) {
|
||||
return new ReturnT<String>(500, (I18nUtil.getString("system_please_input")+"AppName") );
|
||||
}
|
||||
if (xxlJobGroup.getAppName().length()<4 || xxlJobGroup.getAppName().length()>64) {
|
||||
if (jobGroup.getAppName().length()<4 || jobGroup.getAppName().length()>64) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_appName_length") );
|
||||
}
|
||||
if (xxlJobGroup.getTitle()==null || xxlJobGroup.getTitle().trim().length()==0) {
|
||||
if (jobGroup.getTitle()==null || jobGroup.getTitle().trim().length()==0) {
|
||||
return new ReturnT<String>(500, (I18nUtil.getString("system_please_input") + I18nUtil.getString("jobgroup_field_title")) );
|
||||
}
|
||||
if (xxlJobGroup.getAddressType() == 0) {
|
||||
if (jobGroup.getAddressType() == 0) {
|
||||
// 0=自动注册
|
||||
List<String> registryList = findRegistryByAppName(xxlJobGroup.getAppName());
|
||||
List<String> registryList = findRegistryByAppName(jobGroup.getAppName());
|
||||
String addressListStr = null;
|
||||
if (registryList!=null && !registryList.isEmpty()) {
|
||||
Collections.sort(registryList);
|
||||
@ -92,13 +92,13 @@ public class JobGroupController {
|
||||
}
|
||||
addressListStr = addressListStr.substring(0, addressListStr.length()-1);
|
||||
}
|
||||
xxlJobGroup.setAddressList(addressListStr);
|
||||
jobGroup.setAddressList(addressListStr);
|
||||
} else {
|
||||
// 1=手动录入
|
||||
if (xxlJobGroup.getAddressList()==null || xxlJobGroup.getAddressList().trim().length()==0) {
|
||||
if (jobGroup.getAddressList()==null || jobGroup.getAddressList().trim().length()==0) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_addressType_limit") );
|
||||
}
|
||||
String[] addresss = xxlJobGroup.getAddressList().split(",");
|
||||
String[] addresss = jobGroup.getAddressList().split(",");
|
||||
for (String item: addresss) {
|
||||
if (item==null || item.trim().length()==0) {
|
||||
return new ReturnT<String>(500, I18nUtil.getString("jobgroup_field_registryList_unvalid") );
|
||||
@ -106,7 +106,7 @@ public class JobGroupController {
|
||||
}
|
||||
}
|
||||
|
||||
int ret = jobGroupMapper.update(xxlJobGroup);
|
||||
int ret = jobGroupMapper.update(jobGroup);
|
||||
return (ret>0)?ReturnT.SUCCESS:ReturnT.FAIL;
|
||||
}
|
||||
|
||||
|
||||
@ -35,7 +35,7 @@ import java.util.Map;
|
||||
public class JobInfoController {
|
||||
|
||||
@Resource
|
||||
private JobService xxlJobService;
|
||||
private JobService jobService;
|
||||
|
||||
|
||||
public static void validPermission(HttpServletRequest request, int jobGroup) {
|
||||
@ -49,39 +49,39 @@ public class JobInfoController {
|
||||
@ApiOperation("任务列表")
|
||||
public ReturnT<Map<String, Object>> pageList(@RequestParam(required = false, defaultValue = "0") int current,
|
||||
@RequestParam(required = false, defaultValue = "10") int size,
|
||||
int jobGroup, int triggerStatus, String jobDesc, String executorHandler, String author) {
|
||||
int jobGroup, int triggerStatus, String jobDesc, String glueType, String author) {
|
||||
|
||||
return new ReturnT<>(xxlJobService.pageList((current-1)*size, size, jobGroup, triggerStatus, jobDesc, executorHandler, author));
|
||||
return new ReturnT<>(jobService.pageList((current-1)*size, size, jobGroup, triggerStatus, jobDesc, glueType, author));
|
||||
}
|
||||
|
||||
@PostMapping("/add")
|
||||
@ApiOperation("添加任务")
|
||||
public ReturnT<String> add(@RequestBody JobInfo jobInfo) {
|
||||
return xxlJobService.add(jobInfo);
|
||||
return jobService.add(jobInfo);
|
||||
}
|
||||
|
||||
@PostMapping("/update")
|
||||
@ApiOperation("更新任务")
|
||||
public ReturnT<String> update(@RequestBody JobInfo jobInfo) {
|
||||
return xxlJobService.update(jobInfo);
|
||||
return jobService.update(jobInfo);
|
||||
}
|
||||
|
||||
@PostMapping(value = "/remove/{id}")
|
||||
@ApiOperation("移除任务")
|
||||
public ReturnT<String> remove(@PathVariable(value = "id") int id) {
|
||||
return xxlJobService.remove(id);
|
||||
return jobService.remove(id);
|
||||
}
|
||||
|
||||
@RequestMapping(value = "/stop",method = RequestMethod.POST)
|
||||
@ApiOperation("停止任务")
|
||||
public ReturnT<String> pause(int id) {
|
||||
return xxlJobService.stop(id);
|
||||
return jobService.stop(id);
|
||||
}
|
||||
|
||||
@RequestMapping(value = "/start",method = RequestMethod.POST)
|
||||
@ApiOperation("开启任务")
|
||||
public ReturnT<String> start(int id) {
|
||||
return xxlJobService.start(id);
|
||||
return jobService.start(id);
|
||||
}
|
||||
|
||||
@PostMapping(value = "/trigger")
|
||||
|
||||
@ -50,52 +50,52 @@ public class UserController {
|
||||
|
||||
@PostMapping("/add")
|
||||
@ApiOperation("添加用户")
|
||||
public ReturnT<String> add(@RequestBody JobUser xxlJobUser) {
|
||||
public ReturnT<String> add(@RequestBody JobUser jobUser) {
|
||||
|
||||
// valid username
|
||||
if (!StringUtils.hasText(xxlJobUser.getUsername())) {
|
||||
if (!StringUtils.hasText(jobUser.getUsername())) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_please_input")+I18nUtil.getString("user_username") );
|
||||
}
|
||||
xxlJobUser.setUsername(xxlJobUser.getUsername().trim());
|
||||
if (!(xxlJobUser.getUsername().length()>=4 && xxlJobUser.getUsername().length()<=20)) {
|
||||
jobUser.setUsername(jobUser.getUsername().trim());
|
||||
if (!(jobUser.getUsername().length()>=4 && jobUser.getUsername().length()<=20)) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_lengh_limit")+"[4-20]" );
|
||||
}
|
||||
// valid password
|
||||
if (!StringUtils.hasText(xxlJobUser.getPassword())) {
|
||||
if (!StringUtils.hasText(jobUser.getPassword())) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_please_input")+I18nUtil.getString("user_password") );
|
||||
}
|
||||
xxlJobUser.setPassword(xxlJobUser.getPassword().trim());
|
||||
if (!(xxlJobUser.getPassword().length()>=4 && xxlJobUser.getPassword().length()<=20)) {
|
||||
jobUser.setPassword(jobUser.getPassword().trim());
|
||||
if (!(jobUser.getPassword().length()>=4 && jobUser.getPassword().length()<=20)) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_lengh_limit")+"[4-20]" );
|
||||
}
|
||||
xxlJobUser.setPassword(bCryptPasswordEncoder.encode(xxlJobUser.getPassword()));
|
||||
jobUser.setPassword(bCryptPasswordEncoder.encode(jobUser.getPassword()));
|
||||
|
||||
// check repeat
|
||||
JobUser existUser = jobUserMapper.loadByUserName(xxlJobUser.getUsername());
|
||||
JobUser existUser = jobUserMapper.loadByUserName(jobUser.getUsername());
|
||||
if (existUser != null) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("user_username_repeat") );
|
||||
}
|
||||
|
||||
// write
|
||||
jobUserMapper.save(xxlJobUser);
|
||||
jobUserMapper.save(jobUser);
|
||||
return ReturnT.SUCCESS;
|
||||
}
|
||||
|
||||
@PostMapping(value = "/update")
|
||||
@ApiOperation("更新用户信息")
|
||||
public ReturnT<String> update(@RequestBody JobUser xxlJobUser) {
|
||||
public ReturnT<String> update(@RequestBody JobUser jobUser) {
|
||||
// valid password
|
||||
if (StringUtils.hasText(xxlJobUser.getPassword())) {
|
||||
xxlJobUser.setPassword(xxlJobUser.getPassword().trim());
|
||||
if (!(xxlJobUser.getPassword().length()>=4 && xxlJobUser.getPassword().length()<=20)) {
|
||||
if (StringUtils.hasText(jobUser.getPassword())) {
|
||||
jobUser.setPassword(jobUser.getPassword().trim());
|
||||
if (!(jobUser.getPassword().length()>=4 && jobUser.getPassword().length()<=20)) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_lengh_limit")+"[4-20]" );
|
||||
}
|
||||
xxlJobUser.setPassword(bCryptPasswordEncoder.encode(xxlJobUser.getPassword()));
|
||||
jobUser.setPassword(bCryptPasswordEncoder.encode(jobUser.getPassword()));
|
||||
} else {
|
||||
xxlJobUser.setPassword(null);
|
||||
jobUser.setPassword(null);
|
||||
}
|
||||
// write
|
||||
jobUserMapper.update(xxlJobUser);
|
||||
jobUserMapper.update(jobUser);
|
||||
return ReturnT.SUCCESS;
|
||||
}
|
||||
|
||||
@ -108,8 +108,8 @@ public class UserController {
|
||||
|
||||
@PostMapping(value = "/updatePwd")
|
||||
@ApiOperation("修改密码")
|
||||
public ReturnT<String> updatePwd(@RequestBody JobUser xxlJobUser){
|
||||
String password=xxlJobUser.getPassword();
|
||||
public ReturnT<String> updatePwd(@RequestBody JobUser jobUser){
|
||||
String password=jobUser.getPassword();
|
||||
// valid password
|
||||
if (password==null || password.trim().length()==0){
|
||||
return new ReturnT<String>(ReturnT.FAIL.getCode(), "密码不可为空");
|
||||
@ -119,7 +119,7 @@ public class UserController {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("system_lengh_limit")+"[4-20]" );
|
||||
}
|
||||
// do write
|
||||
JobUser existUser = jobUserMapper.loadByUserName(xxlJobUser.getUsername());
|
||||
JobUser existUser = jobUserMapper.loadByUserName(jobUser.getUsername());
|
||||
existUser.setPassword(bCryptPasswordEncoder.encode(password));
|
||||
jobUserMapper.update(existUser);
|
||||
return ReturnT.SUCCESS;
|
||||
|
||||
@ -130,6 +130,8 @@ public class JobTrigger {
|
||||
triggerParam.setReplaceParam(jobInfo.getReplaceParam());
|
||||
triggerParam.setStartTime(jobInfo.getIncStartTime());
|
||||
triggerParam.setTriggerTime(triggerTime);
|
||||
triggerParam.setPartitionInfo(jobInfo.getPartitionInfo());
|
||||
|
||||
// 3、init address
|
||||
String address = null;
|
||||
ReturnT<String> routeAddressResult = null;
|
||||
|
||||
@ -89,4 +89,6 @@ public class JobInfo {
|
||||
@ApiModelProperty("增量初始时间")
|
||||
private Date incStartTime;
|
||||
|
||||
@ApiModelProperty("分区信息")
|
||||
private String partitionInfo;
|
||||
}
|
||||
|
||||
@ -86,4 +86,6 @@ public class JobTemplate {
|
||||
@ApiModelProperty("增量初始时间")
|
||||
private Date incStartTime;
|
||||
|
||||
@ApiModelProperty("分区信息")
|
||||
private String partitionInfo;
|
||||
}
|
||||
|
||||
@ -20,21 +20,21 @@ public interface JobInfoMapper {
|
||||
@Param("jobGroup") int jobGroup,
|
||||
@Param("triggerStatus") int triggerStatus,
|
||||
@Param("jobDesc") String jobDesc,
|
||||
@Param("executorHandler") String executorHandler,
|
||||
@Param("glueType") String glueType,
|
||||
@Param("author") String author);
|
||||
public int pageListCount(@Param("offset") int offset,
|
||||
@Param("pagesize") int pagesize,
|
||||
@Param("jobGroup") int jobGroup,
|
||||
@Param("triggerStatus") int triggerStatus,
|
||||
@Param("jobDesc") String jobDesc,
|
||||
@Param("executorHandler") String executorHandler,
|
||||
@Param("glueType") String glueType,
|
||||
@Param("author") String author);
|
||||
|
||||
public int save(JobInfo info);
|
||||
|
||||
public JobInfo loadById(@Param("id") int id);
|
||||
|
||||
public int update(JobInfo xxlJobInfo);
|
||||
public int update(JobInfo jobInfo);
|
||||
|
||||
public int delete(@Param("id") long id);
|
||||
|
||||
|
||||
@ -13,7 +13,7 @@ import java.util.List;
|
||||
@Mapper
|
||||
public interface JobLogGlueMapper {
|
||||
|
||||
public int save(JobLogGlue xxlJobLogGlue);
|
||||
public int save(JobLogGlue jobLogGlue);
|
||||
|
||||
public List<JobLogGlue> findByJobId(@Param("jobId") int jobId);
|
||||
|
||||
|
||||
@ -20,11 +20,11 @@ public interface JobService {
|
||||
* @param length
|
||||
* @param jobGroup
|
||||
* @param jobDesc
|
||||
* @param executorHandler
|
||||
* @param glueType
|
||||
* @param author
|
||||
* @return
|
||||
*/
|
||||
public Map<String, Object> pageList(int start, int length, int jobGroup, int triggerStatus, String jobDesc, String executorHandler, String author);
|
||||
public Map<String, Object> pageList(int start, int length, int jobGroup, int triggerStatus, String jobDesc, String glueType, String author);
|
||||
|
||||
/**
|
||||
* add job
|
||||
|
||||
@ -47,7 +47,7 @@ public class JdbcDatasourceQueryServiceImpl implements JdbcDatasourceQueryServic
|
||||
return Lists.newArrayList();
|
||||
}
|
||||
BaseQueryTool queryTool = QueryToolFactory.getByDbType(jdbcDatasource);
|
||||
return queryTool.getColumnNames(tableName,jdbcDatasource.getJdbcDriverClass());
|
||||
return queryTool.getColumnNames(tableName,jdbcDatasource.getDatasource());
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@ -46,11 +46,11 @@ public class JobServiceImpl implements JobService {
|
||||
private final static ConcurrentMap<String, String> jobTmpFiles = Maps.newConcurrentMap();
|
||||
|
||||
@Override
|
||||
public Map<String, Object> pageList(int start, int length, int jobGroup, int triggerStatus, String jobDesc, String executorHandler, String author) {
|
||||
public Map<String, Object> pageList(int start, int length, int jobGroup, int triggerStatus, String jobDesc, String glueType, String author) {
|
||||
|
||||
// page list
|
||||
List<JobInfo> list = jobInfoMapper.pageList(start, length, jobGroup, triggerStatus, jobDesc, executorHandler, author);
|
||||
int list_count = jobInfoMapper.pageListCount(start, length, jobGroup, triggerStatus, jobDesc, executorHandler, author);
|
||||
List<JobInfo> list = jobInfoMapper.pageList(start, length, jobGroup, triggerStatus, jobDesc, glueType, author);
|
||||
int list_count = jobInfoMapper.pageListCount(start, length, jobGroup, triggerStatus, jobDesc, glueType, author);
|
||||
|
||||
// package result
|
||||
Map<String, Object> maps = new HashMap<String, Object>();
|
||||
@ -70,7 +70,7 @@ public class JobServiceImpl implements JobService {
|
||||
if (!CronExpression.isValidExpression(jobInfo.getJobCron())) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("jobinfo_field_cron_unvalid") );
|
||||
}
|
||||
if (jobInfo.getJobJson().trim().length()<=2) {
|
||||
if (jobInfo.getGlueType().equals(GlueTypeEnum.BEAN.getDesc()) && jobInfo.getJobJson().trim().length()<=2) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, (I18nUtil.getString("system_please_input")+I18nUtil.getString("jobinfo_field_jobjson")) );
|
||||
}
|
||||
if (jobInfo.getJobDesc()==null || jobInfo.getJobDesc().trim().length()==0) {
|
||||
@ -151,7 +151,7 @@ public class JobServiceImpl implements JobService {
|
||||
if (!CronExpression.isValidExpression(jobInfo.getJobCron())) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("jobinfo_field_cron_unvalid") );
|
||||
}
|
||||
if (jobInfo.getJobJson().trim().length()<=2) {
|
||||
if (jobInfo.getGlueType().equals(GlueTypeEnum.BEAN.getDesc()) && jobInfo.getJobJson().trim().length()<=2) {
|
||||
return new ReturnT<String>(ReturnT.FAIL_CODE, (I18nUtil.getString("system_please_input")+I18nUtil.getString("jobinfo_field_jobjson")) );
|
||||
}
|
||||
if (jobInfo.getJobDesc()==null || jobInfo.getJobDesc().trim().length()==0) {
|
||||
@ -233,11 +233,21 @@ public class JobServiceImpl implements JobService {
|
||||
exists_jobInfo.setExecutorFailRetryCount(jobInfo.getExecutorFailRetryCount());
|
||||
exists_jobInfo.setChildJobId(jobInfo.getChildJobId());
|
||||
exists_jobInfo.setTriggerNextTime(nextTriggerTime);
|
||||
exists_jobInfo.setJobJson(jobInfo.getJobJson());
|
||||
exists_jobInfo.setReplaceParam(jobInfo.getReplaceParam());
|
||||
exists_jobInfo.setJvmParam(jobInfo.getJvmParam());
|
||||
exists_jobInfo.setIncStartTime(jobInfo.getIncStartTime());
|
||||
exists_jobInfo.setUpdateTime(new Date());
|
||||
exists_jobInfo.setGlueType(jobInfo.getGlueType());
|
||||
exists_jobInfo.setPartitionInfo(jobInfo.getPartitionInfo());
|
||||
|
||||
if(GlueTypeEnum.BEAN.getDesc().equals(jobInfo.getGlueType())){
|
||||
exists_jobInfo.setJobJson(jobInfo.getJobJson());
|
||||
exists_jobInfo.setGlueSource(null);
|
||||
}else{
|
||||
exists_jobInfo.setGlueSource(jobInfo.getGlueSource());
|
||||
exists_jobInfo.setJobJson(null);
|
||||
}
|
||||
exists_jobInfo.setGlueUpdatetime(new Date());
|
||||
jobInfoMapper.update(exists_jobInfo);
|
||||
|
||||
|
||||
@ -302,10 +312,10 @@ public class JobServiceImpl implements JobService {
|
||||
int jobInfoCount = jobInfoMapper.findAllCount();
|
||||
int jobLogCount = 0;
|
||||
int jobLogSuccessCount = 0;
|
||||
JobLogReport xxlJobLogReport = jobLogReportMapper.queryLogReportTotal();
|
||||
if (xxlJobLogReport != null) {
|
||||
jobLogCount = xxlJobLogReport.getRunningCount() + xxlJobLogReport.getSucCount() + xxlJobLogReport.getFailCount();
|
||||
jobLogSuccessCount = xxlJobLogReport.getSucCount();
|
||||
JobLogReport jobLogReport = jobLogReportMapper.queryLogReportTotal();
|
||||
if (jobLogReport != null) {
|
||||
jobLogCount = jobLogReport.getRunningCount() + jobLogReport.getSucCount() + jobLogReport.getFailCount();
|
||||
jobLogSuccessCount = jobLogReport.getSucCount();
|
||||
}
|
||||
|
||||
// executor count
|
||||
|
||||
@ -4,13 +4,10 @@ import com.google.common.collect.Maps;
|
||||
import com.wugui.datatx.core.biz.model.ReturnT;
|
||||
import com.wugui.datatx.core.enums.ExecutorBlockStrategyEnum;
|
||||
import com.wugui.datatx.core.glue.GlueTypeEnum;
|
||||
import com.wugui.datatx.core.util.DateUtil;
|
||||
import com.wugui.datax.admin.core.cron.CronExpression;
|
||||
import com.wugui.datax.admin.core.route.ExecutorRouteStrategyEnum;
|
||||
import com.wugui.datax.admin.core.thread.JobScheduleHelper;
|
||||
import com.wugui.datax.admin.core.util.I18nUtil;
|
||||
import com.wugui.datax.admin.entity.JobGroup;
|
||||
import com.wugui.datax.admin.entity.JobLogReport;
|
||||
import com.wugui.datax.admin.entity.JobTemplate;
|
||||
import com.wugui.datax.admin.mapper.*;
|
||||
import com.wugui.datax.admin.service.JobTemplateService;
|
||||
@ -20,7 +17,6 @@ import org.springframework.stereotype.Service;
|
||||
|
||||
import javax.annotation.Resource;
|
||||
import java.text.MessageFormat;
|
||||
import java.text.ParseException;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
|
||||
@ -219,6 +215,7 @@ public class JobTemplateServiceImpl implements JobTemplateService {
|
||||
exists_jobTemplate.setReplaceParam(jobTemplate.getReplaceParam());
|
||||
exists_jobTemplate.setJvmParam(jobTemplate.getJvmParam());
|
||||
exists_jobTemplate.setIncStartTime(jobTemplate.getIncStartTime());
|
||||
exists_jobTemplate.setPartitionInfo(jobTemplate.getPartitionInfo());
|
||||
exists_jobTemplate.setUpdateTime(new Date());
|
||||
jobTemplateMapper.update(exists_jobTemplate);
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@ import com.wugui.datax.admin.tool.datax.reader.*;
|
||||
import com.wugui.datax.admin.tool.datax.writer.*;
|
||||
import com.wugui.datax.admin.tool.pojo.DataxHivePojo;
|
||||
import com.wugui.datax.admin.tool.pojo.DataxRdbmsPojo;
|
||||
import com.wugui.datax.admin.util.Constant;
|
||||
import com.wugui.datatx.core.util.Constant;
|
||||
import lombok.Data;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
@ -179,11 +179,11 @@ public class DataxJsonHelper implements DataxJsonInterface {
|
||||
public Map<String, Object> buildHiveReader() {
|
||||
DataxHivePojo dataxHivePojo = new DataxHivePojo();
|
||||
dataxHivePojo.setJdbcDatasource(readerDatasource);
|
||||
List<Map<String, String>> columns = Lists.newArrayList();
|
||||
List<Map<String, Object>> columns = Lists.newArrayList();
|
||||
readerColumns.forEach(c -> {
|
||||
Map<String, String> column = Maps.newLinkedHashMap();
|
||||
column.put("name", c.split(Constant.SPLIT_SCOLON)[0]);
|
||||
column.put("type", c.split(Constant.SPLIT_SCOLON)[1]);
|
||||
Map<String, Object> column = Maps.newLinkedHashMap();
|
||||
column.put("index", c.split(Constant.SPLIT_SCOLON)[0]);
|
||||
column.put("type", c.split(Constant.SPLIT_SCOLON)[2]);
|
||||
columns.add(column);
|
||||
});
|
||||
dataxHivePojo.setColumns(columns);
|
||||
@ -208,11 +208,11 @@ public class DataxJsonHelper implements DataxJsonInterface {
|
||||
public Map<String, Object> buildHiveWriter() {
|
||||
DataxHivePojo dataxHivePojo = new DataxHivePojo();
|
||||
dataxHivePojo.setJdbcDatasource(writerDatasource);
|
||||
List<Map<String, String>> columns = Lists.newArrayList();
|
||||
List<Map<String, Object>> columns = Lists.newArrayList();
|
||||
writerColumns.forEach(c -> {
|
||||
Map<String, String> column = Maps.newLinkedHashMap();
|
||||
column.put("name", c.split(Constant.SPLIT_SCOLON)[0]);
|
||||
column.put("type", c.split(Constant.SPLIT_SCOLON)[1]);
|
||||
Map<String, Object> column = Maps.newLinkedHashMap();
|
||||
column.put("name", c.split(Constant.SPLIT_SCOLON)[1]);
|
||||
column.put("type", c.split(Constant.SPLIT_SCOLON)[2]);
|
||||
columns.add(column);
|
||||
});
|
||||
dataxHivePojo.setColumns(columns);
|
||||
|
||||
@ -15,7 +15,7 @@ import java.util.Map;
|
||||
public class HiveReader extends BaseReaderPlugin implements DataxReaderInterface {
|
||||
@Override
|
||||
public String getName() {
|
||||
return "hivereader";
|
||||
return "hdfsreader";
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@ -15,7 +15,7 @@ import java.util.Map;
|
||||
public class HiveWriter extends BaseWriterPlugin implements DataxWriterInterface {
|
||||
@Override
|
||||
public String getName() {
|
||||
return "hivewriter";
|
||||
return "hdfswriter";
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -20,7 +20,7 @@ public class DataxHivePojo {
|
||||
/**
|
||||
* hive列名
|
||||
*/
|
||||
private List<Map<String,String>> columns;
|
||||
private List<Map<String,Object>> columns;
|
||||
|
||||
/**
|
||||
* 数据源信息
|
||||
|
||||
@ -5,6 +5,7 @@ import com.alibaba.druid.util.JdbcConstants;
|
||||
import com.alibaba.druid.util.JdbcUtils;
|
||||
import com.google.common.collect.ImmutableList;
|
||||
import com.google.common.collect.Lists;
|
||||
import com.wugui.datatx.core.util.Constant;
|
||||
import com.wugui.datax.admin.core.util.LocalCacheUtil;
|
||||
import com.wugui.datax.admin.entity.JobJdbcDatasource;
|
||||
import com.wugui.datax.admin.tool.database.ColumnInfo;
|
||||
@ -12,7 +13,6 @@ import com.wugui.datax.admin.tool.database.DasColumn;
|
||||
import com.wugui.datax.admin.tool.database.TableInfo;
|
||||
import com.wugui.datax.admin.tool.meta.DatabaseInterface;
|
||||
import com.wugui.datax.admin.tool.meta.DatabaseMetaFactory;
|
||||
import com.wugui.datax.admin.util.Constant;
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
@ -251,7 +251,7 @@ public abstract class BaseQueryTool implements QueryToolInterface {
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<String> getColumnNames(String tableName, String driverClass) {
|
||||
public List<String> getColumnNames(String tableName, String datasource) {
|
||||
|
||||
List<String> res = Lists.newArrayList();
|
||||
Statement stmt = null;
|
||||
@ -269,11 +269,11 @@ public abstract class BaseQueryTool implements QueryToolInterface {
|
||||
int columnCount = metaData.getColumnCount();
|
||||
for (int i = 1; i <= columnCount; i++) {
|
||||
String columnName = metaData.getColumnName(i);
|
||||
if (JdbcConstants.HIVE_DRIVER.equals(driverClass)) {
|
||||
if (JdbcConstants.HIVE.equals(datasource)) {
|
||||
if (columnName.contains(Constant.SPLIT_POINT)) {
|
||||
res.add(columnName.substring(columnName.indexOf(Constant.SPLIT_POINT) + 1) + Constant.SPLIT_SCOLON + metaData.getColumnTypeName(i));
|
||||
res.add(i - 1 + Constant.SPLIT_SCOLON + columnName.substring(columnName.indexOf(Constant.SPLIT_POINT) + 1) + Constant.SPLIT_SCOLON + metaData.getColumnTypeName(i));
|
||||
} else {
|
||||
res.add(columnName + Constant.SPLIT_SCOLON + metaData.getColumnTypeName(i));
|
||||
res.add(i - 1 + Constant.SPLIT_SCOLON + columnName + Constant.SPLIT_SCOLON + metaData.getColumnTypeName(i));
|
||||
}
|
||||
} else {
|
||||
res.add(columnName);
|
||||
|
||||
@ -51,7 +51,7 @@ public interface QueryToolInterface {
|
||||
* @param tableName
|
||||
* @return2
|
||||
*/
|
||||
public List<String> getColumnNames(String tableName,String driverClass);
|
||||
public List<String> getColumnNames(String tableName,String datasource);
|
||||
|
||||
|
||||
/**
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
package com.wugui.datax.admin.util;
|
||||
|
||||
import com.alibaba.druid.util.JdbcConstants;
|
||||
import com.wugui.datatx.core.util.Constant;
|
||||
|
||||
/**
|
||||
* Created by judy.lt on 2015/6/5.
|
||||
|
||||
BIN
datax-admin/src/main/lib/HiveJDBC41.jar
Normal file
BIN
datax-admin/src/main/lib/HiveJDBC41.jar
Normal file
Binary file not shown.
@ -114,6 +114,10 @@ jobinfo_field_id=任务ID
|
||||
jobinfo_field_jobgroup=执行器
|
||||
jobinfo_field_jobdesc=任务描述
|
||||
jobinfo_field_jobjson=任务JSON
|
||||
jobinfo_field_readerDataSource=Reader数据源
|
||||
jobinfo_field_writerDataSource=Writer数据源
|
||||
jobinfo_field_readerColumns=Reader Column
|
||||
jobinfo_field_writerColumns=Writer Column
|
||||
jobinfo_field_gluetype=运行模式
|
||||
jobinfo_field_executorparam=任务参数
|
||||
jobinfo_field_cron_unvalid=Cron格式非法
|
||||
|
||||
@ -38,6 +38,8 @@
|
||||
<result column="replace_param" property="replaceParam" />
|
||||
<result column="jvm_param" property="jvmParam" />
|
||||
<result column="inc_start_time" property="incStartTime" />
|
||||
<result column="partition_info" property="partitionInfo" />
|
||||
|
||||
</resultMap>
|
||||
|
||||
<sql id="Base_Column_List">
|
||||
@ -66,7 +68,8 @@
|
||||
t.job_json,
|
||||
t.replace_param,
|
||||
t.jvm_param,
|
||||
t.inc_start_time
|
||||
t.inc_start_time,
|
||||
t.partition_info
|
||||
</sql>
|
||||
|
||||
<select id="pageList" parameterType="java.util.HashMap" resultMap="JobInfo">
|
||||
@ -82,8 +85,8 @@
|
||||
<if test="jobDesc != null and jobDesc != ''">
|
||||
AND t.job_desc like CONCAT(CONCAT('%', #{jobDesc}), '%')
|
||||
</if>
|
||||
<if test="executorHandler != null and executorHandler != ''">
|
||||
AND t.executor_handler like CONCAT(CONCAT('%', #{executorHandler}), '%')
|
||||
<if test="glueType != null and glueType != ''">
|
||||
AND t.glue_type like CONCAT(CONCAT('%', #{glueType}), '%')
|
||||
</if>
|
||||
<if test="author != null and author != ''">
|
||||
AND t.author like CONCAT(CONCAT('%', #{author}), '%')
|
||||
@ -106,8 +109,8 @@
|
||||
<if test="jobDesc != null and jobDesc != ''">
|
||||
AND t.job_desc like CONCAT(CONCAT('%', #{jobDesc}), '%')
|
||||
</if>
|
||||
<if test="executorHandler != null and executorHandler != ''">
|
||||
AND t.executor_handler like CONCAT(CONCAT('%', #{executorHandler}), '%')
|
||||
<if test="glueType != null and glueType != ''">
|
||||
AND t.glue_type like CONCAT(CONCAT('%', #{glueType}), '%')
|
||||
</if>
|
||||
<if test="author != null and author != ''">
|
||||
AND t.author like CONCAT(CONCAT('%', #{author}), '%')
|
||||
@ -141,7 +144,8 @@
|
||||
job_json,
|
||||
replace_param,
|
||||
jvm_param,
|
||||
inc_start_time
|
||||
inc_start_time,
|
||||
partition_info
|
||||
) VALUES (
|
||||
#{jobGroup},
|
||||
#{jobCron},
|
||||
@ -167,7 +171,8 @@
|
||||
#{jobJson},
|
||||
#{replaceParam},
|
||||
#{jvmParam},
|
||||
#{incStartTime}
|
||||
#{incStartTime},
|
||||
#{partitionInfo}
|
||||
);
|
||||
<!--<selectKey resultType="java.lang.Integer" order="AFTER" keyProperty="id">
|
||||
SELECT LAST_INSERT_ID()
|
||||
@ -207,7 +212,8 @@
|
||||
job_json=#{jobJson},
|
||||
replace_param=#{replaceParam},
|
||||
jvm_param=#{jvmParam},
|
||||
inc_start_time=#{incStartTime}
|
||||
inc_start_time=#{incStartTime},
|
||||
partition_info=#{partitionInfo}
|
||||
WHERE id = #{id}
|
||||
</update>
|
||||
|
||||
|
||||
@ -37,6 +37,8 @@
|
||||
<result column="replace_param" property="replaceParam" />
|
||||
<result column="jvm_param" property="jvmParam" />
|
||||
<result column="inc_start_time" property="incStartTime" />
|
||||
|
||||
<result column="partition_info" property="partitionInfo" />
|
||||
</resultMap>
|
||||
|
||||
<sql id="Base_Column_List">
|
||||
@ -64,7 +66,8 @@
|
||||
t.job_json,
|
||||
t.replace_param,
|
||||
t.jvm_param,
|
||||
t.inc_start_time
|
||||
t.inc_start_time,
|
||||
t.partition_info
|
||||
</sql>
|
||||
|
||||
<select id="pageList" parameterType="java.util.HashMap" resultMap="JobTemplate">
|
||||
@ -132,7 +135,8 @@
|
||||
job_json,
|
||||
replace_param,
|
||||
jvm_param,
|
||||
inc_start_time
|
||||
inc_start_time,
|
||||
partition_info
|
||||
) VALUES (
|
||||
#{jobGroup},
|
||||
#{jobCron},
|
||||
@ -157,7 +161,8 @@
|
||||
#{jobJson},
|
||||
#{replaceParam},
|
||||
#{jvmParam},
|
||||
#{incStartTime}
|
||||
#{incStartTime},
|
||||
#{partitionInfo}
|
||||
);
|
||||
<!--<selectKey resultType="java.lang.Integer" order="AFTER" keyProperty="id">
|
||||
SELECT LAST_INSERT_ID()
|
||||
@ -196,7 +201,8 @@
|
||||
job_json=#{jobJson},
|
||||
replace_param=#{replaceParam},
|
||||
jvm_param=#{jvmParam},
|
||||
inc_start_time=#{incStartTime}
|
||||
inc_start_time=#{incStartTime},
|
||||
partition_info=#{partitionInfo}
|
||||
WHERE id = #{id}
|
||||
</update>
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}.json-editor[data-v-fad11014]{height:100%;position:relative}.json-editor[data-v-fad11014] .CodeMirror{height:auto;min-height:300px}.json-editor[data-v-fad11014] .CodeMirror-scroll{min-height:300px}.json-editor[data-v-fad11014] .cm-s-rubyblue span.cm-string{color:#f08047}.shell-editor[data-v-eb90f6fe]{height:100%;position:relative}.shell-editor[data-v-eb90f6fe] .CodeMirror{height:auto;min-height:300px}.shell-editor[data-v-eb90f6fe] .CodeMirror-scroll{min-height:300px}.shell-editor[data-v-eb90f6fe] .cm-s-rubyblue span.cm-string{color:#f08047}.python-editor[data-v-a093f8fc]{height:100%;position:relative}.python-editor[data-v-a093f8fc] .CodeMirror{height:auto;min-height:300px}.python-editor[data-v-a093f8fc] .CodeMirror-scroll{min-height:300px}.python-editor[data-v-a093f8fc] .cm-s-rubyblue span.cm-string{color:#f08047}.powershell-editor[data-v-708ede48]{height:100%;position:relative}.powershell-editor[data-v-708ede48] .CodeMirror{height:auto;min-height:300px}.powershell-editor[data-v-708ede48] .CodeMirror-scroll{min-height:300px}.powershell-editor[data-v-708ede48] .cm-s-rubyblue span.cm-string{color:#f08047}.el-dropdown+.el-dropdown{margin-left:15px}
|
||||
@ -1 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.el-dropdown+.el-dropdown{margin-left:15px}
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}.el-dropdown+.el-dropdown{margin-left:15px}
|
||||
@ -1 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.el-dropdown+.el-dropdown{margin-left:15px}
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}
|
||||
@ -1 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.log-container[data-v-0d60fa66]{margin-bottom:20px;background:#f5f5f5;width:100%;height:500px;overflow:scroll}.log-container pre[data-v-0d60fa66]{display:block;padding:10px;margin:0 0 10.5px;word-break:break-all;word-wrap:break-word;color:#334851;background-color:#f5f5f5;border-radius:1px}
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}.log-container[data-v-0d60fa66]{margin-bottom:20px;background:#f5f5f5;width:100%;height:500px;overflow:scroll}.log-container pre[data-v-0d60fa66]{display:block;padding:10px;margin:0 0 10.5px;word-break:break-all;word-wrap:break-word;color:#334851;background-color:#f5f5f5;border-radius:1px}
|
||||
@ -1 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}
|
||||
@ -1 +0,0 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}
|
||||
@ -1 +1 @@
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}
|
||||
.waves-ripple{position:absolute;border-radius:100%;background-color:rgba(0,0,0,.15);background-clip:padding-box;pointer-events:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transform:scale(0);transform:scale(0);opacity:1}.waves-ripple.z-active{opacity:0;-webkit-transform:scale(2);transform:scale(2);-webkit-transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,-webkit-transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out;transition:opacity 1.2s ease-out,transform .6s ease-out,-webkit-transform .6s ease-out}.pagination-container[data-v-f3b72548]{background:#fff;padding:32px 16px}.pagination-container.hidden[data-v-f3b72548]{display:none}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -1 +1 @@
|
||||
(window["webpackJsonp"]=window["webpackJsonp"]||[]).push([["chunk-5bfc3adc"],{"00e3":function(t,s,a){},"1bc5":function(t,s,a){"use strict";var c=a("00e3"),e=a.n(c);e.a},"1db4":function(t,s,a){"use strict";a.r(s);var c=function(){var t=this,s=t.$createElement,a=t._self._c||s;return a("div",{staticClass:"wscn-http404-container"},[a("div",{staticClass:"wscn-http404"},[t._m(0),t._v(" "),a("div",{staticClass:"bullshit"},[a("div",{staticClass:"bullshit__oops"},[t._v("OOPS!")]),t._v(" "),t._m(1),t._v(" "),a("div",{staticClass:"bullshit__headline"},[t._v(t._s(t.message))]),t._v(" "),a("div",{staticClass:"bullshit__info"},[t._v("Please check that the URL you entered is correct, or click the button below to return to the homepage.")]),t._v(" "),a("a",{staticClass:"bullshit__return-home",attrs:{href:""}},[t._v("Back to home")])])])])},e=[function(){var t=this,s=t.$createElement,c=t._self._c||s;return c("div",{staticClass:"pic-404"},[c("img",{staticClass:"pic-404__parent",attrs:{src:a("a36b"),alt:"404"}}),t._v(" "),c("img",{staticClass:"pic-404__child left",attrs:{src:a("26fc"),alt:"404"}}),t._v(" "),c("img",{staticClass:"pic-404__child mid",attrs:{src:a("26fc"),alt:"404"}}),t._v(" "),c("img",{staticClass:"pic-404__child right",attrs:{src:a("26fc"),alt:"404"}})])},function(){var t=this,s=t.$createElement,a=t._self._c||s;return a("div",{staticClass:"bullshit__info"},[t._v("All rights reserved\n "),a("a",{staticStyle:{color:"#20a0ff"},attrs:{href:"https://wallstreetcn.com",target:"_blank"}},[t._v("wallstreetcn")])])}],i={name:"Page404",computed:{message:function(){return"The webmaster said that you can not enter this page..."}}},l=i,n=(a("1bc5"),a("2877")),r=Object(n["a"])(l,c,e,!1,null,"22c31b5a",null);s["default"]=r.exports},"26fc":function(t,s,a){t.exports=a.p+"static/img/404_cloud.0f4bc32b.png"},a36b:function(t,s,a){t.exports=a.p+"static/img/404.a57b6f31.png"}}]);
|
||||
(window["webpackJsonp"]=window["webpackJsonp"]||[]).push([["chunk-64238bd4"],{"00e3":function(t,s,a){},"1bc5":function(t,s,a){"use strict";var e=a("00e3"),c=a.n(e);c.a},"1db4":function(t,s,a){"use strict";a.r(s);var e=function(){var t=this,s=t.$createElement,a=t._self._c||s;return a("div",{staticClass:"wscn-http404-container"},[a("div",{staticClass:"wscn-http404"},[t._m(0),t._v(" "),a("div",{staticClass:"bullshit"},[a("div",{staticClass:"bullshit__oops"},[t._v("OOPS!")]),t._v(" "),t._m(1),t._v(" "),a("div",{staticClass:"bullshit__headline"},[t._v(t._s(t.message))]),t._v(" "),a("div",{staticClass:"bullshit__info"},[t._v("Please check that the URL you entered is correct, or click the button below to return to the homepage.")]),t._v(" "),a("a",{staticClass:"bullshit__return-home",attrs:{href:""}},[t._v("Back to home")])])])])},c=[function(){var t=this,s=t.$createElement,e=t._self._c||s;return e("div",{staticClass:"pic-404"},[e("img",{staticClass:"pic-404__parent",attrs:{src:a("a36b"),alt:"404"}}),t._v(" "),e("img",{staticClass:"pic-404__child left",attrs:{src:a("26fc"),alt:"404"}}),t._v(" "),e("img",{staticClass:"pic-404__child mid",attrs:{src:a("26fc"),alt:"404"}}),t._v(" "),e("img",{staticClass:"pic-404__child right",attrs:{src:a("26fc"),alt:"404"}})])},function(){var t=this,s=t.$createElement,a=t._self._c||s;return a("div",{staticClass:"bullshit__info"},[t._v("All rights reserved\n "),a("a",{staticStyle:{color:"#20a0ff"},attrs:{href:"https://wallstreetcn.com",target:"_blank"}},[t._v("wallstreetcn")])])}],i={name:"Page404",computed:{message:function(){return"The webmaster said that you can not enter this page..."}}},l=i,n=(a("1bc5"),a("2877")),r=Object(n["a"])(l,e,c,!1,null,"22c31b5a",null);s["default"]=r.exports},"26fc":function(t,s,a){t.exports=a.p+"static/img/404_cloud.0f4bc32b.png"},a36b:function(t,s,a){t.exports=a.p+"static/img/404.a57b6f31.png"}}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -1 +1 @@
|
||||
(window["webpackJsonp"]=window["webpackJsonp"]||[]).push([["chunk-46fcab30"],{"24e2":function(t,a,i){"use strict";i.r(a);var e=function(){var t=this,a=t.$createElement,i=t._self._c||a;return i("div",{staticClass:"errPage-container"},[i("el-button",{staticClass:"pan-back-btn",attrs:{icon:"arrow-left"},on:{click:t.back}},[t._v("\n 返回\n ")]),t._v(" "),i("el-row",[i("el-col",{attrs:{span:12}},[i("h1",{staticClass:"text-jumbo text-ginormous"},[t._v("\n Oops!\n ")]),t._v("\n gif来源"),i("a",{attrs:{href:"https://zh.airbnb.com/",target:"_blank"}},[t._v("airbnb")]),t._v(" 页面\n "),i("h2",[t._v("你没有权限去该页面")]),t._v(" "),i("h6",[t._v("如有不满请联系你领导")]),t._v(" "),i("ul",{staticClass:"list-unstyled"},[i("li",[t._v("或者你可以去:")]),t._v(" "),i("li",{staticClass:"link-type"},[i("router-link",{attrs:{to:"/dashboard"}},[t._v("\n 回首页\n ")])],1),t._v(" "),i("li",{staticClass:"link-type"},[i("a",{attrs:{href:"https://www.taobao.com/"}},[t._v("随便看看")])]),t._v(" "),i("li",[i("a",{attrs:{href:"#"},on:{click:function(a){a.preventDefault(),t.dialogVisible=!0}}},[t._v("点我看图")])])])]),t._v(" "),i("el-col",{attrs:{span:12}},[i("img",{attrs:{src:t.errGif,width:"313",height:"428",alt:"Girl has dropped her ice cream."}})])],1),t._v(" "),i("el-dialog",{attrs:{visible:t.dialogVisible,title:"随便看"},on:{"update:visible":function(a){t.dialogVisible=a}}},[i("img",{staticClass:"pan-img",attrs:{src:t.ewizardClap}})])],1)},s=[],n=i("cc6c"),r=i.n(n),c={name:"Page401",data:function(){return{errGif:r.a+"?"+ +new Date,ewizardClap:"https://wpimg.wallstcn.com/007ef517-bafd-4066-aae4-6883632d9646",dialogVisible:!1}},methods:{back:function(){this.$route.query.noGoBack?this.$router.push({path:"/dashboard"}):this.$router.go(-1)}}},l=c,o=(i("efaf"),i("2877")),u=Object(o["a"])(l,e,s,!1,null,"2c77c6fb",null);a["default"]=u.exports},5757:function(t,a,i){},cc6c:function(t,a,i){t.exports=i.p+"static/img/401.089007e7.gif"},efaf:function(t,a,i){"use strict";var e=i("5757"),s=i.n(e);s.a}}]);
|
||||
(window["webpackJsonp"]=window["webpackJsonp"]||[]).push([["chunk-73865ec4"],{"24e2":function(t,a,i){"use strict";i.r(a);var e=function(){var t=this,a=t.$createElement,i=t._self._c||a;return i("div",{staticClass:"errPage-container"},[i("el-button",{staticClass:"pan-back-btn",attrs:{icon:"arrow-left"},on:{click:t.back}},[t._v("\n 返回\n ")]),t._v(" "),i("el-row",[i("el-col",{attrs:{span:12}},[i("h1",{staticClass:"text-jumbo text-ginormous"},[t._v("\n Oops!\n ")]),t._v("\n gif来源"),i("a",{attrs:{href:"https://zh.airbnb.com/",target:"_blank"}},[t._v("airbnb")]),t._v(" 页面\n "),i("h2",[t._v("你没有权限去该页面")]),t._v(" "),i("h6",[t._v("如有不满请联系你领导")]),t._v(" "),i("ul",{staticClass:"list-unstyled"},[i("li",[t._v("或者你可以去:")]),t._v(" "),i("li",{staticClass:"link-type"},[i("router-link",{attrs:{to:"/dashboard"}},[t._v("\n 回首页\n ")])],1),t._v(" "),i("li",{staticClass:"link-type"},[i("a",{attrs:{href:"https://www.taobao.com/"}},[t._v("随便看看")])]),t._v(" "),i("li",[i("a",{attrs:{href:"#"},on:{click:function(a){a.preventDefault(),t.dialogVisible=!0}}},[t._v("点我看图")])])])]),t._v(" "),i("el-col",{attrs:{span:12}},[i("img",{attrs:{src:t.errGif,width:"313",height:"428",alt:"Girl has dropped her ice cream."}})])],1),t._v(" "),i("el-dialog",{attrs:{visible:t.dialogVisible,title:"随便看"},on:{"update:visible":function(a){t.dialogVisible=a}}},[i("img",{staticClass:"pan-img",attrs:{src:t.ewizardClap}})])],1)},s=[],n=i("cc6c"),r=i.n(n),c={name:"Page401",data:function(){return{errGif:r.a+"?"+ +new Date,ewizardClap:"https://wpimg.wallstcn.com/007ef517-bafd-4066-aae4-6883632d9646",dialogVisible:!1}},methods:{back:function(){this.$route.query.noGoBack?this.$router.push({path:"/dashboard"}):this.$router.go(-1)}}},l=c,o=(i("efaf"),i("2877")),u=Object(o["a"])(l,e,s,!1,null,"2c77c6fb",null);a["default"]=u.exports},5757:function(t,a,i){},cc6c:function(t,a,i){t.exports=i.p+"static/img/401.089007e7.gif"},efaf:function(t,a,i){"use strict";var e=i("5757"),s=i.n(e);s.a}}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
(window["webpackJsonp"]=window["webpackJsonp"]||[]).push([["chunk-commons"],{"1c64":function(e,t,n){},"1cc6":function(e,t,n){"use strict";var a=n("1c64"),i=n.n(a);i.a},"333d":function(e,t,n){"use strict";var a=function(){var e=this,t=e.$createElement,n=e._self._c||t;return n("div",{staticClass:"pagination-container",class:{hidden:e.hidden}},[n("el-pagination",e._b({attrs:{background:e.background,"current-page":e.currentPage,"page-size":e.pageSize,layout:e.layout,"page-sizes":e.pageSizes,total:e.total},on:{"update:currentPage":function(t){e.currentPage=t},"update:current-page":function(t){e.currentPage=t},"update:pageSize":function(t){e.pageSize=t},"update:page-size":function(t){e.pageSize=t},"size-change":e.handleSizeChange,"current-change":e.handleCurrentChange}},"el-pagination",e.$attrs,!1))],1)},i=[],r=(n("c5f6"),n("09f4")),u={name:"Pagination",props:{total:{required:!0,type:Number},page:{type:Number,default:1},limit:{type:Number,default:20},pageSizes:{type:Array,default:function(){return[10,20,30,50]}},layout:{type:String,default:"total, sizes, prev, pager, next, jumper"},background:{type:Boolean,default:!0},autoScroll:{type:Boolean,default:!0},hidden:{type:Boolean,default:!1}},computed:{currentPage:{get:function(){return this.page},set:function(e){this.$emit("update:page",e)}},pageSize:{get:function(){return this.limit},set:function(e){this.$emit("update:limit",e)}}},methods:{handleSizeChange:function(e){this.$emit("pagination",{page:this.currentPage,limit:e}),this.autoScroll&&Object(r["a"])(0,800)},handleCurrentChange:function(e){this.$emit("pagination",{page:e,limit:this.pageSize}),this.autoScroll&&Object(r["a"])(0,800)}}},o=u,s=(n("1cc6"),n("2877")),c=Object(s["a"])(o,a,i,!1,null,"f3b72548",null);t["a"]=c.exports},b8b0:function(e,t,n){"use strict";var a=n("f84c"),i=n.n(a);i.a},f84c:function(e,t,n){},fa7e:function(e,t,n){"use strict";var a=function(){var e=this,t=e.$createElement,n=e._self._c||t;return n("div",{staticClass:"json-editor"},[n("textarea",{ref:"textarea"})])},i=[],r=n("56b3"),u=n.n(r);n("0dd0"),n("a7be"),n("acdf"),n("f9d4"),n("8822"),n("d2de");n("ae67");var o={name:"JsonEditor",props:["value"],data:function(){return{jsonEditor:!1}},watch:{value:function(e){var t=this.jsonEditor.getValue();e!==t&&this.jsonEditor.setValue(JSON.stringify(this.value,null,2))}},mounted:function(){var e=this;this.jsonEditor=u.a.fromTextArea(this.$refs.textarea,{lineNumbers:!0,mode:"application/json",gutters:["CodeMirror-lint-markers"],theme:"rubyblue",lint:!0}),this.jsonEditor.setValue(JSON.stringify(this.value,null,2)),this.jsonEditor.on("change",(function(t){e.$emit("changed",t.getValue()),e.$emit("input",t.getValue())}))},methods:{getValue:function(){return this.jsonEditor.getValue()}}},s=o,c=(n("b8b0"),n("2877")),l=Object(c["a"])(s,a,i,!1,null,"fad11014",null);t["a"]=l.exports}}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -12,16 +12,16 @@ import java.util.List;
|
||||
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
public class XxlJobGroupMapperTest {
|
||||
public class JobGroupMapperTest {
|
||||
|
||||
@Resource
|
||||
private JobGroupMapper xxlJobGroupMapper;
|
||||
private JobGroupMapper jobGroupMapper;
|
||||
|
||||
@Test
|
||||
public void test(){
|
||||
List<JobGroup> list = xxlJobGroupMapper.findAll();
|
||||
List<JobGroup> list = jobGroupMapper.findAll();
|
||||
|
||||
List<JobGroup> list2 = xxlJobGroupMapper.findByAddressType(0);
|
||||
List<JobGroup> list2 = jobGroupMapper.findByAddressType(0);
|
||||
|
||||
JobGroup group = new JobGroup();
|
||||
group.setAppName("setAppName");
|
||||
@ -30,18 +30,18 @@ public class XxlJobGroupMapperTest {
|
||||
group.setAddressType(0);
|
||||
group.setAddressList("setAddressList");
|
||||
|
||||
int ret = xxlJobGroupMapper.save(group);
|
||||
int ret = jobGroupMapper.save(group);
|
||||
|
||||
JobGroup group2 = xxlJobGroupMapper.load(group.getId());
|
||||
JobGroup group2 = jobGroupMapper.load(group.getId());
|
||||
group2.setAppName("setAppName2");
|
||||
group2.setTitle("setTitle2");
|
||||
group2.setOrder(2);
|
||||
group2.setAddressType(2);
|
||||
group2.setAddressList("setAddressList2");
|
||||
|
||||
int ret2 = xxlJobGroupMapper.update(group2);
|
||||
int ret2 = jobGroupMapper.update(group2);
|
||||
|
||||
int ret3 = xxlJobGroupMapper.remove(group.getId());
|
||||
int ret3 = jobGroupMapper.remove(group.getId());
|
||||
}
|
||||
|
||||
}
|
||||
@ -13,20 +13,20 @@ import java.util.List;
|
||||
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
public class XxlJobInfoMapperTest {
|
||||
public class JobInfoMapperTest {
|
||||
|
||||
@Resource
|
||||
private JobInfoMapper xxlJobInfoMapper;
|
||||
private JobInfoMapper jobInfoMapper;
|
||||
|
||||
@Test
|
||||
public void pageList(){
|
||||
List<JobInfo> list = xxlJobInfoMapper.pageList(0, 20, 0, -1, null, null, null);
|
||||
int list_count = xxlJobInfoMapper.pageListCount(0, 20, 0, -1, null, null, null);
|
||||
List<JobInfo> list = jobInfoMapper.pageList(0, 20, 0, -1, null, null, null);
|
||||
int list_count = jobInfoMapper.pageListCount(0, 20, 0, -1, null, null, null);
|
||||
|
||||
System.out.println(list);
|
||||
System.out.println(list_count);
|
||||
|
||||
List<JobInfo> list2 = xxlJobInfoMapper.getJobsByGroup(1);
|
||||
List<JobInfo> list2 = jobInfoMapper.getJobsByGroup(1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@ -50,9 +50,9 @@ public class XxlJobInfoMapperTest {
|
||||
info.setUpdateTime(new Date());
|
||||
info.setGlueUpdatetime(new Date());
|
||||
|
||||
int count = xxlJobInfoMapper.save(info);
|
||||
int count = jobInfoMapper.save(info);
|
||||
|
||||
JobInfo info2 = xxlJobInfoMapper.loadById(info.getId());
|
||||
JobInfo info2 = jobInfoMapper.loadById(info.getId());
|
||||
info2.setJobCron("jobCron2");
|
||||
info2.setJobDesc("desc2");
|
||||
info2.setAuthor("setAuthor2");
|
||||
@ -68,13 +68,13 @@ public class XxlJobInfoMapperTest {
|
||||
info2.setChildJobId("1");
|
||||
|
||||
info2.setUpdateTime(new Date());
|
||||
int item2 = xxlJobInfoMapper.update(info2);
|
||||
int item2 = jobInfoMapper.update(info2);
|
||||
|
||||
xxlJobInfoMapper.delete(info2.getId());
|
||||
jobInfoMapper.delete(info2.getId());
|
||||
|
||||
List<JobInfo> list2 = xxlJobInfoMapper.getJobsByGroup(1);
|
||||
List<JobInfo> list2 = jobInfoMapper.getJobsByGroup(1);
|
||||
|
||||
int ret3 = xxlJobInfoMapper.findAllCount();
|
||||
int ret3 = jobInfoMapper.findAllCount();
|
||||
|
||||
}
|
||||
|
||||
@ -13,10 +13,10 @@ import java.util.List;
|
||||
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
public class XxlJobLogGlueMapperTest {
|
||||
public class JobLogGlueMapperTest {
|
||||
|
||||
@Resource
|
||||
private JobLogGlueMapper xxlJobLogGlueMapper;
|
||||
private JobLogGlueMapper jobLogGlueMapper;
|
||||
|
||||
@Test
|
||||
public void test(){
|
||||
@ -28,13 +28,13 @@ public class XxlJobLogGlueMapperTest {
|
||||
|
||||
logGlue.setAddTime(new Date());
|
||||
logGlue.setUpdateTime(new Date());
|
||||
int ret = xxlJobLogGlueMapper.save(logGlue);
|
||||
int ret = jobLogGlueMapper.save(logGlue);
|
||||
|
||||
List<JobLogGlue> list = xxlJobLogGlueMapper.findByJobId(1);
|
||||
List<JobLogGlue> list = jobLogGlueMapper.findByJobId(1);
|
||||
|
||||
int ret2 = xxlJobLogGlueMapper.removeOld(1, 1);
|
||||
int ret2 = jobLogGlueMapper.removeOld(1, 1);
|
||||
|
||||
int ret3 = xxlJobLogGlueMapper.deleteByJobId(1);
|
||||
int ret3 = jobLogGlueMapper.deleteByJobId(1);
|
||||
}
|
||||
|
||||
}
|
||||
@ -13,22 +13,22 @@ import java.util.List;
|
||||
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
public class XxlJobLogMapperTest {
|
||||
public class JobLogMapperTest {
|
||||
|
||||
@Resource
|
||||
private JobLogMapper xxlJobLogMapper;
|
||||
private JobLogMapper jobLogMapper;
|
||||
|
||||
@Test
|
||||
public void test(){
|
||||
List<JobLog> list = xxlJobLogMapper.pageList(0, 10, 1, 1, null, null, 1);
|
||||
int list_count = xxlJobLogMapper.pageListCount(0, 10, 1, 1, null, null, 1);
|
||||
List<JobLog> list = jobLogMapper.pageList(0, 10, 1, 1, null, null, 1);
|
||||
int list_count = jobLogMapper.pageListCount(0, 10, 1, 1, null, null, 1);
|
||||
|
||||
JobLog log = new JobLog();
|
||||
log.setJobGroup(1);
|
||||
log.setJobId(1);
|
||||
|
||||
long ret1 = xxlJobLogMapper.save(log);
|
||||
JobLog dto = xxlJobLogMapper.load(log.getId());
|
||||
long ret1 = jobLogMapper.save(log);
|
||||
JobLog dto = jobLogMapper.load(log.getId());
|
||||
|
||||
log.setTriggerTime(new Date());
|
||||
log.setTriggerCode(1);
|
||||
@ -36,20 +36,20 @@ public class XxlJobLogMapperTest {
|
||||
log.setExecutorAddress("1");
|
||||
log.setExecutorHandler("1");
|
||||
log.setExecutorParam("1");
|
||||
ret1 = xxlJobLogMapper.updateTriggerInfo(log);
|
||||
dto = xxlJobLogMapper.load(log.getId());
|
||||
ret1 = jobLogMapper.updateTriggerInfo(log);
|
||||
dto = jobLogMapper.load(log.getId());
|
||||
|
||||
|
||||
log.setHandleTime(new Date());
|
||||
log.setHandleCode(2);
|
||||
log.setHandleMsg("2");
|
||||
ret1 = xxlJobLogMapper.updateHandleInfo(log);
|
||||
dto = xxlJobLogMapper.load(log.getId());
|
||||
ret1 = jobLogMapper.updateHandleInfo(log);
|
||||
dto = jobLogMapper.load(log.getId());
|
||||
|
||||
|
||||
List<Long> ret4 = xxlJobLogMapper.findClearLogIds(1, 1, new Date(), 100, 100);
|
||||
List<Long> ret4 = jobLogMapper.findClearLogIds(1, 1, new Date(), 100, 100);
|
||||
|
||||
int ret2 = xxlJobLogMapper.delete(log.getJobId());
|
||||
int ret2 = jobLogMapper.delete(log.getJobId());
|
||||
|
||||
}
|
||||
|
||||
@ -14,21 +14,21 @@ import java.util.List;
|
||||
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
public class XxlJobRegistryMapperTest {
|
||||
public class JobRegistryMapperTest {
|
||||
|
||||
@Resource
|
||||
private JobRegistryMapper xxlJobRegistryMapper;
|
||||
private JobRegistryMapper jobRegistryMapper;
|
||||
|
||||
@Test
|
||||
public void test(){
|
||||
int ret = xxlJobRegistryMapper.registryUpdate("g1", "k1", "v1", new Date());
|
||||
int ret = jobRegistryMapper.registryUpdate("g1", "k1", "v1", new Date());
|
||||
if (ret < 1) {
|
||||
ret = xxlJobRegistryMapper.registrySave("g1", "k1", "v1", new Date());
|
||||
ret = jobRegistryMapper.registrySave("g1", "k1", "v1", new Date());
|
||||
}
|
||||
|
||||
List<JobRegistry> list = xxlJobRegistryMapper.findAll(1, new Date());
|
||||
List<JobRegistry> list = jobRegistryMapper.findAll(1, new Date());
|
||||
|
||||
int ret2 = xxlJobRegistryMapper.removeDead(Arrays.asList(1));
|
||||
int ret2 = jobRegistryMapper.removeDead(Arrays.asList(1));
|
||||
}
|
||||
|
||||
}
|
||||
@ -7,7 +7,7 @@
|
||||
<parent>
|
||||
<groupId>com.wugui</groupId>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
|
||||
<artifactId>datax-core</artifactId>
|
||||
|
||||
@ -35,6 +35,8 @@ public class TriggerParam implements Serializable{
|
||||
private Date startTime;
|
||||
private Date triggerTime;
|
||||
|
||||
private String partitionInfo;
|
||||
|
||||
|
||||
public int getJobId() {
|
||||
return jobId;
|
||||
@ -180,6 +182,14 @@ public class TriggerParam implements Serializable{
|
||||
this.triggerTime = triggerTime;
|
||||
}
|
||||
|
||||
public String getPartitionInfo() {
|
||||
return partitionInfo;
|
||||
}
|
||||
|
||||
public void setPartitionInfo(String partitionInfo) {
|
||||
this.partitionInfo = partitionInfo;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "TriggerParam{" +
|
||||
@ -201,6 +211,7 @@ public class TriggerParam implements Serializable{
|
||||
", jvmParam=" + jvmParam +
|
||||
", startTime=" + startTime +
|
||||
", triggerTime=" + triggerTime +
|
||||
", partitionInfo=" + partitionInfo +
|
||||
'}';
|
||||
}
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ public abstract class IJobHandler {
|
||||
* @return
|
||||
* @throws Exception
|
||||
*/
|
||||
public abstract ReturnT<String> executeDataX(TriggerParam tgParam) throws Exception;
|
||||
public abstract ReturnT<String> execute(TriggerParam tgParam) throws Exception;
|
||||
|
||||
/**
|
||||
* init handler, invoked when JobThread init
|
||||
|
||||
@ -22,8 +22,8 @@ public class GlueJobHandler extends IJobHandler {
|
||||
}
|
||||
|
||||
@Override
|
||||
public ReturnT<String> executeDataX(TriggerParam tgParam) throws Exception {
|
||||
public ReturnT<String> execute(TriggerParam tgParam) throws Exception {
|
||||
JobLogger.log("----------- glue.version:"+ glueUpdatetime +" -----------");
|
||||
return jobHandler.executeDataX(tgParam);
|
||||
return jobHandler.execute(tgParam);
|
||||
}
|
||||
}
|
||||
|
||||
@ -48,7 +48,7 @@ public class ScriptJobHandler extends IJobHandler {
|
||||
|
||||
|
||||
@Override
|
||||
public ReturnT<String> executeDataX(TriggerParam tgParam) throws Exception {
|
||||
public ReturnT<String> execute(TriggerParam tgParam) throws Exception {
|
||||
if (!glueType.isScript()) {
|
||||
return new ReturnT<String>(IJobHandler.FAIL.getCode(), "glueType["+ glueType +"] invalid.");
|
||||
}
|
||||
|
||||
@ -134,7 +134,7 @@ public class JobThread extends Thread {
|
||||
Thread futureThread = null;
|
||||
try {
|
||||
final TriggerParam tgParamT = tgParam;
|
||||
FutureTask<ReturnT<String>> futureTask = new FutureTask<ReturnT<String>>(() -> handler.executeDataX(tgParamT));
|
||||
FutureTask<ReturnT<String>> futureTask = new FutureTask<ReturnT<String>>(() -> handler.execute(tgParamT));
|
||||
futureThread = new Thread(futureTask);
|
||||
futureThread.start();
|
||||
|
||||
@ -150,7 +150,7 @@ public class JobThread extends Thread {
|
||||
}
|
||||
} else {
|
||||
// just execute
|
||||
executeResult = handler.executeDataX(tgParam);
|
||||
executeResult = handler.execute(tgParam);
|
||||
}
|
||||
|
||||
if (executeResult == null) {
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
package com.wugui.datax.admin.util;
|
||||
package com.wugui.datatx.core.util;
|
||||
|
||||
public final class Constant {
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
<parent>
|
||||
<groupId>com.wugui</groupId>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
<artifactId>datax-executor</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
@ -16,4 +16,10 @@ public class DataxOption {
|
||||
public static final String JVM_CM = "-j";
|
||||
|
||||
public static final String PARAMS_CM = "-p";
|
||||
|
||||
public static final String PARAMS_CM_V_PT = "-Dpartition=%s";
|
||||
|
||||
public static final String DEFAULT_JSON = "jsons";
|
||||
|
||||
public static final String DEFAULT_DATAX_PY = "bin/datax.py";
|
||||
}
|
||||
|
||||
@ -9,6 +9,8 @@ import com.wugui.datatx.core.handler.IJobHandler;
|
||||
import com.wugui.datatx.core.handler.annotation.JobHandler;
|
||||
import com.wugui.datatx.core.log.JobLogger;
|
||||
import com.wugui.datatx.core.thread.ProcessCallbackThread;
|
||||
import com.wugui.datatx.core.util.Constant;
|
||||
import com.wugui.datatx.core.util.DateUtil;
|
||||
import com.wugui.datatx.core.util.ProcessUtil;
|
||||
import com.wugui.datax.executor.util.SystemUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
@ -17,6 +19,7 @@ import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Date;
|
||||
import java.util.List;
|
||||
|
||||
@ -30,10 +33,6 @@ import java.util.List;
|
||||
@Component
|
||||
public class ExecutorJobHandler extends IJobHandler {
|
||||
|
||||
private static final String DEFAULT_JSON = "jsons";
|
||||
|
||||
private static final String DEFAULT_DATAX_PY = "bin/datax.py";
|
||||
|
||||
@Value("${datax.executor.jsonpath}")
|
||||
private String jsonpath;
|
||||
|
||||
@ -41,8 +40,7 @@ public class ExecutorJobHandler extends IJobHandler {
|
||||
private String dataXPyPath;
|
||||
|
||||
@Override
|
||||
public ReturnT<String> executeDataX(TriggerParam tgParam) throws Exception {
|
||||
|
||||
public ReturnT<String> execute(TriggerParam tgParam) throws Exception {
|
||||
int exitValue = -1;
|
||||
Thread inputThread = null;
|
||||
Thread errThread = null;
|
||||
@ -50,20 +48,7 @@ public class ExecutorJobHandler extends IJobHandler {
|
||||
//生成Json临时文件
|
||||
tmpFilePath = generateTemJsonFile(tgParam.getJobJson());
|
||||
try {
|
||||
String doc = buildStartCommand(tgParam.getJvmParam(), tgParam.getTriggerTime(), tgParam.getReplaceParam(), tgParam.getStartTime());
|
||||
JobLogger.log("------------------命令参数: " + doc);
|
||||
// command process
|
||||
//"--loglevel=debug"
|
||||
List<String> cmdarray = new ArrayList<>();
|
||||
cmdarray.add("python");
|
||||
String dataXHomePath = SystemUtils.getDataXHomePath();
|
||||
if (StringUtils.isNotEmpty(dataXHomePath)) dataXPyPath = dataXHomePath + DEFAULT_DATAX_PY;
|
||||
cmdarray.add(dataXPyPath);
|
||||
if (StringUtils.isNotBlank(doc)) {
|
||||
cmdarray.add(doc.replaceAll(DataxOption.SPLIT_SPACE, DataxOption.TRANSFORM_SPLIT_SPACE));
|
||||
}
|
||||
cmdarray.add(tmpFilePath);
|
||||
String[] cmdarrayFinal = cmdarray.toArray(new String[cmdarray.size()]);
|
||||
String[] cmdarrayFinal = buildCmd(tgParam, tmpFilePath);
|
||||
final Process process = Runtime.getRuntime().exec(cmdarrayFinal);
|
||||
String processId = ProcessUtil.getProcessId(process);
|
||||
JobLogger.log("------------------DataX运行进程Id: " + processId);
|
||||
@ -113,6 +98,22 @@ public class ExecutorJobHandler extends IJobHandler {
|
||||
}
|
||||
}
|
||||
|
||||
private String[] buildCmd(TriggerParam tgParam, String tmpFilePath) {
|
||||
// command process
|
||||
//"--loglevel=debug"
|
||||
List<String> cmdarray = new ArrayList<>();
|
||||
cmdarray.add("python");
|
||||
String dataXHomePath = SystemUtils.getDataXHomePath();
|
||||
if (StringUtils.isNotEmpty(dataXHomePath)) dataXPyPath = dataXHomePath + DataxOption.DEFAULT_DATAX_PY;
|
||||
cmdarray.add(dataXPyPath);
|
||||
String doc = buildDataXParam(tgParam);
|
||||
if (StringUtils.isNotBlank(doc)) {
|
||||
cmdarray.add(doc.replaceAll(DataxOption.SPLIT_SPACE, DataxOption.TRANSFORM_SPLIT_SPACE));
|
||||
}
|
||||
cmdarray.add(tmpFilePath);
|
||||
return cmdarray.toArray(new String[cmdarray.size()]);
|
||||
}
|
||||
|
||||
/**
|
||||
* 数据流reader（Input自动关闭，Output不处理）
|
||||
*
|
||||
@ -135,24 +136,47 @@ public class ExecutorJobHandler extends IJobHandler {
|
||||
}
|
||||
}
|
||||
|
||||
private String buildStartCommand(String jvmParam, Date triggerTime, String replaceParam, Date startTime) {
|
||||
private String buildDataXParam(TriggerParam tgParam) {
|
||||
StringBuilder doc = new StringBuilder();
|
||||
String jvmParam = tgParam.getJvmParam();
|
||||
String partitionStr = tgParam.getPartitionInfo();
|
||||
if (StringUtils.isNotBlank(jvmParam)) {
|
||||
doc.append(DataxOption.JVM_CM).append(DataxOption.TRANSFORM_QUOTES).append(jvmParam).append(DataxOption.TRANSFORM_QUOTES);
|
||||
}
|
||||
long tgSecondTime = triggerTime.getTime() / 1000;
|
||||
if (StringUtils.isNotBlank(replaceParam)) {
|
||||
long lastTime = startTime.getTime() / 1000;
|
||||
if (doc.indexOf(DataxOption.JVM_CM) != -1) doc.append(DataxOption.SPLIT_SPACE);
|
||||
doc.append(DataxOption.PARAMS_CM).append(DataxOption.TRANSFORM_QUOTES).append(String.format(replaceParam, lastTime, tgSecondTime)).append(DataxOption.TRANSFORM_QUOTES);
|
||||
long tgSecondTime = tgParam.getTriggerTime().getTime() / 1000;
|
||||
if (StringUtils.isNotBlank(tgParam.getReplaceParam())) {
|
||||
long lastTime = tgParam.getStartTime().getTime() / 1000;
|
||||
if (doc.length() > 0) doc.append(DataxOption.SPLIT_SPACE);
|
||||
doc.append(DataxOption.PARAMS_CM).append(DataxOption.TRANSFORM_QUOTES).append(String.format(tgParam.getReplaceParam(), lastTime, tgSecondTime));
|
||||
if (StringUtils.isNotBlank(partitionStr)) {
|
||||
doc.append(DataxOption.SPLIT_SPACE);
|
||||
List<String> partitionInfo = Arrays.asList(partitionStr.split(Constant.SPLIT_COMMA));
|
||||
doc.append(String.format(DataxOption.PARAMS_CM_V_PT, buildPartition(partitionInfo)));
|
||||
}
|
||||
doc.append(DataxOption.TRANSFORM_QUOTES);
|
||||
}else{
|
||||
if (StringUtils.isNotBlank(partitionStr)) {
|
||||
List<String> partitionInfo = Arrays.asList(partitionStr.split(Constant.SPLIT_COMMA));
|
||||
if (doc.length() > 0) doc.append(DataxOption.SPLIT_SPACE);
|
||||
doc.append(DataxOption.PARAMS_CM).append(DataxOption.TRANSFORM_QUOTES).append(String.format(DataxOption.PARAMS_CM_V_PT, buildPartition(partitionInfo))).append(DataxOption.TRANSFORM_QUOTES);
|
||||
}
|
||||
}
|
||||
JobLogger.log("------------------命令参数: " + doc);
|
||||
return doc.toString();
|
||||
}
|
||||
|
||||
private String buildPartition(List<String> partitionInfo) {
|
||||
String field = partitionInfo.get(0);
|
||||
int timeOffset = Integer.parseInt(partitionInfo.get(1));
|
||||
String timeFormat = partitionInfo.get(2);
|
||||
String partitionTime = DateUtil.format(DateUtil.addDays(new Date(), timeOffset), timeFormat);
|
||||
return field + Constant.EQUAL + partitionTime;
|
||||
}
|
||||
|
||||
private String generateTemJsonFile(String jobJson) {
|
||||
String tmpFilePath;
|
||||
String dataXHomePath = SystemUtils.getDataXHomePath();
|
||||
if (StringUtils.isNotEmpty(dataXHomePath)) jsonpath = dataXHomePath + DEFAULT_JSON;
|
||||
if (StringUtils.isNotEmpty(dataXHomePath)) jsonpath = dataXHomePath + DataxOption.DEFAULT_JSON;
|
||||
if (!FileUtil.exist(jsonpath)) FileUtil.mkdir(jsonpath);
|
||||
tmpFilePath = jsonpath + "jobTmp-" + IdUtil.simpleUUID() + ".conf";
|
||||
// 根据json写入到临时本地文件
|
||||
|
||||
@ -22,7 +22,7 @@ import java.io.File;
|
||||
public class KillJobHandler extends IJobHandler {
|
||||
|
||||
@Override
|
||||
public ReturnT<String> executeDataX(TriggerParam tgParam) throws Exception {
|
||||
public ReturnT<String> execute(TriggerParam tgParam) throws Exception {
|
||||
String processId = tgParam.getProcessId();
|
||||
boolean result = ProcessUtil.killProcessByPid(processId);
|
||||
// 删除临时文件
|
||||
|
||||
@ -28,13 +28,13 @@ public class SystemUtils {
|
||||
if (StringUtils.isNotEmpty(DATAX_HOME)) return DATAX_HOME;
|
||||
String dataXHome = System.getenv("DATAX_HOME");
|
||||
if (StringUtils.isBlank(dataXHome)) {
|
||||
LOGGER.warn("DATAX_HOME 环境变量为NULL");
|
||||
//LOGGER.warn("DATAX_HOME 环境变量为NULL");
|
||||
return null;
|
||||
}
|
||||
DATAX_HOME = System.getProperty("os.name").contains("Windows") ?
|
||||
(!dataXHome.endsWith("\\") ? dataXHome.concat("\\") : dataXHome) :
|
||||
(!dataXHome.endsWith("/") ? dataXHome.concat("/") : dataXHome);
|
||||
LOGGER.info("DATAX_HOME:{}", DATAX_HOME);
|
||||
//LOGGER.info("DATAX_HOME:{}", DATAX_HOME);
|
||||
return DATAX_HOME;
|
||||
}
|
||||
}
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
<parent>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<groupId>com.wugui</groupId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
<artifactId>datax-registry</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
<parent>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<groupId>com.wugui</groupId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
<artifactId>datax-rpc</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
@ -31,7 +31,7 @@ datax.json
|
||||
"parameter": {
|
||||
"defaultFS": "hdfs://localhost:9000",
|
||||
"fileType": "text",
|
||||
"path": "/user/hive/warehouse/offline.db/fgw_company_evaluate_gg/datety=2019-12-24",
|
||||
"path": "/user/hive/warehouse/offline.db/fgw_company_evaluate_gg/datety=2019-12-22",
|
||||
"fileName": "test",
|
||||
"column": [
|
||||
{
|
||||
|
||||
93
doc/datax-web/分区动态传参使用.md
Normal file
93
doc/datax-web/分区动态传参使用.md
Normal file
@ -0,0 +1,93 @@
|
||||
# DataX增量动态参数抽取数据到动态参数分区
|
||||
|
||||
## DataX Web配置
|
||||
|
||||
|
||||
## DataX Json配置
|
||||
|
||||
datax.json
|
||||
|
||||
```json
|
||||
{
|
||||
"job": {
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "mysqlreader",
|
||||
"parameter": {
|
||||
"column": [
|
||||
"*"
|
||||
],
|
||||
"connection": [
|
||||
{
|
||||
"jdbcUrl": [
|
||||
"jdbc:mysql://localhost:3306/order?useUnicode=true&characterEncoding=utf-8&useSSL=false&rewriteBatchedStatements=true"
|
||||
],
|
||||
"querySql": [
|
||||
"select * from test_order where updateTime >= FROM_UNIXTIME(${lastTime}) and operationDate < FROM_UNIXTIME(${currentTime})"
|
||||
]
|
||||
}
|
||||
],
|
||||
"password": "root",
|
||||
"username": "root"
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "hdfswriter",
|
||||
"parameter": {
|
||||
"defaultFS": "hdfs://localhost:9000",
|
||||
"fileType": "text",
|
||||
"path": "/user/hive/warehouse/offline.db/test_order/${partition}",
|
||||
"fileName": "test_order",
|
||||
"column": [
|
||||
{
|
||||
"name": "keyno",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"name": "code",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"name": "province",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"writeMode": "append",
|
||||
"fieldDelimiter": ","
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- 增量时间字段：${lastTime}增量开始时间, 注意：定时任务启动之后，第一次的开始时间为页面输入时间，当任务执行成功后，该时间被更新为上一次的任务触发时间，任务失败不更新。${currentTime}任务的触发时间。
|
||||
- 分区字段：${partition}为固定格式，不能自定义。
|
||||
- 拼接结果： -p"-DlastTime=1572537600 -DcurrentTime=1579317145 -Dpartition=datety=2020-01-18"。
|
||||
- JVM启动参数拼接结果为： -j "-Xms2G -Xmx2G"。
|
||||
|
||||
## Datax启动命令
|
||||
|
||||
```shell
|
||||
python datax.py -j "-Xms2G -Xmx2G" -p "-DlastTime=1577009172 -DcurrentTime=1579317145 -Dpartition=datety=2020-01-18" datax.json
|
||||
```
|
||||
|
||||
@ -290,3 +290,12 @@ CREATE TABLE `job_template` (
|
||||
*/
|
||||
ALTER TABLE `datax_web`.`job_jdbc_datasource`
|
||||
ADD COLUMN `datasource` VARCHAR(45) NOT NULL COMMENT '数据源' AFTER `datasource_name`;
|
||||
|
||||
/**
|
||||
添加分区字段
|
||||
*/
|
||||
ALTER TABLE `datax_web`.`job_info`
|
||||
ADD COLUMN `partition_info` VARCHAR(100) NULL DEFAULT NULL COMMENT '分区信息' AFTER `inc_start_time`;
|
||||
|
||||
ALTER TABLE `datax_web`.`job_template`
|
||||
ADD COLUMN `partition_info` VARCHAR(100) NULL DEFAULT NULL COMMENT '分区信息' AFTER `inc_start_time`;
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 92 KiB |
BIN
doc/img/datax.png
Normal file
BIN
doc/img/datax.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
doc/img/partition.png
Normal file
BIN
doc/img/partition.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
BIN
doc/img/powershell.png
Normal file
BIN
doc/img/powershell.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
BIN
doc/img/python.png
Normal file
BIN
doc/img/python.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
BIN
doc/img/shell.png
Normal file
BIN
doc/img/shell.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Loading…
Reference in New Issue
Block a user