参考:https://www.cnblogs.com/LQBlog/p/12177295.html#autoid-3-0-0
源码修改参考:https://www.cnblogs.com/LQBlog/p/14661238.html
server: port: 8085 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null canal.conf: mode: tcp # kafka rocketMQ 数据来源TCP则是binlog 其他则增量数据来源是kafka和rocketMQ #canalServerHost: 192.168.20.5:11111 #canal 单机地址和端口 zookeeperHosts: 192.168.20.4:2181 #基于zookeeper集群 batchSize: 500 syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: subscribe: merge_test.pro_brand,merge_test.soa_ord_order_summary,merge_test.soa_ord_order,merge_test.soa_ord_order_item #我自己改源码加的防止订阅不关心的消息 srcDataSources: defaultDS: url: jdbc:mysql://127.0.0.1:3306/merge_test?useUnicode=true #数据库 username: kuaihe password: Kuaihe0910Mysql canalAdapters: - instance: kuaihe_db_test # canal instance Name or mq topic name 对应destance创建的文件夹 groups: - groupId: g1 outerAdapters: - name: es7.4.0 #适配器名称 hosts: 127.0.0.1.1:9200 # es 集群地址, 逗号分隔 properties: mode: rest # transport # or rest security: auth: elastic:T3StdABAk #es用户名密码
我是基于源码自己打包 所以 直接在源码修改打包 启动会自动扫描es 目录下的所有yml
pro_brand.yml例子 我的是单表同步 其他复杂同步 可以看官方文档
dataSourceKey: defaultDS # 源数据源的key, 对应上面配置的srcDataSources中的值 #outerAdapterKey: exampleKey # 对应application.yml中es配置的key destination: kuaihe_db_test # cannal的instance或者MQ的topic 源码com.alibaba.otter.canal.client.impl.SimpleCanalConnector.getWithoutAck(int, java.lang.Long, java.util.concurrent.TimeUnit) groupId: g1 # 对应MQ模式下的groupId, 只会同步对应groupId的数据 esMapping: _index: pro_brand _type: _doc _id: id pk: id sql: " select id,create_timestamp,update_timestamp,`code`,brand_code,brand_mall_idx,delete_flag,depot_id,description,idx,kuaihe_idx,logo,`name`,name_en,prefix,sale_channel,seo_description,seo_keywords,seo_title,`status`,vendor_id from pro_brand " commitBatch: 3000
导的时候记得看日志 有没有报错
post:http://192.168.20.5:8085/etl/es7.4.0/soa_ord_order_item.yml
{ "succeeded": true, "resultMessage": "导入ES 数据:61513 条" }
增量是启动自动触发binlog订阅
canal-adapter-进行全量和增量到ES7.*(六)
原文:https://www.cnblogs.com/LQBlog/p/14661570.html