文章 二月 25, 2018

通过logstash-input-jdbc将mysql数据导入es5.x

文章字数 2.7k 阅读约需 2 mins. 阅读次数 0

logstash-input-jdbc是logstash的一个jdbc插件

1.在es5.x中通过如下命令安装该插件

cd /Users/xy/Workspace/search/logstash/logstash-5.3.0
bin/logstash-plugin install logstash-input-jdbc
Validating logstash-input-jdbc
Installing logstash-input-jdbc
Installation successful

2.配置一个mysql计划任务的logstash config文件

cd /Users/xy/Workspace/search/logstash/logstash-5.3.0/config
touch mysqlsync.conf

2.1文件内容如下

input {
    stdin {
    }
    jdbc {
      # mysql jdbc connection string to our backup databse
      jdbc_connection_string => "jdbc:mysql://localhost:3306/btdb"
      # the user we wish to excute our statement as
      jdbc_user => "root"
      jdbc_password => "123456"
      # the path to our downloaded jdbc driver
      jdbc_driver_library => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/bin/mysql-connector-java-5.1.41.jar"
      # the name of the driver class for mysql
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_default_timezone => "Asia/Shanghai"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      statement_filepath => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/config/mysqlsync.sql"
      schedule => "* * * * *"
      type => "jdbc"
      use_column_value => true
      tracking_column => time_created
      record_last_run => true
      last_run_metadata_path => "/Users/xy/Workspace/search/logstash/logstash-5.3.0/config/last_run_metadata"
    }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "tmtpost"
        document_type => "auction"
        document_id => "%{id}"
    }
    stdout {
        codec => json_lines
    }
}

2.2output中的document_id是指定的mysqlsync.sql中的一个字段名,一般使用主键就行

2.3input中的jdbc_driver_library从下面地址中下载

https://repo1.maven.org/maven2/mysql/mysql-connector-java/

2.4.statement_filepath指向的是需要同步的sql语句,logstash会监测sql语句中指定的table的update和insert操作,mysqlsync.sql示例如下

select id, name as title,body as summary, time_created from a

3.配置完成后在logstash根目录启动服务

bin/logstash -f /Users/xy/Workspace/search/logstash/logstash-5.3.0/config/mysqlsync.conf

看到如下输出则代表启动成功

试着更新mysql表中数据看看是否有同步到es中


上一篇:
下一篇:
0%