Using Fluentd to Push Data to GridDB | GridDB: Open Source Time Series Database for IoT

Fluentd is an open source data collector that allows you to easily ingest data into GridDB;
this data is most typically information generated by edge devices or other sources not in your
local network. Fluentd supports multiple input sources including HTTP, MQTT and more.
Fluentd also has many different output options, including GridDB.

The GridDB output plugin relies on the GridDB WebAPI, this post will cover the installation, configuration, and usage of Fluentd, GridDB WebAPI, and the GridDB Plugin for Fluentd.

If you don’t have GridDB running already, the quick start is here. To install Fluentd, follow
the guides for CentOS or Debian/Ubuntu. While following these guides, you’ll also install ruby and gem which will be required to build the GridDB plugin for Fluentd. To install the WebAPI, download and untar the release and then copy the webapi folder to $GS_HOME

$ sudo cp -a webapi /var/lib/gridstore
$ sudo chown gsadm.gridstore /var/lib/gridstore/webapi

Then edit conf/repository.json to reflect your GridDB cluster configuration. You will likely
only need to set the cluster name.

{ "clusters" : [ { "name" : "defaultCluster", "mode" : "MULTICAST", "address" : "239.0.0.1", "port" : 31999, "jdbcAddress" : "239.0.0.1", "jdbcPort" : 41999, "transactionMember": "", "sqlMember": "", "providerUrl": null } ] }

Now to run WebAPI:

$ sudo su - gsadm export CLASSPATH=:/usr/share/java/gridstore.jar 
$ cd webapi/ 
$ java -jar lib/griddb-webapi-ce-2.1.0.jar 2>&1 /dev/null &

Check $GS_HOME/logs/gs_webapi*.log for errors, if it has successfully started, the logs should
contain:

[main] org.springframework.boot.web.embedded.tomcat.TomcatWebServer: Tomcat started on port(s): 8080 (http) with context path '' [main] org.springframework.boot.StartupInfoLogger: Started Application in 2.677 seconds (JVM running for 3.278)

Now that the GridDB WebAPI is installed and running, it’s time to install the GridDB Fluentd
connector. After downloading and un-tarring the release, run:

$ cd fluent-plugin-griddb-1.0.0/ 
$ /opt/td-agent/embedded/bin/gem build fluent-plugin-griddb.gemspec 
$ /opt/td-agent/embedded/bin/gem install --force --local fluent-plugin-griddb-1.0.0.gem

The simplest GridDB configuration follows, this chunk would be added to /etc/td-agent/td-agent.conf and then the td-agent will need to be restarted.

@type griddb host http://localhost:8080/ cluster defaultCluster database public container container_1 username admin password admin

Before Fluentd can write data to GridDB, you need to create the container using the GridDB
WebAPI:

$ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"container_name":"container_1", "container_type":"TIME_SERIES", "rowkey":true, "columns":[{"name": "date", "type": "TIMESTAMP" }, {"name": "value", "type": "DOUBLE" },{"name": "str", "type": "STRING" }] }' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers

With the GridDB plugin in place, you can POST data to Fluentd:

$ curl -X POST -d 'json={"date":"2018-09-20T12:08:21.112Z", "value":"4.23", "str":"Hello World"}' http://localhost:8888/griddb And using the GridDB WebAPI, we can see the data we POSTed to FluentD: $ curl -X POST --basic -u admin:admin -H "Content-type:application/json" -d '{"limit":1000}' http://localhost:8080/griddb/v2/defaultCluster/dbs/public/containers/conta iner_1/rows

This is just the beginning of what you can do with Fluentd and its GridDB plugin. For more information, please check out the detailed documentation.

Originally published at https://griddb.net on January 21, 2020.