Introduction

Today, we will cover how to scrape data from any website using Python’s library Scrapy. We will then save the data in a JSON and HTML file. Finally, we will see how we can also store this data in GridDB for long-term and efficient use.

Pre-requisites

This post requires the prior installation of the following:

We also recommend installing Anaconda Navigator, if not already installed. Anaconda provides a large range of tools for data scientists to experiment with. Also, a virtual environment can help you meet the specific version requirements while running an application without interfering with the actual system paths.

Creating a new project using Scrapy

…

Introduction

In a recent blog, we covered querying geospatial data with manual bounding boxes. In this blog, we’ll take the same dataset/query but perform the query using GeoHashes instead.

GeoHashes are alpha numeric encodings of areas on earth. The more bits in the encoding, the more precise the area is prescribing. A 1-digit long Geohash covers approximately 2500km while an 8-digit long Geohash covers approximately 20m. For example, the 9q geohash covers most of California and the Western United States while San Francisco / San Jose bay area is spread over several geohashes, 9qb, 9qc, 9q8, and 9q9.

Introduction

Today, we will be building a Bank Loan Classification model from scratch using the data stored in GridDB. In this post, we will cover the following:

1. Storing the data in GridDB
2. 2. Extracting the data from GridDB
3. 3. Building a Logistic Regression Model using Pandas
4. 4. Evaluating our model using heat map and correlation matrix

We will begin with installing the prerequisites and setting up our environment. We will be using GridDB’s Python connector for this tutorial as the primary language used to model building is Python. …

Introduction: What is a Phishing Website?

Curiosity alone can lead to getting your personal information leaked to bad actors. Are you the type that just clicks on any link a friend sends to you? It’s dangerous! Hackers have so many ways of gathering information — they can gain your trust by social engineering and make you to do the things you would not do if it were just asked by a normal friend, and one of them is making you click on links.

Phishing is a great technique used by hackers to gather information: passwords, emails, name, etcetera. …

In this project, we use GridDB to create a Machine Learning platform where we Kafka is used to import stock market data from Alphavantage, a market data provider. Tensorflow and Keras train a model that is then stored in GridDB, and then finally uses LSTM prediction to find anomalies in daily intraday trading history. The last piece is that the data is visualized in Grafana and then we configure GridDB to send notifications via its REST Trigger function to Twilio’s Sendgrid.

The actual machine learning portion of this project was inspired by posts on Towards Data Science and Curiously. This…

Introduction

In this tutorial, we will build a trivial linear regression model with the data stored in GridDB. We will begin with GridDB’s python-connector to insert and access the data. Afterwards, we will see how to retrieve and convert the data using pandas and numpy. In the end, we will train and visualize our regression model using scikit-learn and matplotlib.

Prerequisites and Environment Setup

The following tutorial is carried out on Ubuntu Operating system (v. 18.04) with gcc version 7.5.0. GridDB (v. 4.5.2) has been installed using their documentation available on Github.

To enable the python connector for GridDB, prerequisites include GridDB C-client, SWIG and…

Take a look at this half-tongue-in-cheek, half-heartbreaking website: http://www.iscaliforniaonfire.com/. Though I suspect it started out as a joke, the web page helps to illustrate — by sheer power of existence — the constant, existential threat faced by the Golden State; the state-wide wildfires can wreak havoc in countless ways, but today we want to focus on air quality. Specifically we want to look at whether carbon monoxide (CO) and/or Nitrogen dioxide (NO2) emissions rise when California is on fire, and by how much.

The full source code for this project can be found at the bottom of this page: FULL…

Introduction and Purpose

The Use Case

Imagine that you are the owner of a building complex. You have installed smart meters all over the place to monitor power usage. Each of the devices creates a timestamp two times in a minute. It saves power usage data in kW and timestamps in the form of epoch seconds. The IoT provider sends you a .CSV file every month.

Objectives

You want to propose an energy-saving plan. To start, you’d like to identify patterns in power usage: how the power is consumed, when less of it is consumed, and when power usage is particularly extensive. …