Mapping the UK’s Traffic Accident Hotspots

While looking for some interesting geographical data to work with, I came across the Road Safety Data published by the UK government. This is a very comprehensive road accident data set that includes the incident’s geographical coordinates, as well as other related data such as the local weather conditions, visibility, police attendance and more. There … Read more Mapping the UK’s Traffic Accident Hotspots

What Does It Really Mean to Operationalize a Predictive Model?

It is not enough to just stand up a web service that can make predictions. Aug 13, 2018 Original Image Source — Meme overlay by Imgflip In a 2017 SAS survey, 83% of organizations have made moderate-to- significant investments in big data, but only 33% say they have derived value from their investments. Other more recent surveys have … Read more What Does It Really Mean to Operationalize a Predictive Model?

Practical tips for class imbalance in binary classification

4. Class weighted / cost sensitive learning Without resampling the data, one can also make the classifier aware of the imbalanced data by incorporating the weights of the classes into the cost function (aka objective function). Intuitively, we want to give higher weight to minority class and lower weight to majority class. scikit-learn has a … Read more Practical tips for class imbalance in binary classification

Feature Engineering for Healthcare Fraud Detection

The nature of the problem: medical fraud and abuse The U.S. department of health and human services in a pamphlet Avoiding Medicare Fraud and Abuse: A Roadmap for Physicians states “most physicians strive to work ethically, render high-quality medical care to their patients, and submit proper claims for payment,” yet “the presence of some dishonest … Read more Feature Engineering for Healthcare Fraud Detection

Azure SQL DWH – Overview

There are a multitude of options when it comes to storing and processing data. In this post I want to give you a brief overview of Azure SQL datawarehouse, Microsoft’s datawareshouse solution for the Azure cloud and its answer to Amazon Redshift on AWS. I will start of by talking briefly about its technical architecture … Read more Azure SQL DWH – Overview

Math Behind Reinforcement Learning, the Easy Way

Aug 2, 2018 Photo by JESHOOTS.COM on Unsplash Look at this equation: Value function of Reinforcement Learning If it does not intimidate you, then you are a mathematical savvy and there is no point in reading this article 🙂 This article is not about teaching Reinforcement Learning (RL) but about explaining the math behind it. So it … Read more Math Behind Reinforcement Learning, the Easy Way

Cooking with Machine Learning: Dimension Reduction

Recently I came across this cooking recipes data set in Kaggle, and it inspired me to combine 2 of my main interests in life. Food and machine learning. What makes this data set special is that it contains recipes from 20 different cuisines, 6714 different ingredients, but only 26648 samples. Some cuisines have way fewer … Read more Cooking with Machine Learning: Dimension Reduction

An In-depth Review of Andrew Ng’s deeplearning.ai Speciliazation

So you’ve seen the recent news about how artificial intelligence (AI) is changing everything. However, the idea of AI has been around for a long time. Machines that think and talk like humans have been the inspiration for movies and stories for decades. But what’s the deal? Why has AI been getting better and better … Read more An In-depth Review of Andrew Ng’s deeplearning.ai Speciliazation

An Advanced Example of Tensorflow Estimators Part (1/3)

Estimators were introduced in version 1.3 of the Tensorflow API, and are used to abstract and simplify training, evaluation and prediction. If you haven’t worked with Estimators before I suggest to start by reading this article and get some familiarity as I won’t be covering all of the basics when using estimators. In no means … Read more An Advanced Example of Tensorflow Estimators Part (1/3)

Hypothesis Analysis Explained

Jul 19, 2018 Hypothesis analysis is a widely known concept and is used extensively by researchers, statisticians and quantitative analysts. It allows them to follow a set of formal steps to perform calculated analysis on their data. It is also widely used in machine learning and artificial intelligence. In this article, I will be explaining core concepts of … Read more Hypothesis Analysis Explained

PySpark ML and XGBoost full integration tested on the Kaggle Titanic dataset

Jul 8, 2018 In this tutorial we will discuss about integrating PySpark and XGBoost using a standard machine learing pipeline. We will use data from the Titanic: Machine learning from disaster one of the many Kaggle competitions. Before getting started please know that you should be familiar with Apache Spark and Xgboost and Python. The … Read more PySpark ML and XGBoost full integration tested on the Kaggle Titanic dataset

More advanced SQL Server for Data Scientists

In the previous post I covered the basics you need to know to work with SQL Server. In this post, I want to show you some more advanced techniques that I found pretty helpful. The topics I will cover include: How to speed up your queries with indices and using columnstore Using Views and Table … Read more More advanced SQL Server for Data Scientists

Acoustic Noise Cancellation by Machine Learning

DIY Noise-Cancellation System prototype made with TensorFlow. Jun 25, 2018 Image by TheDigitalArtist on Pixabay In this post I describe how I built an active noise cancellation system by means of neural networks on my own. I’ve just got my first results which I am sharing, but the system looks like a ravel of scripts, binaries, … Read more Acoustic Noise Cancellation by Machine Learning

Scrum PSM I

After getting scrum.org the PSM I I wanted to capture the relevant content. The complete guido can be downloaded here: scrumguides.org 1. What is Scrum? Scrum is a framework for developing and sustaining complex products. A framework in which complex adaptive problems can be addressed. It is lightweight, simple to understand and yet difficult to … Read more Scrum PSM I

Finding Good Learning Rate and The One Cycle Policy.

Introduction Learning rate might be the most important hyper parameter in deep learning, as learning rate decides how much gradient to be back propagated. This in turn decides by how much we move towards minima. The small learning rate makes model converge slowly, while the large learning rate makes model diverge. So, the learning rate … Read more Finding Good Learning Rate and The One Cycle Policy.

Recommendation Systems — Models and Evaluation

I’ve been involved in building several different types of recommendation systems, and one thing I’ve noticed is that each use case is different from the next, as each aims to solve a different business problem. Let’s consider a few examples: Movie/Book/News Recommendations — Suggest new content that increases user engagement. The aim is to introduce users to … Read more Recommendation Systems — Models and Evaluation

R vs Python: Image Classification with Keras

Many data professionals are strict on the language to be used for ANN models limiting their dev. environment exclusively to Python. I decided to test performance of Python vs. R in terms of time required to train a convolutional neural network based model for image recognition. As the starting point, I took the blog post … Read more R vs Python: Image Classification with Keras

IoT Made Easy: ESP-MicroPython-MQTT-ThingSpeak

Using MQTT protocol, we will get captured data from sensors, logging them to an IoT service, ThingSpeak.com and to a mobile App, Thingsview. 1. Introduction In my previous article, MicroPython on ESP using Jupyter, we learned how to install and run MicroPython on an ESP device. Using Jupyter Notebook as our development environment, we also … Read more IoT Made Easy: ESP-MicroPython-MQTT-ThingSpeak

From Git to Colab, via SSH

When you are using Google’s Colaboratory (Colab) for running your Deep Learning models the most obvious way to access the large datasets is by storing them on Google Drive and then mounting Drive onto the Colab environment. But a lot of open sourced large datasets that are available for research purposes, are hosted on Github/Gitlab. … Read more From Git to Colab, via SSH

Object Oriented Programming in Data Science with R

Since R is mostly a functional language and data science work lends itself to be expressed in a functional form you can come by just fine without learning about object-oriented programming. Personally, I mostly follow a functional programming style (although often not a pure one, i.e. w/o side-effects, because of limited RAM). Expressing mathematical concepts in … Read more Object Oriented Programming in Data Science with R

DevOps: To do or not to do?

Over the past few decades, four key change initiatives have been taking place in the organizations: strategic planning, re-engineering, total quality management and downsizing. The aim of these initiatives was to achieve economic effectiveness, but around 75% of them failed or created problems that were serious enough to threaten organization’s survival (1). It has been … Read more DevOps: To do or not to do?

Estimating Intervention Effects using Baysian Models in R

Measuring the effect of an intervention on some metric is an important problem in many areas of business and academia. Imagine, you want to know the effect of a recently launched advertising campaign on product sales. In an ideal setting, you would have a treatment and a control group so that you can measure the … Read more Estimating Intervention Effects using Baysian Models in R

A Framework to tackle tough Data Science Problems

One of the things I particularly like about working in data science, is the science part: Figuring out the right questions to ask, how to frame a problem correctly and finally trying to solve it. While there are many problems that you can simply solve by library(caret) or from sklearn import * and dumping your … Read more A Framework to tackle tough Data Science Problems

The Mathematics of Decision Trees, Random Forest and Feature Importance in Scikit-learn and Spark

This post attempts to consolidate information on tree algorithms and their implementations in Scikit-learn and Spark. In particular, it was written to provide clarification on how feature importance is calculated. There are many great resources online discussing how decision trees and random forests are created and this post is not intended to be that. Although … Read more The Mathematics of Decision Trees, Random Forest and Feature Importance in Scikit-learn and Spark

Package development in R – Overview

Creating an R package is as easy as typing: package.skeleton(name = “YourPackageName”) As you might have guessed, this function creates the basic file and folder structure you need to create an R package. You will get: YourPackageName/ DESCRIPTION man/ NAMESPACE R/ You can also use RStudio to create a package with File > New Project … Read more Package development in R – Overview

Agile Project Management for Data Science

Many data scientists are former academics who are used to working on a specific and often quite narrow research problems for long periods of time, often years. With data science being in high demand at the moment in nearly all industries, more and more researchers switch from an academic career to one in the private … Read more Agile Project Management for Data Science

Implementing QANet (Question Answering Network) with CNNs and self attentions

Apr 15, 2018 In this post, we will tackle one of the most challenging yet interesting problems in Natural Language Processing, aka Question Answering. We will implement Google’s QANet in Tensorflow. Just like its machine translation counterpart Transformer network, QANet doesn’t use RNNs at all which makes it faster to train / test. I’m assuming … Read more Implementing QANet (Question Answering Network) with CNNs and self attentions

What I wish I’d done differently as a data science manager

On centralizing siloed data Apr 12, 2018 I still get nostalgic looking at the very first Pebbles. (Photo courtesy of Pebble’s first Kickstarter) In 2014, I joined Pebble, the smartwatch maker later acquired by Fitbit, to lead their data science & analytics team. I was interested in the challenges of managing a data organization at a … Read more What I wish I’d done differently as a data science manager

Machine Learning for People Who Don’t Care About Machine Learning

Greg Lamp, previous co-founder of the data science startup Yhat, and current co-founder & CTO of Waldo shares his thoughts on Machine Learning for those of us who just don’t care about Machine Learning. What is Machine Learning? The definition I have come up with for Machine Learning is as follows… machine learning is using … Read more Machine Learning for People Who Don’t Care About Machine Learning

Hierarchical Clustering on Categorical Data in R

Dissimilarity MatrixArguably, this is the backbone of your clustering. Dissimilarity matrix is a mathematical expression of how different, or distant, the points in a data set are from each other, so you can later group the closest ones together or separate the furthest ones — which is a core idea of clustering. This is the step where … Read more Hierarchical Clustering on Categorical Data in R

Coding the Matrix

How to test the solutions python3 submit.py python_lab.py Lab 1: Introduction to Python—sets, lists, dictionaries, and comprehensions Python provides some simple data structures for grouping together multiple values, and integrates them with the rest of the language. These data structures are called collections. Sets A set is an unordered collection in which each value occurs … Read more Coding the Matrix

Which Leading Artificial Intelligence Course Should You Take and What Should You Do After?

4. Course Content The content of each course is world-class. I’ve frequently stated in my videos these are the best courses I’ve ever taken. The DLND is broken into six parts with five of the parts having significant projects attached. 1. Introduction2. Neural Networks — creating your first neural network.3. Convolutional Neural Networks — building … Read more Which Leading Artificial Intelligence Course Should You Take and What Should You Do After?

Automatic GPUs

A reproducible R / Python approach to getting up and running quickly on GCloud with GPUs in Tensorflow “A high view of a sea of clouds covering a mountain valley in the Dolomites” by paul morris on Unsplash Backstory After completing Google’s excellent Data Engineering Certified Specialization on Coursera recently (*which I highly recommend), I … Read more Automatic GPUs

Crossing Your Data Science Chasm

An analytics roadmap for growth Scenario — You’re an up-and-coming ecommerce/SaaS startup. You’ve got your site up, you have A/B tested your message, and you’ve got your SEO, and social ad buys. You’ve set up your email drip campaign and reminders. You also have basic BI reporting telling you channel traffic and conversions. Traffic is … Read more Crossing Your Data Science Chasm

Python WebServer With Flask and Raspberry Pi

Let’s create a simple WebServer to control things in your home. There are a lot of ways to do that. For example, on my tutorial: IoT — Controlling a Raspberry Pi Robot Over Internet With HTML and Shell Scripts Only, we have explored how to control a robot over the local network using the LIGHTTPD WebServer. For … Read more Python WebServer With Flask and Raspberry Pi

Parallel processing in R using Azure Batch and Docker

While (personal) computers have become increasingly powerful over the last years there are still lots of workloads that easily bring even the best workstation to its knees. Running huge Monte-Carlo simulations or training thousands of models takes hours, if not days even on very beefy machines. Now enter Azure Batch processing. Azure Batch is a … Read more Parallel processing in R using Azure Batch and Docker

Azure Container Registry – Quick Start Guide

Azure Container Registry is the Microsoft equivalent to private Dockerhub repositories. First, I will show you how to quickly push an image to Azure Container Registry. In a second step, I will cover how to manage your registries and repositories using the PowerShell cmdlet AzureRM as well as the Azure CLI. Quick start To push … Read more Azure Container Registry – Quick Start Guide

Azure Machine Learning Services – Overview

We rely heavily on Microsoft’s cloud platform Azure during for our analytics workloads at the Austrian Postal Service. Azure has grown rapidly over the past few years and is adding features at a very fast pace, so it is easy to lose track which services are (still) offered and what services one should use . … Read more Azure Machine Learning Services – Overview

Quick implementation of Yolo V2 with Keras!

Feb 22, 2018 I do not hold ownership to any of the above pictures. These are merely used for educational purposes to describe the concepts. Real time multiple object localization remains a grand debate in the field of digital image processing since many years. With the invent of Deep Learning and convolutional neural networks, the … Read more Quick implementation of Yolo V2 with Keras!