Image Processing Class (EGBE443) #2 -Histogram

Computing the histogram In this section, the histogram was calculated by implementation of python programming code (Python 3.6). For python 3.6, There are a lot of common modules using in image processing such as Pillow, Numpy, OpenCV, etc. but in this program Pillow and Numpy module was used. To import the image from your computer, … Read more

Lazy Neural Networks

Before I get into solutions I think it is important to discuss some overarching themes of deep learning. Training Objectives Remember that when we create a neural network, what we are effectively doing is designing an experiment. We have data, a model architecture and a training objective. The data you provide is the models universe … Read more

How to get fbprophet working on AWS Lambda

Solving package size issues of fbprophet serverless deployment Adi Goldstein / Unsplash I assume you’re reading this post because you’re looking for ways to use the awesome fbprophet (Facebook open source forecasting) library on AWS Lambda and you’re already familiar with the various issues around getting it done. I will be using a python 3.6 … Read more

Multi-Layer perceptron using Tensorflow

Sep 11, 2018 In this blog, we are going to build a neural network(multilayer perceptron) using TensorFlow and successfully train it to recognize digits in the image. Tensorflow is a very popular deep learning framework released by, and this notebook will guide for build a neural network with this library. If you want to understand … Read more

Diving into K-Means…

Sep 9, 2018 We have completed our first basic supervised learning model i.e. Linear Regression model in the last post here. Thus in this post we get started with the most basic unsupervised learning algorithm- K-means Clustering. Let’s get started without further ado! Background: K-means clustering as the name itself suggests, is a clustering algorithm, … Read more

3 approaches for backtesting historical data

Reading and processing data for statistical and quantitative analysis in trading Sep 8, 2018 Anyone interested in the statistical analysis of financial markets has the need to process historical data. Historical data is needed in order to backtest or train: Quantitative trading. Statistical trading. Price action replay/walkthrough. Each need comes from different goals. 3 examples on … Read more

Microsoft Big Data Overview

https://academy.microsoft.com/en-us/professional-program/tracks/big-data/ Block 1 – Data Fundamentals Learn data science basics. Explore topics like data queries, data analysis, data visualization and how statistics informs data science practices. Please choose from Course 2a or Course 2b to complete the unit. Course 1: Microsoft Professional Program: Introduction to Big Data Course 2a: Analyzing and Visualizing Data with Power … Read more

Box Cox Transformation

When we do time series analysis, we are usually interested either in uncovering causal relationships (Does \(X_t\) influence \(Y_{t+1}\)?) or in getting the most accurate forecast possible. Especially in the second case it can be beneficial to transform our historical data to make it easier to extract a signal. A very common transformation is to … Read more

Doing XGBoost hyper-parameter tuning the smart way — Part 1 of 2

Aug 29, 2018 Picture taken from Pixabay In this post and the next, we will look at one of the trickiest and most critical problems in Machine Learning (ML): Hyper-parameter tuning. After reviewing what hyper-parameters, or hyper-params for short, are and how they differ from plain vanilla learnable parameters, we introduce three general purpose discrete optimization … Read more

Introduction to stochastic control theory

I had my first contact with stochastic control theory in one of my Master’s courses about Continuous Time Finance. I found the subject really interesting and decided to write my thesis about optimal dividend policy which is mainly about solving stochastic control problems. In this post I want to give you a brief overview of … Read more

Automatic Image Quality Assessment in Python

Aug 28, 2018 Image quality is a notion that highly depends on observers. Generally, it is linked to the conditions in which it is viewed; therefore, it is a highly subjective topic. Image quality assessment aims to quantitatively represent the human perception of quality. These metrics are commonly used to analyze the performance of algorithms in … Read more

Neural Processes: Probabilistic Gaussian Process+Deep Learning

Neural Processes (NPs) caught my attention as they essentially are a neural network (NN) based probabilistic model which can represent a distribution over stochastic processes. So NPs combine elements from two worlds:

Deep Learning – neural networks are flexible non-linear functions which are straightforward to train
Gaussian Processes – GPs offer a probabilistic framework for learning a distribution over a wide class of non-linear functions

Categories Featured ExcerptFavorite

Opensourcing TransmogrifAI: Automated ML for Structured Data

Despite huge progress in machine learning over the past decade, building production-ready machine learning systems is still hard. Three years ago when we set out to build machine learning capabilities into the Salesforce platform, we learned that building enterprise-scale machine learning systems is even harder.

Categories Featured ExcerptFavorite

Program Sythesis: Can We Teach Computers to Write Code?

Can we teach computers to write code? This is the question that brings out an entire branch of research specialized in program synthesis. Programming is a demanding task that requires extensive knowledge, experience and not a frivolous degree of creativity.

Categories Featured ExcerptFavorite

The One Probability Review That You Need

Probability and statistics are everywhere: from finance and demographic projections to casino games, these disciplines help us make sense of the world. They also underlie much of the machine learning apparatus that is the rage nowadays. What resources should we turn to, if we were to dust off our knowledge of them? (Disclaimer: I received … Read more

Differentiable Rendering

Sounds cool, but … what is it? As I’ve started to pay more attention to machine learning, differentiable rendering is one topic that caught my attention and has been popping up with some frequency. My first thought was, “cooooool is this a new system for generating pixels that somehow can leverage machine learning?” After digging … Read more

Mapping the UK’s Traffic Accident Hotspots

While looking for some interesting geographical data to work with, I came across the Road Safety Data published by the UK government. This is a very comprehensive road accident data set that includes the incident’s geographical coordinates, as well as other related data such as the local weather conditions, visibility, police attendance and more. There … Read more

Data Science Austria

The last few months I set out to build up to build a news and event aggregator. You can see the work in progress here: data-science-austria.at WordPress Plugins Here is a list of plugins that I use for the site grouped by the general overall purpose. The first one is a collection that I would … Read more

What Does It Really Mean to Operationalize a Predictive Model?

It is not enough to just stand up a web service that can make predictions. Aug 13, 2018 Original Image Source — Meme overlay by Imgflip In a 2017 SAS survey, 83% of organizations have made moderate-to- significant investments in big data, but only 33% say they have derived value from their investments. Other more recent surveys have … Read more

Practical tips for class imbalance in binary classification

4. Class weighted / cost sensitive learning Without resampling the data, one can also make the classifier aware of the imbalanced data by incorporating the weights of the classes into the cost function (aka objective function). Intuitively, we want to give higher weight to minority class and lower weight to majority class. scikit-learn has a … Read more

Feature Engineering for Healthcare Fraud Detection

The nature of the problem: medical fraud and abuse The U.S. department of health and human services in a pamphlet Avoiding Medicare Fraud and Abuse: A Roadmap for Physicians states “most physicians strive to work ethically, render high-quality medical care to their patients, and submit proper claims for payment,” yet “the presence of some dishonest … Read more

Azure SQL DWH – Overview

There are a multitude of options when it comes to storing and processing data. In this post I want to give you a brief overview of Azure SQL datawarehouse, Microsoft’s datawareshouse solution for the Azure cloud and its answer to Amazon Redshift on AWS. I will start of by talking briefly about its technical architecture … Read more

Math Behind Reinforcement Learning, the Easy Way

Aug 2, 2018 Photo by JESHOOTS.COM on Unsplash Look at this equation: Value function of Reinforcement Learning If it does not intimidate you, then you are a mathematical savvy and there is no point in reading this article 🙂 This article is not about teaching Reinforcement Learning (RL) but about explaining the math behind it. So it … Read more

Hypothesis Analysis Explained

Jul 19, 2018 Hypothesis analysis is a widely known concept and is used extensively by researchers, statisticians and quantitative analysts. It allows them to follow a set of formal steps to perform calculated analysis on their data. It is also widely used in machine learning and artificial intelligence. In this article, I will be explaining core concepts of … Read more

Docker Basics

Docker is a tool which helps developers build and ship high quality applications, faster, anywhere. Source Why Docker With Docker, developers can build any app in any language using any toolchain. Dockerized apps are completely portable and can run anywhere. Developers can get going by just spinning any container out of list on Docker Hub. … Read more

PySpark ML and XGBoost full integration tested on the Kaggle Titanic dataset

Jul 8, 2018 In this tutorial we will discuss about integrating PySpark and XGBoost using a standard machine learing pipeline. We will use data from the Titanic: Machine learning from disaster one of the many Kaggle competitions. Before getting started please know that you should be familiar with Apache Spark and Xgboost and Python. The … Read more

More advanced SQL Server for Data Scientists

In the previous post I covered the basics you need to know to work with SQL Server. In this post, I want to show you some more advanced techniques that I found pretty helpful. The topics I will cover include: How to speed up your queries with indices and using columnstore Using Views and Table … Read more

Acoustic Noise Cancellation by Machine Learning

DIY Noise-Cancellation System prototype made with TensorFlow. Jun 25, 2018 Image by TheDigitalArtist on Pixabay In this post I describe how I built an active noise cancellation system by means of neural networks on my own. I’ve just got my first results which I am sharing, but the system looks like a ravel of scripts, binaries, … Read more

Scrum PSM I

After getting scrum.org the PSM I I wanted to capture the relevant content. The complete guido can be downloaded here: scrumguides.org 1. What is Scrum? Scrum is a framework for developing and sustaining complex products. A framework in which complex adaptive problems can be addressed. It is lightweight, simple to understand and yet difficult to … Read more

Finding Good Learning Rate and The One Cycle Policy.

Introduction Learning rate might be the most important hyper parameter in deep learning, as learning rate decides how much gradient to be back propagated. This in turn decides by how much we move towards minima. The small learning rate makes model converge slowly, while the large learning rate makes model diverge. So, the learning rate … Read more

Recommendation Systems — Models and Evaluation

I’ve been involved in building several different types of recommendation systems, and one thing I’ve noticed is that each use case is different from the next, as each aims to solve a different business problem. Let’s consider a few examples: Movie/Book/News Recommendations — Suggest new content that increases user engagement. The aim is to introduce users to … Read more

Linear Regression Model

Jun 14, 2018 This post is about implementing simple linear regression model for ML beginners in step by step way with detailed explanation. If you are new to machine learning, check this post for getting a clear idea about Machine Learning and it’s basics. What is the logic behind simple linear regression model? As the … Read more

IoT Made Easy: ESP-MicroPython-MQTT-ThingSpeak

Using MQTT protocol, we will get captured data from sensors, logging them to an IoT service, ThingSpeak.com and to a mobile App, Thingsview. 1. Introduction In my previous article, MicroPython on ESP using Jupyter, we learned how to install and run MicroPython on an ESP device. Using Jupyter Notebook as our development environment, we also … Read more

From Git to Colab, via SSH

When you are using Google’s Colaboratory (Colab) for running your Deep Learning models the most obvious way to access the large datasets is by storing them on Google Drive and then mounting Drive onto the Colab environment. But a lot of open sourced large datasets that are available for research purposes, are hosted on Github/Gitlab. … Read more

Object Oriented Programming in Data Science with R

Since R is mostly a functional language and data science work lends itself to be expressed in a functional form you can come by just fine without learning about object-oriented programming. Personally, I mostly follow a functional programming style (although often not a pure one, i.e. w/o side-effects, because of limited RAM). Expressing mathematical concepts in … Read more

DevOps: To do or not to do?

Over the past few decades, four key change initiatives have been taking place in the organizations: strategic planning, re-engineering, total quality management and downsizing. The aim of these initiatives was to achieve economic effectiveness, but around 75% of them failed or created problems that were serious enough to threaten organization’s survival (1). It has been … Read more

Estimating Intervention Effects using Baysian Models in R

Measuring the effect of an intervention on some metric is an important problem in many areas of business and academia. Imagine, you want to know the effect of a recently launched advertising campaign on product sales. In an ideal setting, you would have a treatment and a control group so that you can measure the … Read more

Using data.table deep copy

data.table is an awesome R package, but there are a few things you need to watch out for when using it. R usually does not modify objects in place (e.g. by reference), but makes a copy when you change a value and saves this copy. This can be a problem if you work with large datasets … Read more

The Mathematics of Decision Trees, Random Forest and Feature Importance in Scikit-learn and Spark

This post attempts to consolidate information on tree algorithms and their implementations in Scikit-learn and Spark. In particular, it was written to provide clarification on how feature importance is calculated. There are many great resources online discussing how decision trees and random forests are created and this post is not intended to be that. Although … Read more

Blogging with hugo & netlify

Here is a great tutorial on how to host hugo on netlify Other examples using the exact same theme: Creating the hugo site In order to create a new hugo site simply go: hugo new site [path] [flags] Create a new repository via git init the git repo and push it to the guthub repo: … Read more