Month: August 2020

Photo by Antoine Dautry on Unsplash With the hype surrounding machine learning these days, it’s quite easy to into the trap of overcomplicating a problem that could be solved with a simple linear or logistic regression. In some cases, the required infrastructure for a complex machine learning pipeline might not even be available. Most data … Read more My First Month as a Data Scientist

Making logistic regression less linear Logistic regression is one of the most used machine learning techniques. Its main advantages are clarity of results and its ability to explain the relationship between dependent and independent features in a simple manner. It requires comparably less processing power, and is, in general, faster than Random Forest or Gradient … Read more Combining logistic regression and decision tree

When finding that one trustworthy politician in your dataset is like looking for a needle in the haystack, here is what you can do. Illustration of standard techniques, Image by Author When we speak of imbalanced data, what we mean is that at least one class is underrepresented. For example, when considering the problem of … Read more Imbalanced Data, What Can You Do?

Introducing #66DaysOfData I’m feeling stuck. In my current work and in the content I create (videos and blog posts), I feel like I’ve begun to stall out. Most of the consumers of my content are at the start of their data science journey. The longer I’m in the field, the less I feel I can … Read more Why I’m Starting Data Science Over.

A real-world client-facing project with real sensor data Photo by Marcin Jozwiak on Unsplash Note: The detailed project report and the datasets used in this post can be found in my GitHub Page. This project was assigned to me by a client. There is no non-disclosure agreement required and the project does not contain any … Read more Time Series Pattern Recognition with Air Quality Sensor Data

Understanding and utilizing imbalanced data. Classification is a type of supervised learning in Machine Learning that deals with categorizing data into classes. Supervised learning implies models that take input with their matched output to train a model which can later make useful predictions on a new set of data with no output. Some examples of … Read more Classification Framework for Imbalanced Data.

Source: Unsplash Coming out of college with a background in mathematics, I fell upward into the rapidly growing field of data analytics. It wasn’t until years later that I realized the incredible power that comes with the position. As Uncle Ben told Peter Parker (aka Spiderman), “With great power, comes great responsibility”. The proverb echoed … Read more Data Professionals: The Arbiters of Truth

Now looking at this we can see that actually speaking, comedy is the most common genre unlike hentai which was supposedly the most common before we did the preprocessing and plotting. When we have a categorical variable and a numerical variable, we can resort to using barcharts. Depending on whether the categorical column is on … Read more Exploratory Data Analysis on Anime Data

4.2 Model Architecture Overview The fundamental TCN model architecture mentioned here is derived from Section 3 above —a generic TCN architecture consisting of causal convolutions, residual connections and dilated convolutions. The overview of KDTCN architecture is shown below: Illustration of the KDTCN framework Original model inputs are price values X, news corpus N , and … Read more Farewell RNNs, Welcome TCNs

Description: In Harry Potter, we obtain four different communities. One involves Harry’s family, including his true parents and the Dudleys, as well as their entourage. A second one is formed by Dumbledore’s friends, which are a bit outside of the scope of the book’s main plot and are mentioned during one chapter. The members of … Read more Explore Harry Potter via a dynamic social network of characters

Photo by Morning Brew on Unsplash This article is an attempt to establish the intuition behind Deep Q Networks. Before we get to that, let’s spend some good amount of time on understanding what is Q Learning. Too long, do read! In simple terms, Q Learning is a model-free Reinforcement Learning approach to enable the … Read more A layman’s derivative on Deep Q Learning | Reinforcement Learning

Quantum Classification Using a Real-World Dataset Dr. Vishal Sharma, a Postdoctoral Research Fellow, recently asked me via LinkedIn messaging about quantum classification. I won’t reveal anything about his 5-column, 26,156-row dataset other than its size — 130,780 total data points — but I will reveal the modified algorithm that I shared with him. Introduction Quantum … Read more 130,780-point Quantum Classification

The reason behind Moravec’s Paradox. Photo by Stephen Andrews on Unsplash At the most basic level, the reason for Moravec’s Paradox is simple: We don’t know how to program general intelligence (yet). We’re already good at getting AI to do specific things, but most toddler level skills require learning new things and transferring them into … Read more Why is AI So Smart and Yet So Dumb?

Great movies, speeches, and business presentations all share one thing: an emotional arc. How can it spice up your story? You’re launching a digital transformation initiative in the middle of the ongoing pandemic. You are pretty excited about this big-ticket investment, which has the potential to solve remote-work challenges that your organization faces. However, the … Read more How to Become a Better Storyteller: 3 Key Points

Before I explore the above regression techniques further. Let’s gain an understanding of the assumptions associated with regression techniques. Regression Assumptions Residuals of the regression line are normally distributed — To ensure that the result from the regression model are valid, the residuals (difference between observed and predicted values) should follow a normal distribution (with … Read more Supervised learning — the What, When, Why, Good and Bad (Part 1)

Now that we finished examining and analyzing our data, we can go ahead and build some machine learning models. We will begin simple, using the K-nearest neighbours, and logistic regression classifiers. K-Nearest Neighbours The key concept of K-nearest neighbours is that, when we see an unknown example, we will look at what the unknown is … Read more Detecting Pneumonia from Chest X-Rays with Deep Learning

When python plays with the internet files. A data analytics cycle starts with gathering and extraction. I hope my previous blog gave an idea about how data from common file formats are gathered using python. In this blog, I’ll focus on extracting the data from files that are not so common but has the most … Read more Gather Your Data: The “Not-So-Spooky” APIs!

Introduction Discussing the same past project Not asking enough questions Assuming interviewers know my past experiences Not considering the business impact Not overviewing the whole Data Science process Summary References I have interviewed with several companies, having some repeated and key mistakes along the way. As a result, I have learned from them and have … Read more 5 Data Science Interview Mistakes I’ve Made

What you need to know Corporations have defined project processes. Stage-gate or steering committee meetings are part of that where outcomes must be presented. Presentations have to be submitted a few days in advance and must contain certain expected information. Also, corporates are always under pressure to deliver financial results. That leads to consistently tight … Read more 8 Determining Factors for the Selection of the Model Approach

What you should consider before migrating to the cloud to make your data warehouse and data lake future-proof & how the separation of storage and compute was approached by Snowflake, Amazon, Google, SAP, and IBM Photo by John Schnobrich on Unsplash Not so long ago, establishing an enterprise data warehouse involved a project that would … Read more Why you are throwing money away if your cloud data warehouse doesn’t separate storage and compute

In Amazonia, fire is associated with several land-practices. Slash-and-Burn is one of the most used practices in Brazilian agriculture (as part of a seasonal cycle called “queimada”). Whether for opening and cleaning agricultural areas or renewing pastures, its importance in the agricultural chain is undeniable. Unfortunately, this is often the cause of wildfires in forests. … Read more Green is the new Black: Saving Amazon Rainforests using AI!

Another approach to identify multicollinearity is via the Variance Inflation Factor. VIF indicates the percentage of the variance inflated for each variable’s coefficient. Beginning at a value of 1 (no collinearity), a VIF between 1–5 indicates moderate collinearity while values above 5 indicate high collinearity. Some cases where high VIF would be acceptable include the … Read more Collinearity Measures