Month: August 2020

Photo by Antoine Dautry on Unsplash With the hype surrounding machine learning these days, it’s quite easy to into the trap of overcomplicating a problem that could be solved with a simple linear or logistic regression. In some cases, the required infrastructure for a complex machine learning pipeline might not even be available. Most data … Read more

Making logistic regression less linear Logistic regression is one of the most used machine learning techniques. Its main advantages are clarity of results and its ability to explain the relationship between dependent and independent features in a simple manner. It requires comparably less processing power, and is, in general, faster than Random Forest or Gradient … Read more

When finding that one trustworthy politician in your dataset is like looking for a needle in the haystack, here is what you can do. Illustration of standard techniques, Image by Author When we speak of imbalanced data, what we mean is that at least one class is underrepresented. For example, when considering the problem of … Read more

Introducing #66DaysOfData I’m feeling stuck. In my current work and in the content I create (videos and blog posts), I feel like I’ve begun to stall out. Most of the consumers of my content are at the start of their data science journey. The longer I’m in the field, the less I feel I can … Read more

A real-world client-facing project with real sensor data Photo by Marcin Jozwiak on Unsplash Note: The detailed project report and the datasets used in this post can be found in my GitHub Page. This project was assigned to me by a client. There is no non-disclosure agreement required and the project does not contain any … Read more

Understanding and utilizing imbalanced data. Classification is a type of supervised learning in Machine Learning that deals with categorizing data into classes. Supervised learning implies models that take input with their matched output to train a model which can later make useful predictions on a new set of data with no output. Some examples of … Read more

Source: Unsplash Coming out of college with a background in mathematics, I fell upward into the rapidly growing field of data analytics. It wasn’t until years later that I realized the incredible power that comes with the position. As Uncle Ben told Peter Parker (aka Spiderman), “With great power, comes great responsibility”. The proverb echoed … Read more

Now looking at this we can see that actually speaking, comedy is the most common genre unlike hentai which was supposedly the most common before we did the preprocessing and plotting. When we have a categorical variable and a numerical variable, we can resort to using barcharts. Depending on whether the categorical column is on … Read more

4.2 Model Architecture Overview The fundamental TCN model architecture mentioned here is derived from Section 3 above —a generic TCN architecture consisting of causal convolutions, residual connections and dilated convolutions. The overview of KDTCN architecture is shown below: Illustration of the KDTCN framework Original model inputs are price values X, news corpus N , and … Read more

Description: In Harry Potter, we obtain four different communities. One involves Harry’s family, including his true parents and the Dudleys, as well as their entourage. A second one is formed by Dumbledore’s friends, which are a bit outside of the scope of the book’s main plot and are mentioned during one chapter. The members of … Read more

Photo by Morning Brew on Unsplash This article is an attempt to establish the intuition behind Deep Q Networks. Before we get to that, let’s spend some good amount of time on understanding what is Q Learning. Too long, do read! In simple terms, Q Learning is a model-free Reinforcement Learning approach to enable the … Read more

Quantum Classification Using a Real-World Dataset Dr. Vishal Sharma, a Postdoctoral Research Fellow, recently asked me via LinkedIn messaging about quantum classification. I won’t reveal anything about his 5-column, 26,156-row dataset other than its size — 130,780 total data points — but I will reveal the modified algorithm that I shared with him. Introduction Quantum … Read more

The reason behind Moravec’s Paradox. Photo by Stephen Andrews on Unsplash At the most basic level, the reason for Moravec’s Paradox is simple: We don’t know how to program general intelligence (yet). We’re already good at getting AI to do specific things, but most toddler level skills require learning new things and transferring them into … Read more

Great movies, speeches, and business presentations all share one thing: an emotional arc. How can it spice up your story? You’re launching a digital transformation initiative in the middle of the ongoing pandemic. You are pretty excited about this big-ticket investment, which has the potential to solve remote-work challenges that your organization faces. However, the … Read more

Before I explore the above regression techniques further. Let’s gain an understanding of the assumptions associated with regression techniques. Regression Assumptions Residuals of the regression line are normally distributed — To ensure that the result from the regression model are valid, the residuals (difference between observed and predicted values) should follow a normal distribution (with … Read more

Now that we finished examining and analyzing our data, we can go ahead and build some machine learning models. We will begin simple, using the K-nearest neighbours, and logistic regression classifiers. K-Nearest Neighbours The key concept of K-nearest neighbours is that, when we see an unknown example, we will look at what the unknown is … Read more

When python plays with the internet files. A data analytics cycle starts with gathering and extraction. I hope my previous blog gave an idea about how data from common file formats are gathered using python. In this blog, I’ll focus on extracting the data from files that are not so common but has the most … Read more

Introduction Discussing the same past project Not asking enough questions Assuming interviewers know my past experiences Not considering the business impact Not overviewing the whole Data Science process Summary References I have interviewed with several companies, having some repeated and key mistakes along the way. As a result, I have learned from them and have … Read more

What you need to know Corporations have defined project processes. Stage-gate or steering committee meetings are part of that where outcomes must be presented. Presentations have to be submitted a few days in advance and must contain certain expected information. Also, corporates are always under pressure to deliver financial results. That leads to consistently tight … Read more

What you should consider before migrating to the cloud to make your data warehouse and data lake future-proof & how the separation of storage and compute was approached by Snowflake, Amazon, Google, SAP, and IBM Photo by John Schnobrich on Unsplash Not so long ago, establishing an enterprise data warehouse involved a project that would … Read more

In Amazonia, fire is associated with several land-practices. Slash-and-Burn is one of the most used practices in Brazilian agriculture (as part of a seasonal cycle called “queimada”). Whether for opening and cleaning agricultural areas or renewing pastures, its importance in the agricultural chain is undeniable. Unfortunately, this is often the cause of wildfires in forests. … Read more

Another approach to identify multicollinearity is via the Variance Inflation Factor. VIF indicates the percentage of the variance inflated for each variable’s coefficient. Beginning at a value of 1 (no collinearity), a VIF between 1–5 indicates moderate collinearity while values above 5 indicate high collinearity. Some cases where high VIF would be acceptable include the … Read more