Towards Data Science

Pandas qcut() To understand how qcut() works, let’s start with histograms: sns.histplot(planets[‘mass’]) Histograms automatically divide an array or list of numbers into several bins, each containing different number of observations. In seaborn, it is possible to control the number of bins: sns.histplot(planets[‘mass’], bins=10) Histograms are the first examples of binning data you might have seen. … Read more How I customarily bin data with Pandas

MLflow 101 Getting your parameters, metrics, artifacts, and more logged to an MLflow tracking server Hey there, friends, and welcome back to another post in our series on MLflow. If this is the first post you’ve seen and would like to catch up, be sure to check out the previous posts here: As always, if … Read more MLflow Part 3: Logging Models to a Tracking Server!

Identify and remove outliers in each cluster from K-Means clustering Image by Gerd Altmann from Pixabay The real-world data often has a lot of outlier values. The cause of outliers can be data corruption or failure to record data. The handling of outliers is very important during the data preprocessing pipeline as the presence of … Read more Handling Outliers in Clusters using Silhouette Analysis

Part 3: Vectorization and embeddings Text vectorization is converting text into vectors of values to represent their meanings. Earlier days, we have one hot encoding method with a vector with a size of our vocabulary, and value 1 wherever the text appear and 0s elsewhere. Nowadays, we have more advanced methods like spacy, GloVe, or … Read more NLP Text Preprocessing: Steps, tools, and examples

Last month I participated in my first Hackathon because I received a random ad E-Mail from my University which promoted a very cool sounding one. I clicked it and saw that they were searching for teams with different skills, also including Data Scientists. Awesome I thought, I’m in! I signed up, applied and got accepted. … Read more Succeed as a Data Scientist in a Hackathon Without Data Being Provided?

Thanks to the internet, now the world knew about the Presidential Debate 2020 that went out of control. All of the major news stations were reporting about how the participants were interrupting and sniping at one another. I decided to put together an article that focuses on analyzing the words used in the event and … Read more Analyzing the chaotic Presidential Debate 2020 with text mining techniques

The event illustrates how TikTok’s algorithms can make mass political communication more accessible, but it is still no democratic utopia. Over the summer, I crunched the numbers on about 80,000 TikTok videos pertaining to the prank on Trump’s re-election rally in Tulsa. My main interest was understanding how TikTok’s algorithms may have played a role … Read more New Study on TikTok’s Algorithms and Trump’s Tulsa Rally

Follow this step by step guide to build an encoder decoder model and create your own translation model Photo by Michael Dziedzic on Unsplash Prerequisites: to understand this article previous knowledge about recurrent neural network (RNN) and encoder decoder is valuable. This article is a practical guide on how to develop an encoder decoder model, … Read more How to build an encoder decoder translation model using LSTM with Python and Keras.

When I ask people who lead data science teams about their data science process, many will describe a data science life cycle (i.e., their data science process workflow — such as first obtaining data, then cleaning the data, and then creating a machine learning model). Others give a vague answer about “working as a team … Read more 3 Steps to Define an Effective Data Science Process

What is the standard error? According to Wikipedia, the standard error of a statistic is the standard deviation of its sampling distribution or an estimate of that standard deviation. There are several concepts in this sentence that needs to be clarified before move forward: First, a statistic is a sampling estimation of a parameter. For … Read more How to estimate the standard error of the median: The Bootstrap Strategy

“Beauty gets the attention but personality gets the heart”. These lines portray the importance of things which lies beyond our vision. What about a Machine Learning algorithm that finds information about the inner beauty like my heart which finds the creamy layer of the Oreo despite the unappetizing outer crunchy biscuits. Factor analysis is one … Read more FACTOR ANALYSIS-MY ML OREO DETECTOR

Run your Jupyter Notebook as a stand alone web app Photo taken by Justin Jairam from @jusspreme (with permission) Jupyter Notebook provides a great platform to produce human-readable documents containing code, equations, analysis, and their descriptions. Some even consider it a powerful development when combining it with NBDev. For such an integral tool, the out … Read more Tutorial: Stop Running Jupyter Notebooks from your Command Line!

Opinion Everything will be big data, so it won’t need a special name Dead Hadoop (image by author) Big data is a great marketing term but in reality, that’s all that it is. It’s a term used to excite business executives to make them feel like Google or Amazon. The reality is that big data … Read more Big Data Will Be Dead in 5 Years

And, why you should too. Photo by Luke Peters on Unsplash If there’s one thing that frustrates me about the data science process, it would probably be the fact that I could spend hours (and maybe days) building and refining a model only to realize putting it in production is another ton worth of work … Read more 5 Reasons why I’m learning Web Development, as a Data Scientist

In this section, we’ll take a deeper dive into the guiding questions presented earlier in this article. A better understanding of the applicability and importance of the guiding questions is realised when exploration of scenarios reveal considerations to take when tackling a problem that requires a deep learning-based solution. Photo by Markus Winkler on Unsplash … Read more How To Approach Problem Definition In Your Next Deep Learning Project

Before we get into the tests themselves, let’s understand what “accuracy” means for detecting an infection. There are 2 parts to a test’s accuracy: Sensitivity and Specificity. Sensitivity measures how often tests report “positive” for people who actually have the disease. Specificity measures how often tests report “negative” for people who don’t have the disease. … Read more Interactive: Visualizing Covid-19 Test Accuracy

Successful data initiatives use a simple architecture that will scale. Most big data projects fail. Why do data initiatives fail? Photo by Serghei Trofimov on Unsplash The team starts building without clear business goals The team tries to solve problems it doesn’t understand fully using applications it understands even less The team chases Big Data … Read more Building a Succesful Data Initiative

Photo by Olenka Sergienko from Pexels 9 core excel functionalities translated into SQL As the world becomes increasingly data-driven, a growing number of professionals are working more closely with data. In many cases, the first introduction to this domain comes in the timeless form of Microsoft Excel. Vlookups, pivot tables, sumifs; maybe a little VBA … Read more You know Excel. Time to learn SQL.

What is logistic regression? Logistic regression is just adapting linear regression to a special case where you can have only 2 outputs: 0 or 1. And this thing is most commonly applied to classification problems where 0 and 1 represent two different classes and we want to distinguish between them. Linear regression outputs a real … Read more Understanding Logistic Regression

Landing a job as a data scientist, machine learning engineer, or really any kind of role writing software takes more than just math and programming knowledge. In reality, these roles require you to make hundreds of decisions every day. These might be big decisions, like: How and where should I store my data? Which algorithm(s) … Read more Want to break into data science? Start building

The past few months have been a turbulent time for most of us, in one form or another, but has our sleep been changed for better or worse? Photo by Matthew Henry on Unsplash The past few months have been a turbulent time for most of us, in one form or another. Whether that turbulence … Read more Analysing My Lockdown Sleep Data | Data Science | Medium

A Further Dig into Business Intelligence for Customer Marketing with Improved models Img from pixabay via link In a previous article, we created a stacked auto-encoder model for movie rating prediction. But as we know, with its encoder part, an auto-encoder model can also help with feature extraction. So, in this article, we will continue … Read more Credit Card Customer Clustering with Autoencoder and K-means

Photo by Luke Chesser on Unsplash Data Analysis can be separated and organized into 6 types, arranged with an increasing order of difficulty. Descriptive Analysis Exploratory Analysis Inferential Analysis Predictive Analysis Causal Analysis Mechanistic Analysis Goal — Describe or Summarize a set of Data Description: The very first analysis performed Generates simple summaries about samples … Read more The Six Types of Data Analysis

A gentle introduction to modern movie recommenders Traditionally, recommender systems are based on methods such as clustering, nearest neighbor and matrix factorization. However, in recent years, deep learning has yielded tremendous success across multiple domains, from image recognition to natural language processing. Recommender systems have also benefited from deep learning’s success. In fact, today’s state-of-the-art … Read more Deep Learning based Recommender Systems

Getting Started with LeTeX + Docker + VSCode Remote Container Photo by Arisa Chattasa on Unsplash IntroductionSetupMethod 1: tianon/latexMethod 2: Remote-ContainersMethod 3: Creating your containerHow to switch Remote containersOpening a PDFConclusionReferences We can run a Docker application in any environment, Linux, Windows, or Mac. Docker provides a set of official base images for most used … Read more Three Ways to Create Dockernized LaTeX Environment

I used the knowledge and wisdom I gained from my work in Artificial Intelligence to understand and teach my two-year-old son more effectively and regain my sanity. Image via Unsplash and Freepik Overview The knowledge I have gained from building Artificial Intelligence (AI) tech is directly applicable to raising my toddler. Not only does it … Read more How AI Techniques Made Me a Better Parent for Our Toddler

Let’s face it; our projects are our babies. They are near and dear to us, we want to keep working on them, and we see the value in them. Each project adds to the amount we have provided, and we don’t always want to share that work. But what if someone comes up and wants … Read more Thoughts on Finding a Data Science Project

A basic tutorial about cleaning data using command-line tools: tr, grep, sort, uniq, sort, awk, sed, and csvlook Photo by JESHOOTS.COM on Unsplash Cleaning data is like cleaning the walls in your house, you clear any scribble, remove the dust, and filter out what is unnecessary that makes your walls ugly and get rid of … Read more How to Clean Text Files at the Command Line

Data Science use cases you might not have explored before… Photo by Daniel Cheung on Unsplash When I wrote the first part of the blog Top 9 Data Science Careers of the Future in July, I did have in mind other Data Science career pathways that needed to be put out. I was overwhelmed to … Read more Top Data Science Careers of the Future— Part 2

Machine Learning Tutorials — Part 1 | Part 2 → soon Complete Your First Kaggle Competition in Less Than 20 Lines of Code with Decision Tree Classifier | Machine Learning Tutorials Since you are reading this article, I am sure that we share similar interests and are/will be in similar industries. So let’s connect via … Read more Kaggle’s Titanic Competition in 10 Minutes | Part-I