Featured

Photo by Roberta Sorge on Unsplash If you are a Scikit-Learn fan, Christmas came a few days early in 2020 with the release of version 0.24.0. Among the new features are 2 experimental classes in the model_selection module that support faster hyperparameter optimization: HalvingGridSearchCV and HalvingRandomSearchCV. Like their close cousins GridSearchCV and RandomizedSearchCV, they use … Read more faster-hyperparameter-tuning-with-scikit-learn

Finding the optimal input parameters when the response value is unknown In this post, I will describe my experience working on finding the maximum value in a multivariate optimization problem where the response value was unknown. I worked on this problem in the final project of a graduate course in Experimental Design that I took … Read more An Experimental Design Approach to Solve a Multivariate Optimization Problem

Let’s begin … Reading the dataset >> Titanic_table = readtable(‘train.csv’);>> Titanic_data = (table2cell(Titanic_table)); The train set has 891 passenger entries and 12 columns. Now, lets take a look into our data. The head function displays the top rows of a table, similar to that in Pandas. >> head(Titanic_table) The ‘Survived’ column is our binary target … Read more Kaggle — Predict survival on the Titanic challenge in MATLAB

Don’t have enough data? Don’t worry! Synthetic data is here! Data shortage has become an important problem to be addressed in industries such as health, sports, manufacturing, and law due to the lack of data, privacy, and confidentiality. Photo by Luke Chesser on Unsplash Today, great advancements are being made in multiple sectors of society, … Read more Synthetic Data to the rescue!

After writing this data story, I realized that it may be helpful to write a piece outlining the power of creating web visualizations for audiences. As a data science graduate student (and data engineer by profession), I’m often disappointed by the lack of follow-through (me included) on how we communicate conclusions of our analyses and … Read more Using D3.js to create dynamic maps and visuals that show competing climate change scenarios for…

Photo by sergio souza on Unsplash It was in August of the fateful year 2020 that I wrote a text right here in the Medium showing the findings of a model of clusters of municipalities from variables related to GDP. The time has come to visit that model again and make new experiments. Here the … Read more Back to the municipality clusters: celestial objects and maps

Let’s get started and write a basic U-Net in Pytorch based on the diagram in Figure 1. To make the code simpler and clean we can define several modules, starting with the ConvLayer that is no more than a 2d convolutional layer followed by a 2d batch normalization and the ReLU activation. Notice that the … Read more Learn AI Today 05: Image segmentation with U-Net models

Setup If you are coming from the previous tutorial, you will need to make some small changes to app.py, FinBertQARanker/__init__.py , and FinBertQARanker/tests/test_finbertqaranker.py. Joan Fontanals Martinez and I have added some helper functions and batching in the Ranker to help speed up the process. Instead of pointing out the changes, I have made a new … Read more Search Engine Evaluation in Jina

Implementing a visual datetime filter for timeseries data in Python Visual by author. Introduction Perhaps the most proliferated type of data that we grapple with on a daily basis is timeseries data. Basically, anything that is indexed using date, time or both can be considered as a timeseries dataset. And more often than not, you … Read more Creating an interactive datetime filter with Pandas and Streamlit

Let’s say we have a list: num_list = [1,2,3,4,5] And we define a function that takes in 5 arguments and returns their sum: def num_sum(num1,num2,num3,num4,num5):return num1 + num2 + num3 + num4 + num5 And we want to find the sum of all the elements in num_list. Well, we can accomplish this by passing in … Read more Three Concepts to Become a Better Python Programmer

Working Smart It takes a sizeable amount of time to prepare for a coding interview. There are so many different topics, data structures, and algorithms to go over. Recursion is one of the most important algorithm types. Because it is the basis for so many important algorithms like divide and conquers, graph algorithms, dynamic programming, … Read more 10 Popular Coding Interview Questions on Recursion

Using LightGMB to predict my song prediction habits based solely on audio features Image by Omid Armin https://unsplash.com/@omidarmin In early 2019, Spotify shared interesting statistics about their platform. Out of 35+ million songs on the service, Spotify users created over 2+ billion playlists (Oskar Stål, 2019). I thought of the analogy that our music taste … Read more Predicting Song Skipping on Spotify

Class. If we have trained our model with images of cats and dogs, then by requesting for a “cat”, we expect the generator to output an image of a cat that looks great. If our model is only trained for one class, then this input feature is not required. Noise. After training a generative model … Read more Intuition Behind GAN’s Generator

Wave motion illustration http://animatedphysics.com/insights/modelling-photon-phase/ GANs and other generative machine learning algorithms are still hyped and work fantastically with images, texts, and sounds. They’re able not only to generate data for fun but solve important theoretical issues and boost production ML pipelines. Unfortunately, typical today’s practical use case is limited to “fine-tune pre-trained StyleGAN2 for zombies … Read more From high-school physics to GANs: essentials for mastering generative machine learning [1/2]

DATA SCIENCE Data Champions and Data Driven Culture Happy New Year 2021 dear readers. Another year ahead of us and as life returns slowly to normal after a pandemic filled year in 2020, it is time to think about the data future of organizations, why this matters in this age of digital transformation and approaches … Read more Who is A Data Champion? Here is Why Your Organization Needs One

A PyTorch and Hugging Face implementation of fine-tuning BERT on the MultiNLI dataset Image from PNGWING. In this article, I will be describing the process of fine-tuning pre-trained models such as BERT and ALBERT on the task of sentence entailment using the MultiNLI dataset (Bowman et al. A Broad-Coverage Challenge Corpus for Sentence Understanding through … Read more Fine-tuning pre-trained transformer models for sentence entailment

HDBSCAN is a clustering algorithm that identifies islands of closely related elements in a sea of noisy outliers. I recently came across the article “How HDBSCAN works” by Leland McInnes, and I was struck by the informative, accessible way he explained a complex machine learning algorithm. Clusters identified with HDBSCAN Unlike clustering algorithms like k-means, … Read more HDBSCAN Clustering with Neo4j

Algorithms are set of instructions designed to modify an input to create the desired output. The Sorting problem is a typical programming task that Data Scientists and other programming disciplines come across. The sorting problem’s main task is to order a sequence of elements in either ascending or descending order. Photo by Andrik Langfield on … Read more Algorithms For Data Scientists — Insertion Sort

To prepare the dataset for training and evaluation, I use the following code to load the SemEval-2014 dataset from my GitHub repository and prepare it for PyTorch: Train Now that we’ve narrowed the scope of models to sentence-pair classification using transformers in conjunction with the preprocessing method defined above, we still have several options for … Read more Where should I eat after the pandemic? (Part 1/2)

Introduction Data Scientist Computer Scientist Similarities and Differences Summary References Data Science and Computer Science often go hand-in-hand, but what really makes them different? What do they have in common? After experiencing several different roles in Data Science at various companies, I have realized general themes of the Data Science process, along with how Computer … Read more Data Science vs Computer Science. Here’s the Difference.

Understanding the components of a time series Source: Photo by geralt from Pixabay For someone who originally comes from an economics background, it might seem quite strange that I would spend some time building models that can predict weather patterns. I often questioned it myself — but there is a reason for it. Temperature patterns … Read more Want To Get Good At Time Series Forecasting? Predict The Weather.

1 Completeness and Correctness — 3.1 Completeness — 3.2 Correctness & Tobler’s law2 Building footprints classification pipeline — 2.1 Generate building features — 2.2 Building blocks segmentation with HDBSCAN — 2.3 Building types classification — XGBoost3 Wrapup Since OSM (OpenStreetMap) is a VGI platform (volunteer geographic information), buildings are traced and tagged voluntarily by the … Read more Kedro hands-on: Build your own demographics atlas. Pt. 2: building footprints classification

Introduction to 3D Data It’s a consensus that synthesizing 3D data from a single perspective is a fundamental human vision functionality which is extremely challenging for computer vision algorithms. But recent advancements in 3D acquisition technology have taken a great leap after the increased availability and affordability of 3D sensors like LiDARs, RGB-D cameras (RealSense, … Read more Deep Learning for 3D Synthesis

This article takes a holistic approach to a machine learning using elementary Systems Engineering principles. Enabling you to understand and manage the fundamental building block in most machine learning systems. You will learn how to get form data to predictions. Figure 1: Machine Learning System Design Systems Engineering principles Machine Learning Exploratory Data Analysis Training … Read more Machine Learning a Systems Engineering Perspective

A comprehensive practical guide Photo by Mineragua Sparkling Water on Unsplash SQL is a programming language used by most relational database management systems (RDBMS) to manage data stored in tabular form (i.e. tables). A relational database consists of multiple tables that relate to each other. The relation between tables is formed with shared columns. When … Read more 7 Examples to Master SQL Joins

The differences and similarities between values, distinct, and all Photo by Noah Näf on Unsplash The functional language DAX used in Power Pivot, SQL Server Analysis Services, and Power BI is powerful. Like all powerful languages, understanding the nuanced differences in their usage is important. On the surface, the two functions VALUES and DISTINCT appear … Read more Distinct Values in DAX

How play success can help you make sense of plain box scores Photo by Tim Mielke on Unsplash In their weekend matchup against the Colts, the Bills QB Josh Allen threw 26 passes off 35 attempts for 324 yards and 2 touchdowns. The Bills run game also had 21 rushes for 96 yards. In response, … Read more Analysing the Flow of NFL Games with One Simple Statistic

This is a well-tested and often touted method of habit-forming. The Habit Loop was developed by journalist and productivity author Charles Duhigg who worked in tandem with neurologists, psychologists, and researchers to create a method for developing habits that people will maintain. He describes the common phenomenon of people entering the new year all gung-ho … Read more 7 Proven Ways to Develop a Coding Habit

Applications of NLP A hybrid approach to an unsupervised classification task Photo by Markus Spiske on Unsplash Text classification is a common task in Natural Language Processing. The main approach tends toward representing the text in a meaningful way — whether through TF-IDF, Word2Vec, or more advanced models like BERT — and training models on … Read more Using keyword extraction for unsupervised text classification in NLP

Parallel inference on millions of strings using transformers with minimal infrastructure setup Photo by Guillaume Jaillet on Unsplash Transformers are taking the NLP word by storm. Every text-related task has benefitted from transformers. The best library to work with transformers is the HuggingFace one, and the easiest way to get started with HuggingFace is through … Read more Rapid transformer inference at scale in the Google Cloud Platform

Overview of the applications of AI in mitigating the dangers due to fire Being a student secretary of the Fire and Security Association of India at my college, I have conducted various surveys and workshops on methodologies to avert the dangers caused due to fire. As an AI enthusiast, I have always aspired to come … Read more Applications of Artificial Intelligence in Fire & Safety

What are the types of Missing Data with Examples? Image by Gino Crescoli from Pixabay The real-world dataset often has a lot of missing values. The cause of the presence of missing values in the dataset can be loss of information, disagreement in uploading the data, and many more. Missing values need to be imputed … Read more Deep Dive Analysis of Missing Values in Dataset

The vertical axis of the MAWD Chart is the continuum of inventory costs. The horizontal axis is the continuum of failure costs. Let’s start in the southwest quadrant, which I labeled “Zone 1”. In this quadrant there are low failure costs and low inventory costs. This is your ”Run to failure” zone, given that storing … Read more The Economics of Equipment Failure

Introduction Business Problem Statement Isolating Business Metrics Algorithm Comparison Code Review MLOps and DevOps Summary References Working professionally in Data Science for a few years now, I have discovered some best practices from a variety of different experiences. I will be highlighting my top five Data Science practices in hopes of helping you in your … Read more The Top 5 Data Science Practices

So you finished your project, you got some good data, you processed it, cleaned it, trained your model, applied it to your data, and got amazing results. That’s it. Not really… Often, a software is developed for others to use, so once the programmer or data scientist finishes building the project, they will need to … Read more Documentation 101: The 6 Pillars of a Good Documentation

With HyperLogLog algorithm, we trade off some accuracy to get the job done faster. But how much faster and at the expense of what level of accuracy? Photo by Crissy Jarvis on Unsplash HyperLogLog (HLL) is an algorithm that estimates how many unique elements the dataset contains. Google BigQuery has leveraged this algorithm to approximately … Read more Counting unique elements faster in Google BigQuery with HyperLogLog

To understand these Flows, let’s start with the Index Flow and look into the individual components one by one. Index Flow The main idea behind the Index Flow is to use a pre-trained BERT model to encode all of our answer passages into embeddings then indexing these embeddings so that they can be searched in … Read more NLP in Finance: Question Answering

A practical guide for ggplot2 data visualization library Photo by Joshua Earle on Unsplash Data visualization is a powerful tool for exploratory data analysis. We can use it to reveal the underlying structure within data or the relationships among variables. An overview of basic descriptive statistics can also be obtained from a data visualization. Data … Read more Exploring a Dataset Visually with ggplot2

About the ups and downs of creating visual generative art Image by author Visual generative art is an intriguing niche of the visual arts that makes use of an autonomous system, typically defined through code, to generate pieces of art. Although it has its roots at least in the 1950s, generative art gained traction with … Read more Is Generative Art for You?