The parameter usecols in pandas.read_csv() is extremely useful to load only the specific columns from the csv data set. Here is the direct comparison of the time taken by read_csv() with and without usecols pandas.read_csv() usecols | Image by Author Importing .csv file to pandas DataFrame using usecols is ⚡️ 2.4X ⚡️ faster than importing … Read more
In order to add customizations such as cluster colors, bubble sizes, and hover-over tips, we need to first add three new columns to our data frame that assign these ‘customization parameters’ to each data point. The following code will add a new column called ‘color’ to the data frame. We first define a function called … Read more
Hopefully you can now see what the app looks like and basically how it is structured. So, now let’s look at how it is put together. We retrieve the data from Our World in Data’s Github repository. All OWID’s data is provided under the CC BY licence, which means it can be used pretty much … Read more
Using geographic data and geospatial images to study the impact of climate changes, natural disasters, or human activity. Geographic data includes geospatial data captured using satellite imagery and geographic postponing systems (GPS) and other geographic data generally described explicitly in terms of geographic coordinates. Geospatial analysis includes collecting, reporting, plotting, and analyzing this data using … Read more
Explore the moving average model and discover how we can use the ACF plot to identify the right MA(q) model for our time series Photo by Pawel Czerwinski on Unsplash One of the foundational models for time series forecasting is the moving average model, denoted as MA(q). This is one of the basic statistical models … Read more
Let’s start with image basics (Image by author, made with draw.io) Today, you’re going to learn some of the most important and fundamental topics in machine learning and deep learning. I guarantee that today’s content will deliver some of the foundational concepts that are key to start learning deep learning — a subset of machine … Read more
I have read many different viewpoints online about data engineering replacing data science as the hottest job of the 21st century. After working closely with both data engineering and data science teams, I have come to the conclusion that both fields are equally valuable. Companies need data engineers. They need people who are able to … Read more
The main keyboard I used is attached to a MacBook Pro. I’m typing this on a MacBook Pro. And being the tech nerd I am, when Apple released a couple of new MacBook Pro’s with upgraded hardware: M1 Pro, M1 Max chips and redesigns and all the rest, I decided, I better test them out. … Read more
Photo by Mikaela Wiedenhoff on Unsplash An assessment of influences of email behaviours on greenhouse emissions using System Dynamics Nowadays, it is quite normal to communicate via digital tools. We have social media platforms to chat with friends, videotelephony services to conduct job interviews and, of course, the good old email. One would think that … Read more
Being a low-resource language, it is not easy to acquire an annotated corpus in Latin, with a substantial size. I used one accessible dataset[4][5] on the internet, which consists of 45 Latin sentences classified into 3 sentiment classes (POSITIVE, NEGATIVE, NEUTRAL, MIXED) and have been extracted from Horace’s Odes (The creators of the dataset have … Read more
Details of Making Histograms The histogram is one of the most popular plots. It is useful to understand the overall distribution of a continuous variable. So, almost in any data analysis or exploratory data analysis, or machine learning project, you will start with some histograms. In this article, I will explain how to make histograms … Read more
Techniques to detect data drift Image by Mediamodifier from Pixabay Model Monitoring is an important component of the end-to-end data science model development pipeline. The robustness of the model not only depends upon the training of the feature engineered data but also depends on how well the model is monitored after deployment. Typically a machine … Read more
Now, we are going to generate visual explanation heat maps. To generate the heat-maps we are going to use Grad-CAM[1] algorithm. The heat-maps identify the image regions that influence the network’s decision. If we look at the heat map, we can easily understand which image pixels contribute to the network’s decision. To work with grad-cam, … Read more
Let’s use the above example to implement the Dijkstra’s and UCS algorithms using Python. The diagram is a schematic representation of the graph with vertices V={a, b, c, d, e}, E={{a, b, 6}, {a, c, 3}, {b, c, 1}, {c, b, 4}, {b, d, 2}, {c, d, 8}, {c, e, 2}, {d, e, 9}, {e, … Read more
Even if you are completely unfamiliar with technology, you likely utilize APIs on a daily basis. Whether ordering from an online store, determining your train schedule, or checking a weather app— we constantly pose requests for information, which is retrieved from a system or database we don’t necessarily know anything about. The layer between our … Read more
Step 2 — Importing Data After project creation, there will already be a default node titled Table 1. The aim now is to create a node for each of the three datasets. We first rename the default node to Drugs by clicking on the ellipsis (•••) menu beside the Table 1 name and select Edit … Read more
Leveraging zero-shot learning for text summarisation with Hugging Face’s Pipeline API Photo by David Dvořáček on Unsplash This is the second part of a tutorial on setting up a text summarisation project. For more context and an overview of this tutorial, please refer back to the introduction as well as part 1 in which we … Read more
How To Update Your Beliefs Based On Evidence Imagine you took a rapid at-home covid-19 test. If you test positive, how worried should you be? Alternatively, if you test negative, how safe should you feel? Photo by Medakit Ltd on Unsplash This article will arm you with the knowledge and the tools to correctly and … Read more
Photo of Yan Krukov from Pexels Goal: This article gives an introduction to vector norms, vector distances and their application in the field of data science Why you should learn it: Vector norms and distances are used to describe attributes of vectors and the relationship of different vectors to each other. It is widely used … Read more
You may think that Self-Training involves some magic or uses a highly complex approach. In reality, though, the idea behind Self-Training is very straightforward and can be explained by the following steps: First, we gather all labeled and unlabeled data, but we only use labeled observations to train our first supervised model. Then we use … Read more
Identifying indoor Wi-Fi users’ locations with a tolerance of uncertainty by Pymc3 Wi-Fi sensor network With the help of GPS, outdoor positioning has witnessed significant development. However, we are suffering from important inaccuracies when facing the indoor case. The existence of Wi-Fi network gives an alternative to build a localization system and significant research has … Read more
Manufacturing strikes in the United States plotted against time (Data source: R data sets) (Image by Author) A step-by-step tutorial to get up and running with the Poisson HMM A Poisson Hidden Markov Model is a mixture of two regression models: A Poisson regression model which is visible and a Markov model which is ‘hidden’. … Read more
Combining the power of two simple things to get something awesome Image by Author Sometimes when you combine two seemingly simple things, you get something awesome. In this article, you will see the power of combining a line chart and a numeric scaler. Just for the sake of vocabulary: The line chart is very useful … Read more
Data Preprocessing — Cleaning the Data Before Analysis Before we choose our FuzzyWuzzy function and start comparing strings, we want to clean the data to ensure that our results will be as accurate as possible. Cleaning the data means removing irrelevant strings, and thus improving the functions’ performance. For example, let’s assume we compare strings … Read more
Get to know FuzzyWuzzy library for string comparison — how it works and when it is useful picture by Anastasiya Romanova on unsplash About a year ago, I saw a colleague of mine working on a large data set, aiming to measure the similarity level between each pair of strings. My colleague started to develop … Read more
PYTHON Here is my take on this cool Python library and why you should give it a try Photo by KAL VISUALS from Unsplash A few weeks ago, I posted a blog about Bamboolib that became quite popular. The blog was very well received, achieving tens of thousands of views in the first week. And … Read more
The secrets of why the SwAV model architecture works so well for self-supervised pre-training Teaching a neural network to understand the world around it without human supervision has been one of the north stars of the computer vision research community for years. Recently, multiple publications have shown the potential of novel methods to make significant … Read more
The mean prediction is the average predicted bonus across all 1000 employees. If you add up all the values in the contribution matrix and add the mean prediction you will get the models actual prediction for that employee. In our case, the mean predicted bonus was $148.93. All the values in the matrix add up … Read more
Creating routes requires a paid Strava subscription, so for safety concerns I won’t share my GPX files with you. Use any GPX file from your routes or the training log. If you’re not using Strava, simply find a sample GPX file online, it should still work. Got the GPX file ready? Awesome — let’s see … Read more
I will start this post with a statement: normalization and standardization will not change the distribution of your data. In other words, if your variable is not normally distributed, it won’t be turn into one with the normalize method. normalize() or StandardScaler() from sklearn won’t change the shape of your data. Standardization Standardization can be … Read more
How to save time and nerves when coding for data analysis in VS Code using Multi-Cursor and selection features Doing multiple thing at once — Photo by Matt Bero at unsplash Working with data can be very dynamic with repeated forward and backward motions through your code to adjust and copy snippets, introducing new assumptions, … Read more
image — shutterstock Can Codex translate commands between AWS and Google Cloud? Many businesses need to deal with multiple cloud environments. AWS, Azure, and Google Cloud each have their own sets of commands to carry out cloud actions, such as setting up buckets, defining service accounts, and other administration tasks. It would be great to … Read more
An open-source library to train and use generative text models in a few lines of code Photo by saeed karimi on Unsplash If you want to skip ahead and just see how to fine-tune your own dataset you can skip the rest of the article and just check out the colab notebook! Presenting guest lectures … Read more
Tip 3: Move Up and Challenge Yourself with More Challenging Questions My own drawing I’ve created the above diagram to differentiate three zones: comfort zone, study zone, and danger zone. The comfort zone refers to the areas we are familiar with, and it takes little effort for us to pull knowledge from the comfort zone. … Read more
An easy-to-understand explanation of the research paper Photo by Scott Graham on Unsplash In this article, I am going to summarize an influential paper in the field of Natural Language Processing, Towards Controlled Generation of Text[1]. This paper was published in 2017, at the ICML conference, and is cited more than 800 times at the … Read more
There is more to being a Data Scientist than just technical skills. Lets take a look at what soft skills can help you on your way to becoming a great Data Scientist Pixabay We all see the job boards and we all see the list of doom when it comes to the required technical skillset. … Read more
Using custom Named Entity Recognition and Relation Extraction models Image by Author: Knowledge Graph in Neo4j In my previous article “Building a Knowledge Graph for Job Search using BERT Transformer”, we explored how to create a knowledge graph from job descriptions using entities and relations extracted by a custom transformer model. While we were able … Read more
Understanding the usage of data engineering and machine learning in the electric mobility world Photo by Kumpan Electric on Unsplash We are living in the world of electric mobility. Globally, the adoption of electric cars and two-wheeler is steeply on the rise. Electric mobility devices rely on expensive rechargeable lithium-ion batteries for power. These batteries … Read more
Hands-on Tutorials When Levenberg-Marquardt meets Quasi-Newton. And yes, we build it from scratch with python! Photo by Jeremy Bishop on Unsplash In previous articles, we’ve seen Gradient Descent and Conjugate Gradient algorithm in action, as two of the simplest optimization method there is. We implemented line search for searching the direction to which the objective … Read more
Meet resampling, a one-size-fits-all modern approach to do stats Estimating confidence intervals and hypothesis testing are two common statistical tasks. They have an air of mystery around them as they rely on math accompanied by complex and case-specific assumptions. While this is still the way many people do stats, I argue it is not the … Read more
The most common reason your medical AI will be rejected by clinicians and how to overcome it Image by Evin Felix. So, you’ve found success training algorithms for industrial use, and now you want to train an algorithm to help doctors help patients. Developing algorithms for medical applications is a challenging pursuit but also a … Read more
Reflections from 6 years of developing human-in-the-loop (HITL) tools For most technical folks, coding things up is easy. If there is a tool you are not happy with, you can hack one up yourself without much of a hassle. If you want to extract data, you can quickly write up some regular expressions. If you … Read more
Using Image-Generator to label the Images Automatically as the Subdirectories Convolutional Neural Network is a great tool for Image classification. It can perform other artificial intelligence tasks as well. But this article will focus primarily on image recognition. I have a detailed article on how a forward pass of a Convolutional Neural Network works and … Read more
An End-To-End SQL Interface for Data Science and Analytics As a data scientist, you might be familiar with both Pandas and SQL. However, there might be some queries, transformations that you feel comfortable doing in SQL instead of Python. Wouldn’t it be nice if you can query a pandas DataFrame like below: … using SQL? … Read more
The id column should always be unique and duplicate id values might have severe consequences. We can easily check for the uniqueness of the values in this column. df.expect_column_values_to_be_unique(column=”id”)# output{“meta”: {},”result”: {“element_count”: 1000,”missing_count”: 0,”missing_percent”: 0.0,”unexpected_count”: 0,”unexpected_percent”: 0.0,”unexpected_percent_total”: 0.0,”unexpected_percent_nonmissing”: 0.0,”partial_unexpected_list”: []},”success”: true,”exception_info”: {“raised_exception”: false,”exception_traceback”: null,”exception_message”: null}} The functions of the Great Expectations library return a json … Read more
Customize your training of deep neural networks – a practical guide Photo by John Schnobrich on Unsplash Callbacks are essential when you want to control the training of a model. And you do want to control the training… Callbacks help us prevent overfitting, visualize our training progress, save checkpoints and much more. But why TensorFlow? … Read more
Based on this formula, we can now start to collect data to calculate the emissions. For learning purposes, I have built a dataset with quantities and destinations. I am sharing it (with the code) in this Github repository (Link). P.S: feel free to follow me on Github 🙂 Data Collection — (Image by Author) Import … Read more
It’s uncommon to find an in-house AI Lab at the municipal level of government. I would love to learn more about the team. Could you share what the Lab does at the City of London? What does a typical day look like? The Municipal Artificial Intelligence Applications Lab is a part of the Information Technology … Read more
We can see that the graphs are smoother, and the story stays roughly intact: the slope wiggles around zero, and the intercept increases over time. However, the graphs differ quite a lot, so it is hard to say how the slope and intercept really evolve over time. In summary, this rolling regression approach depends a … Read more
Converging to the right model in a faster manner Photo by Djim Loic on Unsplash It’s 8 PM and you are still cleaning up the data, performing EDA, and creating more features. Your initial discussion with business is first thing tomorrow and the expectation is to discuss what potential “classification” methodologies are in consideration and … Read more