To do this, you need to first prepare the data source, add image and add the required coordinates in the data source. Here are the steps in detail. 1. Prepare the data source Open a spreadsheet file and add three columns — X, Y, and an identifier. I used Year. Enter random numbers in X … Read more
Whether you are writing a small script for making a detailed project, having a well-structured code with proper names of the modules, correct indentations, and documentation improves the code’s usability in the future. Especially while making projects, you must include a README file for describing your project, the setup.py file for properly setting up your … Read more
Distribution As can be seen above, the quantity of oral antihistamines each year appears to follow a normal distribution. There also appears to be some relationship between the highest pollen count across the types of pollen each day and the quantity of oral antihistamines supplied. This relationship and distrubution must be confirmed and/or quantified. To … Read more
How to schedule a recurring Python script using Google Cloud Scheduler and Cloud Function Photo by Nikolay Tarashchenko on Unsplash Don’t have Airflow set up in your server? Don’t have any experience setting up a cron job to schedule your script? No worries. Today we are going to learn how to schedule your Python script … Read more
Python Generator — from basic to advanced usage If you’re a Python developer, I believe that you must know the Generator in Python. The key to defining a Python generator is to use the “yield” keyword. The Python generator is ubiquitously used in scenarios when we need a large collection, improve the readability of our … Read more
Flip a coin, and you will have one of only two possible outcomes (heads or tails). Pull a card from a standard deck, and you will have one of only fifty-two possible outcomes. When the outcome is one of a fixed number of possible results, we call it a “categorical” outcome. Discrete probability deals with … Read more
IMU (Inertial Measurement Unit) is one of the common sensor to provide motion data in a time-series format. In this post we review it. IMU (Inertial Measurement Unit) sensor provides time-series data, used in Human Activity Recognition problems, Tracking & Navigation problems, and many more. In the AI era, this cheap and reliable sensor can … Read more
Low-code and no-code initiatives appeared a few decades ago as a reaction to the increasingly large skill gap in the coding world. The technical ability to create good code and know how to handle tasks at different points in the design-production pipeline was expensive. As software products got more complex, so did the programming languages. … Read more
As long as you have only short-living requests or long-running tasks with only a few anticipated requests from users, you’re good. Python frameworks like Flask and Django are ideal because you can keep everything in one language — Python. The complexity arises when you have long-running tasks with significant demand. To keep the server up, … Read more
When the data sample follows the power law distribution, we can use log scaling to transform the right skewed distribution into normal distribution. To achieve this, simply use the np.log() function. In this dataset, most variables fall under this category. before transformation (image by author) ## log transformation – power law distribution ##log_var = [‘Income_M$’, … Read more
What are the main NLP text preprocessing steps? The below list of text preprocessing steps is really important and I have written all these steps in a sequence how they should be. Step-1: Remove Accented Characters This is a crucial step to convert all characters like accented characters into machine-understandable language. So that further steps … Read more
A comparison of two different RestAPI frameworks Photo by Daria Shevtsova taken from Unsplash Introduction Being a Data Scientist does not end with Model Building, but working towards the next step, Model Deployment. Model Deployment is important to show your final results to others (or it could be your clients). This group of people might … Read more
Resource policy After replacing the relevant fields in the following JSON add the same resource policy for the API. Finally, in the lambda console, you should observe API triggered Lambda function. API Integration We create an API integration in Snowflake. This integration will create a user and allow that user to assume the role we … Read more
Information Theory An overview regarding one of the emerging research field for Machine Learning and Artificial Intelligence. Image by Author Research on Machine Learning (ML) models has evolved in recent years, leading to the definition of very precise models. In fact, the primary goal of the ML researchers has always been to develop ever more … Read more
How insights from data were used to help build the taxonomy and our approach to assign categories to the user-uploaded documents. Scribd offers a variety of publisher and user-uploaded content to our users and while the publisher content is rich in metadata, user-uploaded content typically is not. Documents uploaded by the users have varied subjects … Read more
8 Vim tips to edit your files faster. (Image by author) Vim is a text editor that in the hands of a skilled user, can enable blazing fast edits closer to the speed of thought — much faster than what’s usually achievable with a traditional text editor. For everything we do on the computer, there … Read more
How to write SQL queries that your team can easily read and maintain? It’s easy to mess with SQL without proper guidelines. Since everybody in your team might have their own habits of writing SQL, you can quickly end up with a confusing code that nobody understands. You probably realized the importance of following a … Read more
In general, we have: Image by Author The number of units in inventory at the beginning of day 1 is 0: Image by Author The daily number of units in inventory cannot be negative: Image by Author The daily production quantity cannot be negative: Image by Author Wait, but don’t we also need to include … Read more
Let’s get started. I have a Raspberry Pi 3, but any Raspberry Pi will work with this setup. All we’ll need is the following to get setup. Raspberry Pi 4GB or greater microSD card Windows, Mac, or Linux computer Adapter(s) to plug in your microSD card into your computer iPhone or Android device Adapter hell. … Read more
First things first, we need to install LIME using pip. You can find the source code for LIME in [2]. pip install lime We will use the iris dataset provided to us by Scikit-learn [3] as an example to demonstrate the package usages. First things first, we need to import the different packages which we … Read more
A step-by-step code and intuition guide to generating album covers. A random sample of generated album covers from WGAN Yeah, GANs can be pretty cool. If you somehow managed to stumble upon this little article, it’s probably safe to say that you’re somewhat interested in generative adversarial networks — GANs. I definitely was. In seeing … Read more
Efficient, organized, and elegant. Photo by Sigmund on Unsplash Real-life data is usually messy. It requires a lot of preprocessing to be ready for use. Pandas being one of the most-widely used data analysis and manipulation libraries offers several functions to preprocess the raw data. In this article, we will focus on one particular function … Read more
The prominent chefs all have their own restaurant and/or unique style of cooking, and they still practice all their fundamentals everyday. There isn’t a single chef breaking the rules that hadn’t mastered the rules in the first place. Essentially, they know the how and why to break rules in a way that’s meaningful. That simply … Read more
And why we need to rethink business intelligence Going for a dive. Photo by Joe Pohle from Unsplashed. As data analysts, we waste too much time on making dashboards for other people and not enough time on answering deep questions about critical business issues. This is a waste of resources for the individual, and a … Read more
How can we classify convenience stores in London? Photo by Viki Mohamad on Unsplash Convenience stores in the UK can be classed as stores that are smaller than 3,000 sqft, located close to consumers’ homes or a large daytime population, and have a wide but shallow product range¹. This means that they are mostly targeted … Read more
Anyone who has ever approached the world of machine learning has certainly heard of supervised learning and unsupervised learning. These are in fact two important possible approaches to Machine Learning that have been widely used for years. Only recently, however, has there been an explosion of a new term, Self-Supervised Learning! But let’s get there … Read more
Finding your Next Job by Building an Indeed.com Job Graph with TigerGraph, Indeed Job Data, and Kaggle API Graphs are everywhere and can help with so much, including finding a job. Platforms like LinkedIn are powered by graph databases to help recommend jobs to you. In this blog, we’ll create an Indeed Graph that can … Read more
In simple words, Copilot really understands what you want to code in the next line. In my case, it even understands bad comments perfectly. Sometimes, it makes a few silly mistakes like declaring the same variable repeatedly; these kinds of bugs were already expected, which is why Github initially gave developers access to give their … Read more
In this article, we’ll take a closer look at how we can measure data variability. In the following sessions, you’ll learn the basics of Standard Deviation and Variance. This work is a sequence of the previous article of this series, where we learned how to measure and visualize the spread of data. If you haven’t … Read more
Most time series forecasting methods assume that the data is ‘stationary,’ but in reality it often needs certain transformations for further processing. Photo by Miguel Luis on Unsplash In the first article, we looked at Simple Moving Average and Exponential Smoothing methods. In this article we will look at more complex methods like ARIMA and … Read more
We try to answer the classic question of our era with topic modeling, sentiment analysis, and some good ol’ stats. Photo by Susan Q Yin on Unsplash It’s the dataset that’s been calling my name. I’m not quite sure what that says about me. Ever since I heard of it, I’ve wanted to explore a … Read more
All Games, without exception, have had cost overruns. For no other type of mega-project is this the case. With Alexander Budzier and Daniel Lunn Photo by Bryan Turner on Unsplash Percentage cost overrun for the Olympic Games 1960–2016 is shown in real terms in the table below. Data on cost overrun were available for 19 … Read more
Keeping up with the Times Photo by Pawel Czerwinski on Unsplash Books, books, books! Before I started my journey in machine learning, I stacked up a pile of books on artificial intelligence from my local Waterstones books store. The idea was to gather up as much knowledge as I can about the potential of AI … Read more
Opinion — University failed me badly to prepare me for my IT and Data Science Career How much worth is a Bachelor’s Degree in IT and Data Science? Photo by Raychan on Unsplash In my current career, I am dealing with several trending topics regarding digitalization such as the renewal of IT through the cloud, … Read more
Which Washington counties will have the most EVs and need the most commercial chargers? If you’ve stepped outside in the past couple months, chances are you’ve felt like a melting scoop of ice cream more than any other summer. Well, it is no coincidence that the global land-only surface temperature for June 2021 was the … Read more
Here are the top 10 retweeted tweets. Top retweeted tweets referencing ‘Simone Biles’ | Skanda Vivek All of the top 10 retweeted tweets are in support of Simone Biles! And here are the top 10 liked tweets. Top 10 liked tweets referencing ‘Simone Biles’ | Skanda Vivek Same in this case — All of the … Read more
The intuition behind ensemble learning is often described with a phenomenon called the Wisdom of the Crowd which means aggregated decisions made by a group of individuals are often better than the individual decisions. There are multiple methods for creating aggregated models (or ensembles) which we can categorize as heterogenous and homogenous ensembles. In heterogeneous … Read more
Path Tracking The kinematic model of the car is: x = vcos(ϕ) y = vsin(ϕ) v = a ϕ = vtan(δ)/L The state vector is: z=[x,y,v,ϕ] x: x-position, y: y-position, v: velocity, φ: yaw angle The input vector is: u=[a,δ] a: acceleration, δ: steering angle Control The MPC controller controls vehicle speed and steering based … Read more
Data Visualisation A ready-to-run tutorial, which describes how to build an animated line chart using Altair and Streamlit. Image by Author Altair is a very popular Python library for data visualisation. Through Altair, you can build very complex charts with few lines of code, since the library follows the guide lines provided by the Vega-lite … Read more
Introduction Arts and Entertainment Utility Script Earth and Nature Summary References This article will outline the most popular data science trends that are designated as tags on Kaggle [2]. From those popular tags, I have picked three that I think are the most surprising. Understanding trends in data science can be helpful in a variety … Read more
In this post, I will explain the data roles that exist today and in particular — who is a data engineer? What are the role definition, responsibilities, and challenges contained in it? Photo by Christina @ wocintechchat.com on Unsplash For the past few years, I have been working as a big-data engineer, and although it … Read more
How to obtain and interpret explanations of predictions BigQuery ML is an easy-to-use way to invoke machine learning models on structured data using just SQL. Although it started with only linear regression, more sophisticated models like Deep Neural Networks and AutoML Tables have been added by connecting BigQuery ML with TensorFlow and Vertex AI as … Read more
In a previous article, we talked about post pruning decision trees. In this article, we will focus on pre-pruning decision trees. Let’s briefly review our motivations for pruning decision trees, how and why post-pruning works, and its advantages and disadvantages. If you’d like some more details, check out this article. Decision Trees are grown using … Read more
There are a total of 5 execution steps when building a Natural Language Processor: Lexical Analysis: Processing of Natural Languages by the NLP algorithm starts with identifying and analyzing the input words’ structure. This part is called Lexical Analysis and Lexicon stands for an anthology of the various words and phrases used in a language. … Read more
Using MLJAR-Supervised for Automating EDA Machine Learning Models and Creating Markdown Reports Source: By Author Exploratory Data Analysis is an important step for understanding the data that we are working on it helps us in identifying any hidden pattern in the data, the correlation between different columns of the data, and in analyzing the properties … Read more
1. Metadata and Features∘ Image features∘ Image metadata2. Semantic Drifts∘ Custom Features — Distances from Cluster Centers3. Conclusion Even though we don’t have an actual model for prediction, let’s assume that our model input is expected to consist mainly of landscape images. Using a simulated production stage, we can test if it’s possible to detect … Read more
Organizations need to deploy AI and machine learning more widely, beyond just data science teams. Yes, democratizing ML will lead to imperfect models, and sometimes even the wrong decision. But imperfect ML is not worse than imperfect Excel business analysis. The available size and scale of data requires more analysts empowered with an upgraded suite … Read more
Table of Contents Introduction What is a Word Embedding? Word2Vec Architecture– CBOW (Continuous Bag of Words) Model– Continuous Skip-Gram Model Implementation– Data– Requirements– Import Data– Preprocess Data– Embed– PCA on Embeddings Concluding Remarks Resources Word2Vec is a recent breakthrough in the world of NLP. Tomas Mikolov a Czech computer scientist and currently a researcher at … Read more
Use of Statsmodels to check Heteroscedasticity Image from Unsplash Introduction Oftentimes, regression analysis is carried out on data that may have a built-in feature of high variance across different independent variable values. One of the artifacts of this type of data is heteroscedasticity which indicates variable variances around the fitted values. When we observe heteroscedasticity, … Read more
Photo by Mika Baumeister on Unsplash There are many datasets on the OWID GitHub repository however they have aggregated many of the key data points into one combined structure. This makes life considerably easier as transformations are in less intensive. Let’s have a look at the CSV. import pandas as pddf = pd.read_csv(‘owid-covid-data.csv’)print(df.shape)(101752, 60)print(‘Total unique … Read more