Featured

How a team obtained private data, constructed a fake AI model, and got away with the money from a platform for adopting neglected pets The cheaters stole from Petfinder.my, a platform for adopting homeless and neglected pets. [pixabay image] Kaggle just announced that the 1st Place Team, Bestpetting[1], has been disqualified from the Petfinder.my competition … Read more Kaggle 1st place winner cheated, $10,000 prize declared irrecoverable

Two reasons why SQL will never, ever die Last week a friend forwarded me an email from a successful entrepreneur that pronounced “SQL is dead.” The entrepreneur claimed that the wildly popular, No-SQL databases like MongoDB and Redis would slowly strangle SQL-based databases out of existence, and therefore learning SQL as a data scientist was … Read more Is No-SQL killing SQL?

Increasing host revenue with regression and time series analysis [This project was done as part of an immersive data science program called Metis. You can find the files for this project at my GitHub and the slides here. The final project is accessible here (interactive web app).] I recently designed a new approach to automatic … Read more Smarter Pricing for Airbnb Using Machine Learning

Conclusions Let’s say you vacation to a remote island for a week with no connection to the outside world. When you come back, Team A and Team B played the 100 games and your friend asks “Hey, what if I told you they only won 93 games?” “HA, I’d say the probability of that happening … Read more Wait, so what’s a P-Value?

3 techniques to make your data analysis faster Pandas has exceptional features for analyzing time-series data, including automatic datetime parsing, advanced filtering capabilities, and several datetime-specific plotting functions. I find myself using those features almost every day, but it took me a long time to discover them: many of Pandas datetime capabilities are not immediately … Read more Getting started with Pandas time-series functionality

Movie Data and Box Office Numbers In order to build my prediction algorithm, I gathered movie data from a couple online sources. I obtained the bulk of my data from the Internet Movie Database (IMDb) which provides a set of files for free download. However, the IMDb files do not contain data on estimated movie … Read more Predicting Movie Profitability and Risk at the Pre-production Phase

The American education world, it seems, is just now catching up to what Data Scientists have known for years. You learn from your mistakes. You’re not alone if it sounds farfetched that our education system is just now figuring this out. It’s an age old adage, but somehow it skipped applied education theory. Don’t believe … Read more Learning from the ROC

NLP, Knowledge Graphs, and the Three Pillars of Intelligence Jibo, Echo/Alexa, Google Home The Three Pillars of Intelligence To Amazon, the reception for its voice agent, Alexa, was a big surprise. Apple’s Siri had put voice input onto smartphones. But here was a new class of device that you could shout at across the kitchen … Read more How Do Conversational Agents Answer Questions?

NLP: Sentiment Analysis, Word Embeddings and Topic Modelling of 3,8K tweets Last week, from the 7th to 9th of January, Oxford hosted the well-established, traditional and businessy Oxford Farming Conference (OFC) and its antidote Oxford Real Farming Conference (ORFC). Both aims to connect actors involved in the agricultural and food sector to tackle the challenges … Read more Oxford (Real) Farming Conference 2020

Cloud computing has become a fundamental requirement for most organizations. With this in mind, cloud computing is massively on the rise in the current day and age. In fact, 81 percent of companies with 1,000 employees or more have a multi-platform strategy. The number is to rise to more than 90 percent by 2024. Between … Read more Exploring the Future of Cloud Computing in 2020 and Beyond

Commercial Activity Data: Points of Interest (POIs) This dataset provides information on POIs. Cells are enriched with the following data from this source: Number of competitors within a 250-meter buffer from the cell’s centroid. Number of POIs within a 250-meter buffer from the cell’s centroid. Here we use POIs as a proxy for commercial activity. … Read more Site Planning for Market Coverage Optimization with Mobility Data

Modern CPUs have undergone significant development. But how does MonetDB exploit this development to maximize its performance? Computer processors have significantly developed in the last three decades. This development involves not only the increasing number of transistors it holds but also the evolution of the architecture. Hence, an application needs to adapt to how the … Read more How MonetDB/X100 Exploits Modern CPU Performance

Source: Disney A definition to save the dataframe from extinction Dataframes emerged from a specific need, but because so many diverse systems now call themselves dataframes, the term is on the verge of meaning nothing. In an effort to preserve the dataframe, we formalized the definition based on the original data model in our recent … Read more Preventing the Death of the Dataframe

Financial habits have historically been something that people place back of mind, but with the rising amount of information and tools available, a new attitude to financial control is creating a rising popularity in transparent, digital banking. A new breed of financial institutions (such as Monzo, Starling, Revolut and N26) are leveraging digital products to … Read more Visualising spending behaviour through open banking and GIS

Data science path is not easy. If you’re a data scientist reading this now. You’ll know what I mean. It’s tough. It’s an ever-changing field. It’s dynamic. It’s moving fast. In other words, you need learn fast and adapt quickly to keep up-to-date with the latest trend and technology being used in the industry. The … Read more How to use the power of “WHY” to achieve what you want

When to bet big and when to bet small A few weeks ago, my friends and I were playing bar trivia at our favorite bar in town. For those who don’t know, bar trivia is a competition where teams try to answer progressively harder trivia questions for points. The team with the most points at … Read more How to wager at the final questions in bar trivia

New research provides a quantitative solution to Fermi’s paradox Simulated settlement trajectories, showing how civilizations could spread through the Galaxy (source) Compared to the age of the Milky Way Galaxy, our 200,000-year old human species has only been around for just the blink of an eye. The Milky Way is at least 10 Billion years … Read more The Galactic Island Hypothesis

Image Source: Unsplash A major breakthrough in the omics area came in early 2000 with the single cell RNA sequencing (scRNA-seq) technology. The ability to isolate and sequence the genetic material of single cells allows researchers to identify which genes are active in each cell. This provides unprecedented opportunities over bulk RNA sequencing technologies, that … Read more Guide to Dimensionality Reduction in single cell RNA-seq analysis

Numerical: data expressed with digits; is measurable. It can either be discrete (finite number of values) or continuous (infinite number of values). Categorical: qualitative data classified into categories. It can be nominal (no order) or ordinal (ordered data). Mean: the average of a dataset.Median: the middle of an ordered dataset; less susceptible to outliers.Mode: the … Read more Basic Statistics You NEED to Know for Data Science

I recently had the privilege of participating on a panel with several AI/Machine Learning experts. There were many great questions, but most were related to how to most effectively establish an AI/Machine Learning (AI/ML) in a large organization. This gave me an opportunity to reflect on my own experiences helping large enterprise accelerate their AI/Machine … Read more Enterprise AI/Machine Learning: Lessons Learned

Where did all these numbers come from? Well, the input nodes were already given, Z is the result for multiplying the input activations with the weights and add them together, y^ is the result for multiplying Z with its weight. Finally, y is also given which is the desired output. Generally, to compute any output, … Read more Under the Hood of Deep Learning

Data load processes should not be written twice, they should be generalized Photo by Max Nelson on Unsplash We use Airflow as our data pipeline orchestrator in order to easily orchestrating and monitoring data processes. Particularly, we’ve been working on data load processes in order to make them easier. These processes allow us for extracting … Read more Generalizing data load processes with Airflow

Photo by Priscilla Du Preez on Unsplash Recently, I had the opportunity to build a PoC (Proof of Concept — a demo) to resolve a specific computer vision problem and it was a cool experience, so why not share it? The goal of the demo was to detect in real-time, having as input a video … Read more Real-Time Fingers Detector Over an Object — A Working Example

Is the only thing standing between you and Hadoop just trying to figure out how to install it on a Mac? A quick internet search will show you the lack of information about this fairly simple process. In this brief tutorial, I will show you how you can very easily install Hadoop 3.2.1 on a … Read more Installing Hadoop on a Mac

Data sources and assumptions For this analysis, I assembled exit poll data, along with voter participation rates and population statistics, all broken down by age. The exit poll data was taken from Edison Research, a firm which does exit polling for a media consortium including ABC, The Associated Press, CBS, CNN, Fox and NBC. Voter … Read more What if more young people had voted in 2016?

Elo rating system The Elo rating system is a method for calculating the relative skill levels of players in zero-sum games. It has been mostly used in chess. But many have applied the same algorithm in other competitive games such as soccer, basketball and scrabble. In this article, we will predict the winner of this … Read more Spoiler Alert: Conor McGregor vs Cowboy? Who Will Win?

A Practical Guide to understanding one of the most important concepts in statistics Photo by Verne Ho on Unsplash Central Limit Theorem (CLT for short) is one of the most important concepts in the field of statistics. In this post, I will try to explain this concept in a simple and non-technical manner. Introduction Let’s … Read more Understanding the Central Limit Theorem

DATA SCIENCE WITH MICROSOFT POWER BI Detecting outliers and anomalies using the three-sigma rule of thumb in Power BI with no code Photo by timJ on Unsplash Even in the smallest of all data projects, one of the most important steps is detecting abnormal values, outliers or anomalies within your data structure. In this brief … Read more 68–95–99.7 — The Three-Sigma Rule of Thumb Used in Power BI.

How to best evaluate a recommender system is a topic of debate. Let us see how we can use NDCG measure to evaluate a recommendation engine. We are in an era of personalization. The user wants personalized content and businesses are capitalizing on the same. Recommendation Engines, usually built using Machine Learning techniques, have become … Read more Evaluate your Recommendation Engine using NDCG

I have been guilty of talking about Information Theory and Shannon at inappropriate times, generally, you would find me blurting out Information about Shannon at random social events. I shouldn’t be making puns, I am clearly bad at them! Some believe that by studying scientific discovery in another day we can learn how to make … Read more The Abundance of Entropies

Data Science Here’s the book you should read to learn Data Science from scratch. Data Science. It’s the sexiest job of the 21st century and everyone is talking about it. Companies are eager to hire the best talents and people are enthusiastic to jump in the data science boat. With data growing exponentially and our … Read more The Best book to Start your Data Science Journey

Congrats~you have gone through all the theoretical concepts of the regression model. Feel bored?! Here’s a real case to get your hands dirty! picture from APPLE Imagine that you are a store manager at the APPLE store, increasing 10% of the sale revenue is your goal this month. Therefore, you need to know who the … Read more From Linear to Logistic Regression Explained Step by Step

A simple explanation of Sequence to Sequence model for Neural Machine Translation(NMT) What is Neural Machine Translation? Neural machine translation is a technique to translate one language to another language. An example could be converting English language to Hindi language. Let’s consider if you were in an Indian village where most of the people do … Read more Intuitive explanation of Neural Machine Translation

In search of unifying paradigms in statistics Prelude I have a terrible memory. In college my first love was biology, but eventually I was turned off by the classes because of their emphasis on memorization. I was drawn to mathematics as a study because I didn’t have to memorize anything. In math, the main point … Read more Everything is just a Regression

This section contains a detailed description of the training experiments performed on our Log-domain MLP, which we designed using the mathematics outlined thus far and neural network MLP architecture knowledge. Fixed-Point Bitwidth analysis in Log-domain All our neural network training and inference experiments are conducted using fixed-point arithmetic. But how does bit-widths scale when we … Read more Neural Networks Training with Approximate Logarithmic Computations

Exploring the RDW license plate data set with BigQuery, Cloud Storage and Data Studio Last year, driving the Dutch highway, I noticed that the amount of Tesla’s on the road has been increasing rapidly. There are a few good reasons for this. The adoption of full-electric (lease) cars is actively supported by the Dutch government … Read more The exponential adoption of Tesla in the Netherlands

How zines can teach state-of-the-art skills Photo by Giulia Bertelli on Unsplash Every day, our kids are swept through the world by algorithms. YouTube algorithms decide what videos they watch, GPS algorithms map what route they take to school, Spotify algorithms select what songs they hear, and personal assistants like Siri and Alexa advise them … Read more Helping Kids Play With Artificial Intelligence

I just searched on Indeed for “Data Scientist.” Of the 20 jobs on the first page 6 of them did not mention a PhD, 3 said it would be nice to have, and 11 had it as a requirement. I didn’t spend hours looking at job boards or automate a system to scrape them all … Read more Most Data Science Jobs Descriptions Should Stop Requiring a PhD

Zynga and the data law Zynga, founded in 2007, is the company behind FarmVille and lots of other very successful mobile games; in 2018 they had close to 1 billion USD in revenue, 15 million USD net income and close to 2,000 employees. In their 12 years of company history, they introduced a data-driven cult … Read more Democratize Data like Zynga, Facebook and Ebay Do