Feature generation is the process of creating new features from one or multiple existing features, potentially for using in statistical analysis. This process adds new information to be accessible during the model construction and therefore hopefully result in more accurate model. In this article I describe how to used feature interaction detection algorithm based on … Read more
“There is no real ending. It’s just the place where you stop the story.” ― Frank Herbert I am now about to start my eight-month as a writer for Towards Data Science (TDS) and today I decided to take a look at my journey so far. Having a PhD in order to work in Data … Read more
Decision trees are among the most powerful Machine Learning tools available today and are used in a wide variety of real-world applications from Ad click predictions at Facebook¹ to Ranking of Airbnb experiences. Yet they are intuitive, easy to interpret — and easy to implement. In this article we’ll train our own decision tree classifier … Read more
Hey Google, what does this smell like? Deep learning has made many advances in sight — using computer vision to identify objects, detect cancer in cells, and self-driving cars. It has also made many advances in sound — live captioning, AI generated music, and offline speech recognition are some examples. It is because of these … Read more
Google Books ruled legal in massive win for fair use (updated), Ars Technica Nov 14 2013. Google Wins: Court Issues a Ringing Endorsement of Google Books, Publishers Weekly, Nov 14, 2013. Google book-scanning project legal, says U.S. appeals court, Reuters, October 16, 2015. “We trust that the Supreme Court will see fit to correct the … Read more
Amazon DocumentDB is a fast, scalable, highly available, and fully managed document database service that supports MongoDB workloads. You can use Amazon DocumentDB in the following AWS regions: US East (N. Virginia, Ohio), US West (Oregon), Europe (Paris, Ireland, Frankfurt, London), and Asia Pacific (Mumbai, Singapore, Sydney, Tokyo, Seoul). For more information on AWS … Read more
TensorFlow 2.0 comes with Keras packaged inside, there is no need to import Keras as a separate module (although you can do this if you need). TensorFlow 2.0 API is simplified and improved. This is good news for us — Machine Learning developers. This is how you import Keras now, from TensorFlow: from tensorflow import … Read more
Generative design technologies can be used to emulate and reconfigure things that already exist. There has already been much discussion on the impact of “deepfakes” — an application of deep learning that creates fake photos, videos, and writing based on their real counterparts. But there’s been less discussion on how entire works might be pulled … Read more
Why do we keep talking about security all the time? Why hasn’t anyone just gone and fixed it? You’ve probably heard these questions, whether from your leadership, or a board member, or just from friends. Then you labor at explaining why security in the cloud is so complex and challenging, the constant arms race, and … Read more
The new challenge for mature FinTechs is to offer less with more and I have the formula to solve that. When the FinTech industry was just coming of age, most companies launched with one simple offering: a mobile wallet, debit card, or spare change investing app. Those products were a hit with users who were … Read more
“At Pandora, we have petabytes of data spread across multiple Google Cloud storage services; accordingly, we expect BigQuery’s federated query capability to be a useful tool for integrating our diverse data assets into a unified analytics ecosystem,” says Greg Kurzhals, product manager at Pandora. “The support for Parquet and other external data source formats will … Read more
Editor’s note: This is the 11th episode of the Towards Data Science podcast’s “Climbing the Data Science Ladder” series, hosted by Jeremie Harris, Edouard Harris and Russell Pollari. Together, they run a data science mentorship startup called SharpestMinds. You can listen to the podcast below: One thing that you might not realize if you haven’t … Read more
One year ago today (yes, on Halloween), I started a data science bootcamp. This seemed like as good a time as any to look back and share some thoughts and takeaways. Bootcamps are not for everyone. The cost can limit who participates. If you do a bootcamp, focus on the bootcamp. What I learned is … Read more
Being an avid techie and problem solver, my mind is always looking out for opportunities to apply what I’ve learnt. Other than during my time in internships, I haven’t really put my school fees to good use. Until one fateful day, a notification showed up on my phone: My mum’s daily routine of collating meal … Read more
Before we dive into the idea behind over and under fitting, let’s try to understand behind the scenes working of a Machine Learning model. What do you think happens when you give data to a machine learning model? Imagine a black room filled with white dots. Since a room implies 3 dimensions (height, width and … Read more
[This article was first published on R – Fantasy Football Analytics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t. ShareTweet The post Gold-Mining Week 9 (2019) appeared first … Read more
One key area where companies often become concerned and willing to exploit data to get guidance on where the problems are located and how they can be solved is related to consumer retention or user engagement. So, let’s imagine that we are analysts working for a technological company whose most important KPI revolves around how … Read more
Our Moneyball approach to the Fantasy EPL (team_id: 2057677) If this is the first time you land on one of my Fantasy EPL Blogs, you might want to check out Part1, Part2, Part3, Part5, and Part9 first to get familiar with our overall approach and the improvements we’ve made over time. My partner in crime … Read more
How can we create a heat map without writing a single line of code? It sounds impossible, but it’s actually very easy to implement. Sometimes, we need to be good at using tools and learning tips to help us save time and improve work efficiency. Today I will show you how to make an earthquake … Read more
A complete end-to-end process for classifying text In this article, we will utilize Tensorflow 2.0 and Python to create an end-to-end process for classifying movie reviews. Most Tensorflow tutorials focus on how to design and train a model using a preprocessed dataset. Typically preprocessing the data is the most time-consuming part of an AI project. … Read more
Photo Credit: Pixabay, licensed by Creative Commons Images Walking your dog in one of the city’s parks may be your daily routine or you may occasionally visit the park to get some fresh air. Regardless of the purpose of your visit, safety is the most critical concern. Talking about safety, I recall a great quote … Read more
For this example, we are going to log in thru Instagram’s website app. I actually use Python and Selenium on my daily workflow. At the company I work for we have our own web application that sends reports online. With each report we have an account. Since we have customers onboarding each day we also … Read more
[This article was first published on NEONIRA, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t. This is the third post on offensive programming, dedicated to using offensive programming … Read more
You know what they say, “It’s what’s on the inside that counts.” While this may be true, you still have to find a way to express what’s inside so that people can understand it. You might have the most functional piece of code you’ve ever written. It might be elegant, thorough, and foolproof. But it … Read more
Photo by Jyotirmoy Gupta on Unsplash It was recently announced that the Google Pixel 4 would be replacing its fingerprint technology with facial recognition technology to increase security. This was one of the most exciting selling points for the model — but it didn’t take long to discover the loopholes. It was quickly discovered that … Read more
From a toy car to AlphaGo and autonomous Teslas In March 2016, Lee Sedol, the greatest Go player of the past decade, was defeated 4–1 by AlphaGo. Computers have beaten the best humans at chess before, but Go is at another next level at complexity. Do you know what’s even crazier? The machine had only … Read more
AWS App Mesh is a service mesh that provides application-level networking to make it easy for your services to communicate with each other across multiple types of compute infrastructure. App Mesh standardizes how your services communicate, giving you end-to-end visibility and ensuring high-availability for your applications. Favorite
Jupyter notebook autocompletion How can you use Jupyter notebook autocompletion? The good news is: you do not install anything as it comes with the standard jupyter notebook set up. To start using autocompletion you have to start typing your variable name and hit the tab button on your keyboard. When you do it the box … Read more
Packages required These are the packages I used import csvfrom bs4 import BeautifulSoupimport pandas as pdimport requests csv allows you to manipulate and create csv files. BeautifulSoup is the web scraping library. Pandas will be used to create a dataframe to put our results into a table. Requests is used to send HTTP requests; to … Read more
It takes context, plots, algorithms, metrics, and fiddling! Some of the students in a data science course showed me this very interesting dataset about Brazilian states they found on the internet. It was love at the first plot! But it took much more than just the first plot to grasp some of its nuances. If … Read more
As before, let’s start with some basic definition: Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them.[2] By Giphy: https://giphy.com/gifs/CiYImHHBivpAs/html5 I know, it’s not the cleanest of definitions, but I find it good enough. It requires some math knowledge, … Read more
In sequence learning problems, we know that the true output at timestep ‘t’ is dependent on all the inputs that the model has seen up to the time step ‘t’. Since we don’t know the true relationship, we need to come up with an approximation such that the function would depend on all the previous … Read more
The data ops team should provide tools to lower the barrier for all your employees, not only the Java or Python developers. ETL/ELT should be easy to do for everyone, maybe via SQL instead of Java or maybe using a data integration tool that has a drag and drop interface, such as [CDAP] or [Matillion]. … Read more
Data is at the heart of all AI initiatives. Put simply, you need to collect and store a lot of it to train a deep learning model, and with the advancements and increased availability of accelerators such as GPUs and Cloud TPUs, the speed of getting the data from its storage location to the training … Read more
Before we can do anything else, we have to find where the data lives and who can provide us access. This step is usually harder than it sounds. Large enterprises will have well-structured data sitting in a dozen different silos (each with their own owner), while startups and smaller companies lack the time and funding … Read more
But most of us will hopefully work for growing, high potential companies (less stress of being laid off). Thus, let’s take a look at the two primary deciders of whether our firm will achieve liftoff and ultimately become profitable and successful — customer lifetime value and customer acquisition cost. Photo by Chevanon Photography from Pexels … Read more
[This article was first published on Fabian Dablander, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t. Bayesian inference using Markov chain Monte Carlo methods can be notoriously slow. … Read more
We’ll begin by gathering our data. We’ll use a time period going back about five years, October 28, 2014 to October 28, 2019. The stocks that we will get data for are the components of the Dow Jones Industrial Average. Yahoo Finance used to be really easy to get data from, but most packages no … Read more
After visiting Portland, OR last weekend I’ve decided to explore some publicly available datasets about the city. In this post, we are going to calculate the number of incidents related to vehicles (theft from or theft of a vehicle) and the number of parking spots in each Portland neighborhood using Athena geo queries. After that, … Read more
The saving grace of the GPU comes in the form of packages designed with CUDA in mind. Most statistical machine-learning operations involve moving high volumes of data in the form of numerical matrices and computing values in real time. Luckily, this is precisely what a graphics card is designed to do. Although there are short-comings … Read more
This post is a continuation of our earlier attempt to make the best of the two worlds, namely Google Colab and Github. In short, we tried to map the usage of these tools in a typical data science workflow. Although we got it to work, the process had its drawbacks: It relied on relative imports, … Read more
Jupyter Notebooks are great (see my full notebook here): Easy way to annotate my thinking as I test out different ideas — helps when revising later or passing it to someone else! Quicker to setup and run than a VS Code proj — the ability to isolate cells and run independently is like modified REPL! … Read more
Let’s dig into the five golden rules. Rule #1: A Clear Layout Gets 90% Job Done. I will let my readers know what the post is about right after they open the link. In this post and this post, I explain my intention in the title and present the main findings in the Executive Summary … Read more
Today I want to break down the central limit theorem and how it relates to so much of the work that a data scientist performs. First things first, a core tool to any data scientist is a very simple chart type called a histogram. While you’re sure to have seen many a histogram, we often … Read more
[This article was first published on R by R(yo), and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t. As another sweltering summer ends, another TokyoR Meetup! With globalwarming in … Read more
This spooky post was written in collaboration with Yoav Kessler (@yoav_kessler) and Naama Yavor (@namivor).. Experimental psychology is moving away from repeated-measures-ANOVAs, and towards linear mixed models (LMM). LMMs have many advantages over rmANOVA, including (but not limited to): Analysis of single trial data (as opposed to aggregated means per condition). Specifying more than one … Read more
Location, Location, Location! You have heard this many times. It is a common mantra in real estate. Does that apply in data science as well? How do we embrace the location component in Data Science? Is it only another column in your dataset? Or perhaps spatial is special. Location data (big data) is ubiquitous as … Read more
Photo by Lenin Estrada on Unsplash Working at a startup, we needed to automate messages in order to get notified of certain events and triggers. For example, the company I work with deals with connections to certain stores. If that connection is broken Python will read that information in our database. We can now send … Read more
We are living in an era where messaging apps deal with all sorts of our daily activities, and in fact, these apps have already overtaken social networks as can be indicated in the BI Intelligence Report. In addition to this clear point, the consumption of messaging platforms is further expected to grow significantly in the … Read more
FORECASTING cost model is a prerequisite to the development and validation of new optimization methods and control tools. Here, I will show a simple yet powerful approach of forecasting using machine learning algorithms. Psychics and fortune tellers have used Tarot cards for hundreds of years, and Trusted Tarot will give us an accurate reading that’s … Read more