Image by author The union of field values in SQL is common, such as firstname+lastname and year (birthday). No matter how many fields an expression contains, they come from the same row. We call this intra-row calculation. Correspondingly, there are inter-row calculations. Examples include getting the difference between the result of the champion and the … Read more SQL Order-based Calculations
Algo Trading ADR price dynamics motivate a trading strategy Looking in the same direction? Photo by SK Yeong on Unsplash Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s … Read more Pairs Trading ADR and SPY. The price dynamics of ADR-SPY spreads motivate a mean reversion trading strategy.
… Not another ‘hand-wavy’ CLT explanation… Let’s actually work through the math Photo by Diego PH on Unsplash For anyone pursuing study in Data Science, Statistics, or Machine Learning, stating that “The Central Limit Theorem (CLT) is important to know” is an understatement. Particularly from a Mathematical Statistics perspective, in most cases the CLT is … Read more Central Limit Theorem: Proofs & Actually Working Through the Math
With many of you applying to Data Science positions, it is expected to be asked various sorts of probability questions during the technical aspect of the interview process. Within this post, I aim to cover 5 different probability questions (increasing in difficulty) which I believe serve as a good blanket to the various types of … Read more 5 Probability Questions to Test Your Skills
An unsupervised learning method for learning filters that can extract meaningful features out of images Data is everything. Especially in deep learning, the amount of data, type of data, and quality of data are the most important factors. Sometimes the amount of labeled data that we have is not enough or the problem domain that … Read more Filter Learning with Unsupervised Learning
Data science is one of the fast-growing fields that I can’t see slowing down any time soon. Not with how our data dependence is overgrowing day by day. Data science is all about data, collecting it, cleaning it, analyzing it, visualizing it, and using it to make our life better. Handling large amounts of data … Read more Databases 101: Introduction to Databases for Data Scientists
My first exploration of Machine learning using the Titanic competition on Kaggle Louis & Lola, survivors of the Titanic disaster (Photo from Library of Congress Prints and Photographs, No known restrictions on publication) This article describes my attempt at the Titanic Machine Learning competition on Kaggle. I have been trying to study Machine Learning but … Read more Practical Machine Learning Basics
An Intuitive Explanation and Exploration Kaggle is the data scientist’s go-to place for datasets, discussions, and perhaps most famously, competitions with prizes of tens of thousands of dollars to build the best model. With all the flurried research and hype around deep learning, one would expect neural network solutions to dominate the leaderboards. It turns … Read more XGBoost, LightGBM, and Other Kaggle Competition Favorites
Building Deep Learning models with Keras in TensorFlow 2.x is possible with the Sequential API, the Functional API, and Model Subclassing Figure 1. The Sequential API, The Functional API, Model Subclassing Methods Side-by-Side If you are going around, checking out different tutorials, doing Google searches, spending a lot of time on Stack Overflow about TensorFlow, … Read more 3 Ways to Build Neural Networks in TensorFlow with the Keras API | by Orhan Gazi Yalçın | Medium
We can also index with a single column (as opposed to list): sales_data.groupby(‘month’).agg(sum)[‘purchase_amount’] In this case, we get a Series object instead of a DataFrame. I tend to prefer working with DataFrames, so I typically go with the first approach. Now that we have the basics down, let’s go through a few of the more … Read more Ultimate Pandas Guide — Mastering the Groupby
Regex is known as the IT skill that drastically increases productivity in everything you do on a computer! Image by Author I was astounded when I first learned how to search for texts using Regex and process text data in ways unimaginable to me before. Today we can utilize its power inside Python, but it … Read more Most important IT side skill, Regex
The main 4 things that we need to think of when applying minimax to 2048, and really not only to 2048 but to any other game, are as follows: 1. How we can think of 2048 as a 2-player game? Who is Max? Who is Min? 2. How do we decide when a game state … Read more Playing 2048 with Minimax Algorithm — 1
Unsplash My name is Palash Shah, and I’m the author of Libra: a machine learning library that lets you build and train models in one line of code. I’m also an undergraduate student at the University of Virginia. My journey in the open source community started as a normal college student — I worked on … Read more What I learned as a college student running a large open-source project
The world probably doesn’t need another neural network, but it needs a coffee chat with those on the front lines. By now, you’ve probably seen a few, if not many, articles on how deep learning could help detect COVID-19. In particular, convolutional neural networks (CNNs) have been studied as a faster and cheaper alternative to … Read more Domain Expertise: What deep learning needs for better COVID-19 detection
Machine Learning and Artificial Intelligence are currently driving innovation in the field of Computer Science and they are being applied on a multitude of fields across disciplines. However, traditional ML models can be still be broadly categorized into solutions of two types of problems. Classification — Which aims at labelling a particular instance of data … Read more Learning to Rank for Information Retrieval: A Deep Dive into RankNet.
A Reinforcement Learning Implementation in Pytorch. Over the last few articles, we’ve discussed and implemented various value-learning architectures for the VizDoom environment, and examined their performance in maximizing reward. To summarize, these include: Overall, vanilla Deep Q-learning is a highly flexible and responsive online reinforcement learning approach that utilizes rapid intra-episodic updates to it’s estimations … Read more Building the Ultimate AI Agent for Doom using Dueling Double Deep Q-Learning
2.5. Mutation Mutations refer to small random choices made in individuals of a population. It is essential in ensuring genetic diversity and enabling the genetic algorithm to search a broader space. In the context of Decision Trees, it can be implemented by randomly change an attribute and split the value of a node randomly selected. … Read more Evolutionary Decision Trees: When Machine Learning draws its Inspiration from Biology
Value scoping, discovery, delivery, and stewardship Created by Authors based on Youtube video Monarch Butterfly Metamorphosis time-lapse FYV 1080 HD In Part 1 of this series we examined the key differences between software and models; in Part 2 we explored the twelve traps of conflating models with software; and in Part 3 we looked at … Read more Model Lifecycle: From ideas to value
Installation process is pretty straight forward. Just open your terminal and insert the following command: pip install matplotlib A. Line Plot After having installed the library, we can jump on to plot creation. The first type we’re going to create is a simple Line Plot: # Begin by importing the necessary libraries:import matplotlib.pyplot as plt … Read more Business Intelligence Visualizations with Python
Just some of the steps involved in prepping a dataset for analysis and machine learning. Source: Image Created by Author Forbes’s survey found that the least enjoyable part of a data scientist’s job encompasses 80% of their time. 20% is spent collecting data and another 60% is spent cleaning and organizing of data sets. Personally, … Read more Data Processing Example using Python
Helping you take your first step into the machine learning lifecycle flow with this handy tool Hello again friends! We’re back here with another quick tip, and because I do attempt to keep these posts quick, this is actually going to be part one in a series of tips related to MLFlow. In the spirit … Read more MLflow Part 1: Getting Started with MLflow!
A Wittgensteinian Approach to Data Science Planetarium from 1766, Photo by Sage Ross, Creative Commons The term model gets thrown around a lot. The word is ubiquitous to the point of lost meaning. The Wikipedia page alone shows the variety of usage of the word model, including statistics, astronomy, biology, product design, art, as well … Read more But What is a Model?
Not long after I began working on machine learning projects in Python, I ran into computationally-intensive tasks that just took a long time to run. Usually this was associated with some kind of iterable process. A couple that immediately come to mind are (1) running a grid search on p, d, and q orders to … Read more Progress bars for Python with tqdm
STEP 3: Query! Now were ready to start querying! The defined class only provides a handful of basic functions. Let’s walk through how to use the class and what we can do with it. First, we’ll need to specify our PostgreSQL connection arguments, and SSH arguments (if SSH tunneling is required to access the remote … Read more How to Query PostgreSQL using Python (with SSH) in 3 Steps
Deep learning is known to work well when applied to unstructured data like text, audio, or images but can sometimes lag behind other machine learning approaches like gradient boosting when applied to structured or tabular data.In this post, we will use semi-supervised learning to improve the performance of deep neural models when applied to structured … Read more Training Better Deep Learning Models for Structured Data using Semi-supervised Learning
TL;DR — Latent Dirichlet Allocation (LDA, sometimes LDirA/LDiA) is one of the most popular and interpretable generative models for finding topics in text data. I’ve provided an example notebook based on web-scraped job description data. Although running LDA on a canonical dataset like 20Newsgroups would’ve provided clearer topics , it’s important to witness how difficult … Read more Latent Dirichlet Allocation: Intuition, math, implementation and visualisation
Learn how to interpret a black box model using SHAP (SHapley Additive exPlanations) Photo by Frank Vessia on Unsplash Article Outline Why SHAP (SHapley Additive exPlanations) About Dataset Loading Dataset Model Fitting Shaply values estimation Variable Importance plot Summary plot Dependence Plot Force Plot Tutorial DataSet Why SHAP (SHapley Additive exPlanations)? The very common problem … Read more Machine Learning Model Explanation using Shapley Values
Data Warehousing | Data Lake | Parquet Understand why Parquet should be used for warehouse/lake storage Apache Parquet is a columnar storage format available to any project […], regardless of the choice of data processing framework, data model or programming language.— https://parquet.apache.org/ This description is a good summary of this format. This post will talk … Read more Understanding Apache Parquet
To provide some more clarity, suppose we have the following table: If we wanted to get the average GPA by gender, we could use an aggregate function and run the following query to get the following result: SELECT Gender, AVG(GPA) as avg_gpaFROM studentsGROUP BY Gender result The next part is key. Now suppose we wanted … Read more SQL Window (Analytic) Functions Explained in 4 minutes
The matrix with topics per documents is rather sparse so we got exactly what we needed. Image by the author It will be convenient to read the articles which relate to the particular topic. So here we can obtain a list of articles sorted on topic probability. Image by the author Building hierarchy The topics … Read more Hierarchical topic modeling with BigARTM library
Artificial Intelligence (AI) is a part of our daily lives — from language translation to medical diagnostics and driverless cars to facial recognition — it’s making more of an impact on industry and society every day. But what exactly is AI? Simply put, AI is a technology that replicates human intelligence through computers, systems or … Read more What is AI? A straight-forward introduction
Let’s create a simple example with some mock data. In this example we will aggregate generic posts and determine how many posts each profile has, then we will aggregate comments. If you are using the code snippets to follow this article, you will want to create a few data points following the style below. However … Read more On Demand Materialized Views: A Scalable Solution for Graphs, Analysis or Machine Learning
“Social distancing” has become very popular these days but it is not always obvious how the rules can fit our daily life. In this story, we are going to study a social distancing problem and find solution to it using Genetic Algorithms. After setting the problem and its constraints, I’ll summarize the principles of Genetic … Read more Solving a Social Distancing Problem using Genetic Algorithms
Working out the cross entropies of each observation shows that when the model incorrectly predicted 1 with a low probability, there was a smaller loss than when the model incorrectly predicted 0 with a high probability. Minimizing this loss function will prevent high probabilities from being assigned to incorrect predictions. To demonstrate why cross entropy … Read more Why and How to use Cross Entropy
Image by Author Forward Pass X is the input image, say (3*3 matrix) and Filter is a (2*2 matrix). Both will be convoluted to give output XX (2*2 matrix). Now, XX will be flattened and will be fed to a fully connected network with w (1*4 matrix) as weights, which will give an— Output. Finally, … Read more Convolution Neural Network Maths Intuition
I come from the world of MATLAB and numerical computing, where for loops are shorn and vectors are king. During my PhD at UVM, Professor Lakoba’s Numerical Analysis class was one of the most challenging courses I took and the deep knowledge of numerical code still sticks with me. My favorite example of a vectorization … Read more Vectorizing code matters
A Comprehensive Guide to Pytest for your Data Science Projects Photo by Startup Stock Photos from Pexels It is fun to apply different python code to process your data in your notebook, but in order to make your code reproducible, you need to put them into functions and classes. When you put your code in … Read more Pytest for Data Scientists
Conducting a data science/analytics project always takes time and has never been easy. A successful and comprehensive analytics project is way beyond coding. Instead, it involves sophisticated planning and a large amount of communication. Photo by Octavian Dan on Unsplash What is the Life Cycle of an Analytics Project? To complete a data science/analytics project, … Read more How to Plan and Organize a Data Science Project? | by Yin Zhang
Lessons learned from building an open-source deep learning for time series framework. Photo by author (taken while hiking at the Cutler Coast Preserve in Machias ME) Note this is roughly based on a presentation I made back in February at the Boston Data Science Meetup Group. You can find the full slide deck here. I … Read more How to make your deep learning experiments reproducible and your code extendible
Photo by Laura Woodbury from Pexels Master the difference between “Merge” and “Join” Everyone who works in data knows this: before you build machine learning models or produce stunning visualizations, you have to get your hands dirty with data wrangling. And one of the core skills in data wrangling is learning how to join together … Read more Ultimate Pandas Guide — Joining data with Python
A retrospective on my summer as a data scientist and how GSI Technology’s summer program breaks the internship status quo. GSI Technology. Reposted with Author’s Permission Data science is a field that can be hard to break into, especially if you are an undergraduate student. My name is Braden Riggs and some of you reading … Read more A Summer as a Data Scientist
Roughly speaking, there are two important operations that make a neural network:1. Forward propagation2. Backpropagation Forward propagation This is the prediction step. The network reads the input data, computes its values across the network, and gives a final output value. But how does the network computes an output value? Let’s see what happens in a … Read more Convolutional Neural Network: How is it different from the other networks? | by YANG Xiaozhou | Sep, 2020 | Towards Data Science
An easy to use trading simulator to test trading (ML/AI) algorithms and strategies on Python Simulation of AAPL on September 9th, 2020, using TMS. Sometime ago, I wrote an article on how to download stocks market data for free using Alpaca, a trading broker and API. I published this article because I had worked on … Read more Introducing TMS: a Trading Market Simulator
👋 It’s easiest to generate a Personal Access Token to use the API. At this point, it’s best to try and access the GitHub API to make sure our Authentication and URLs are correct. Adjust the def’s below to your own info/settings and paste them into the REPL: Promises The HttpKit (http/get) function returns a … Read more A gentle intro to Clojure
It goes without saying that choosing an evaluation set is the most important step in any machine learning process. This becomes even more crucial when it comes to active learning since this will be our measure of how well our model performance improves during our iterative labelling process. Furthermore, it also helps us decide when … Read more How to do more with less data ?— Active learning
If you’re reading this, then you’re most likely going through the grind of preparing for technical interviews (software engineers, data scientists, etc). By now you should know that tech interviews are not like regular ‘old school’ interviews — we can’t just woo the hiring manager with our charm and talk out of our ass — … Read more Become an Expert at the Technical Interview — Part I
Supervised Learning Methods in Machine Learning Image from ¹wikicommons In previous articles, I talked about deep learning and the functions used to predict results. In this article, we will use logistic regression to perform binary classification. Binary classification is named this way because it classifies the data into two results. Simply put, the result will … Read more Logistic Regression for Binary Classification
Especially for recent graduates aspiring to become the next tech superstars Courtesy of Janko Ferlič from Unsplash It is well know that Leaders are Readers, and this does hold true in the tech industry just like it does anywhere else. However, this article is not about preaching to you about the necessity of reading for … Read more 3 Books for Successful Programmers in 2020 and beyond
DATA ANALYSIS Turn raw data into functional form Following acquisition of raw data, data wrangling is the most essential step to transform raw data into more functional form for data analysis, model building and data visualization. If involves preprocessing, restructuring and cleaning operations and the end product is a dataset in a readily accessible format, … Read more Data Wrangling in Pandas : A Downloadable Cheatsheet
As the ultimate store of value, gold has appreciated substantially over the past 18 months; rallying from ~$1,300 in March 2019 to $1,954 on September 18 th; an increase of 33.4%. In fact, gold as an asset class has performed very well over the past 20 years and has markedly outperformed stocks; though this has … Read more Drivers of Gold Returns in R