For data science postGRES is rapidly becoming the go to database. As well as being highly scalable and durable, postGRES goes beyond being a SQL…
Tag: Data Science
A lot of datasets include rows with missing values. These values rows either need to be removed or the missing values need to be approximated.…
PySpark window functions are useful when you want to examine relationships within groups of data rather than between groups of data as for groupBy. To…
There are a variety of ways to filter strings in PySpark, each with their own advantages and disadvantages. This post will consider three of the…
Following on from my pandas to pyspark cheatsheet here is another cheatsheet to help convert SQL queries into PySpark dataframe commands. Like the last one…
Recently the Apache Foundation have released a very useful new storage format for use with Spark called Delta. Delta is an extension to the parquet…
Pyspark is very powerful. However because it is based on Scala we need to be careful about types as they are not Pythonic. And because…
I recently gave a talk at PyData Edinburgh about some of the work I am doing at QueryClick. We investigated the effectiveness of TV and…
This is a follow on post from my last post about starting with PySpark and Databricks. Here is a link to a table I have…
Databricks is a very handy cloud platform for large scale data processing and machine learning using Spark. However it does have some idiosyncrasies. Here are…