/categories/data-science/ Recent content in Data Science on ckp Hugo en-us Tue, 10 Oct 2017 00:00:00 +0000 /2017/10/10/what-s-the-most-popular-car-in-the-uk/ Tue, 10 Oct 2017 00:00:00 +0000 /2017/10/10/what-s-the-most-popular-car-in-the-uk/ <p>The Office of National Statistics is a brilliant source of interesting data sets. <a href="https://www.gov.uk/government/statistics">This page</a> lists recently released datasets. Check regularly and you’ll soon notice these data sets feed into news articles by the Economist, BBC news and more.</p> /2017/08/19/what-is-apache-spark/ Sat, 19 Aug 2017 00:00:00 +0000 /2017/08/19/what-is-apache-spark/ <div id="what-is-spark" class="section level1"> <h1>What is Spark?</h1> <p>Yesterday I completed the Data Camp Introduction to Sparklyr course. This covered an introduction to Spark, as well as sparklyr, a package to interface with spark through R. Before I forget it all, I wanted to write down an explanation of it, some key points and functions.</p> <p>Spark is an open source cluster-computer framework, maintained by the Apache Software Foundation. If like me, that doesn’t help much, then this is a more simplified explanation/motivation for it.</p> <p>Suppose you want to analyse a dataset of your personal taxi (or uber) journeys. Unless you work as a taxi driver (or for uber), you probably won’t have many. You’ve got a bit of expierence with R, so you load it into R Studio and carry on. Simple!</p> <p>But now, let’s say you need to analyse the Uber Journeys for everyone in London. This is a lot more data. R is limited by the size of RAM on your computer. If the size of the data you wish to process is bigger than the size of your RAM, then your out of luck. If it’s even approaching the limit, it will probably be quite slow.</p> <p>What do you do? Well there are many options. An obvious one is to get a faster computer, with more RAM. Spark is an alternative solution. Instead of one powerful computer, it provides a framework to let you create and utilise a “cluster” of many computers to do your data analysis bidding. This is a simplification of all the benefits provided, but it helps to provide a simple motivation.</p> <p>Unfortunately for R users out there, Spark is written in Scala. Until recently to work with it, you’ve needed to use Scala, Java or Python. <code>sparklyr</code> is the R interface to Spark. With it, you can work with Spark with familiar <code>dplyr</code> methods.</p> <p>There are a few differences to understand when using splarklyr to manipulate data in the spark cluster rather than dplyr on your local machine</p> </div> /2017/06/11/how-to-predict-an-election/ Sun, 11 Jun 2017 00:00:00 +0000 /2017/06/11/how-to-predict-an-election/ <p>Whilst the UK is digesting the results of the General Election this weekend, I went down a series of rabbit holes that led to this great article on how the exit polls are used to predict the election results.</p>