{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Tom Augspurger Dplyr/Pandas comparison (copy of 2016-01-01)\n", "\n", "### See result there\n", "http://nbviewer.ipython.org/urls/gist.githubusercontent.com/TomAugspurger/6e052140eaa5fdb6e8c0/raw/627b77addb4bcfc39ab6be6d85cb461e956fb3a3/dplyr_pandas.ipynb\n", "\n", "### to reproduce on your WinPython you'll need to get flights.csv in this directory" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook compares [pandas](http://pandas.pydata.org)\n", "and [dplyr](http://cran.r-project.org/web/packages/dplyr/index.html).\n", "The comparison is just on syntax (verbage), not performance. Whether you're an R user looking to switch to pandas (or the other way around), I hope this guide will help ease the transition.\n", "\n", "We'll work through the [introductory dplyr vignette](http://cran.r-project.org/web/packages/dplyr/vignettes/introduction.html) to analyze some flight data.\n", "\n", "I'm working on a better layout to show the two packages side by side.\n", "But for now I'm just putting the ``dplyr`` code in a comment above each python call.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### using R steps to get flights.csv\n", "\n", "un-comment the next cell unless you have installed R and want to get Flights example from the source\n", "\n", "to install R on your Winpython:\n", "[how to install R](installing_R.ipynb)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#%load_ext rpy2.ipython\n", "#%R install.packages(\"nycflights13\", repos='http://cran.us.r-project.org')\n", "#%R library(nycflights13)\n", "#%R write.csv(flights, \"flights.csv\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### using an internet download to get flight.qcsv" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Downloading and unzipg a file, without R method :\n", "# source= http://stackoverflow.com/a/34863053/3140336\n", "import io\n", "from zipfile import ZipFile\n", "import requests\n", "\n", "def get_zip(file_url):\n", " url = requests.get(file_url)\n", " zipfile = ZipFile(io.BytesIO(url.content))\n", " zip_names = zipfile.namelist()\n", " if len(zip_names) == 1:\n", " file_name = zip_names.pop()\n", " extracted_file = zipfile.open(file_name)\n", " return extracted_file\n", "\n", "url=r'https://github.com/winpython/winpython_afterdoc/raw/master/examples/nycflights13_datas/flights.zip'\n", "with io.open(\"flights.csv\", 'wb') as f:\n", " f.write(get_zip(url).read())\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Some prep work to get the data from R and into pandas\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "#%load_ext rpy2.ipython\n", "\n", "import pandas as pd\n", "import seaborn as sns\n", "\n", "pd.set_option(\"display.max_rows\", 5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data: nycflights13" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "flights = pd.read_csv(\"flights.csv\", index_col=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# dim(flights) <--- The R code\n", "flights.shape # <--- The python code" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# head(flights)\n", "flights.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Single table verbs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "``dplyr`` has a small set of nicely defined verbs. I've listed their closest pandas verbs.\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
dplyrpandas
filter() (and slice())query() (and loc[], iloc[])
arrange()sort_values and sort_index()
select() (and rename())__getitem__ (and rename())
distinct()drop_duplicates()
mutate() (and transmute())assign
summarise()None
sample_n() and sample_frac()sample
%>%pipe
\n", "\n", "\n", "Some of the \"missing\" verbs in pandas are because there are other, different ways of achieving the same goal. For example `summarise` is spread across `mean`, `std`, etc. It's closest analog is actually the `.agg` method on a `GroupBy` object, as it reduces a DataFrame to a single row (per group). This isn't quite what `.describe` does.\n", "\n", "I've also included the `pipe` operator from R (`%>%`), the `pipe` method from pandas, even though it isn't quite a verb." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Filter rows with filter(), query()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# filter(flights, month == 1, day == 1)\n", "flights.query(\"month == 1 & day == 1\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see the first big *language* difference between R and python.\n", "Many python programmers will shun the R code as too magical.\n", "How is the programmer supposed to know that `month` and `day` are supposed to represent columns in the DataFrame?\n", "On the other hand, to emulate this *very* convenient feature of R, python has to write the expression as a string, and evaluate the string in the context of the DataFrame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The more verbose version:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# flights[flights$month == 1 & flights$day == 1, ]\n", "flights[(flights.month == 1) & (flights.day == 1)]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# slice(flights, 1:10)\n", "flights.iloc[:9]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Arrange rows with arrange(), sort()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# arrange(flights, year, month, day) \n", "flights.sort_values(['year', 'month', 'day'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# arrange(flights, desc(arr_delay))\n", "flights.sort_values('arr_delay', ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's worth mentioning the other common sorting method for pandas DataFrames, `sort_index`. Pandas puts much more emphasis on indicies, (or row labels) than R.\n", "This is a design decision that has positives and negatives, which we won't go into here. Suffice to say that when you need to sort a `DataFrame` by the index, use `DataFrame.sort_index`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Select columns with select(), []" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# select(flights, year, month, day) \n", "flights[['year', 'month', 'day']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# select(flights, year:day) \n", "flights.loc[:, 'year':'day']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# select(flights, -(year:day)) \n", "\n", "# No direct equivalent here. I would typically use\n", "# flights.drop(cols_to_drop, axis=1)\n", "# or fligths[flights.columns.difference(pd.Index(cols_to_drop))]\n", "# point to dplyr!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# select(flights, tail_num = tailnum)\n", "flights.rename(columns={'tailnum': 'tail_num'})['tail_num']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But like Hadley mentions, not that useful since it only returns the one column. ``dplyr`` and ``pandas`` compare well here." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# rename(flights, tail_num = tailnum)\n", "flights.rename(columns={'tailnum': 'tail_num'})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pandas is more verbose, but the the argument to `columns` can be any mapping. So it's often used with a function to perform a common task, say `df.rename(columns=lambda x: x.replace('-', '_'))` to replace any dashes with underscores. Also, ``rename`` (the pandas version) can be applied to the Index." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One more note on the differences here.\n", "Pandas could easily include a `.select` method.\n", "[`xray`](http://xray.readthedocs.org/en/stable/), a library that builds on top of NumPy and pandas to offer labeled N-dimensional arrays (along with many other things) does [just that](http://xray.readthedocs.org/en/stable/indexing.html#indexing-with-labeled-dimensions).\n", "Pandas chooses the `.loc` and `.iloc` accessors because *any valid selection is also a valid assignment*. This makes it easier to modify the data.\n", "\n", "```python\n", "flights.loc[:, 'year':'day'] = data\n", "```\n", "\n", "where `data` is an object that is, or can be broadcast to, the correct shape." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Extract distinct (unique) rows " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# distinct(select(flights, tailnum))\n", "flights.tailnum.unique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "FYI this returns a numpy array instead of a Series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# distinct(select(flights, origin, dest))\n", "flights[['origin', 'dest']].drop_duplicates()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK, so ``dplyr`` wins there from a consistency point of view. ``unique`` is only defined on Series, not DataFrames." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add new columns with mutate() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We at pandas shamelessly stole this for [v0.16.0](http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0160-enhancements-assign)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# mutate(flights,\n", "# gain = arr_delay - dep_delay,\n", "# speed = distance / air_time * 60)\n", "\n", "flights.assign(gain=flights.arr_delay - flights.dep_delay,\n", " speed=flights.distance / flights.air_time * 60)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# mutate(flights,\n", "# gain = arr_delay - dep_delay,\n", "# gain_per_hour = gain / (air_time / 60)\n", "# )\n", "\n", "(flights.assign(gain=flights.arr_delay - flights.dep_delay)\n", " .assign(gain_per_hour = lambda df: df.gain / (df.air_time / 60)))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first example is pretty much identical (aside from the names, `mutate` vs. `assign`).\n", "\n", "The second example just comes down to language differences. In `R`, it's possible to implement a function like `mutate` where you can refer to `gain` in the line calcuating `gain_per_hour`, even though `gain` hasn't actually been calcuated yet.\n", "\n", "In Python, you can have arbitrary keyword arguments to functions (which we needed for `.assign`), but the order of the argumnets is arbitrary since `dict`s are unsorted and `**kwargs*` is a `dict`. So you can't have something like `df.assign(x=df.a / df.b, y=x **2)`, because you don't know whether `x` or `y` will come first (you'd also get an error saying `x` is undefined.\n", "\n", "To work around that with pandas, you'll need to split up the assigns, and pass in a *callable* to the second assign. The callable looks at itself to find a column named `gain`. Since the line above returns a DataFrame with the `gain` column added, the pipeline goes through just fine." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# transmute(flights,\n", "# gain = arr_delay - dep_delay,\n", "# gain_per_hour = gain / (air_time / 60)\n", "# )\n", "(flights.assign(gain=flights.arr_delay - flights.dep_delay)\n", " .assign(gain_per_hour = lambda df: df.gain / (df.air_time / 60))\n", " [['gain', 'gain_per_hour']])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summarise values with summarise()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# summarise(flights,\n", "# delay = mean(dep_delay, na.rm = TRUE))\n", "flights.dep_delay.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is only roughly equivalent.\n", "`summarise` takes a callable (e.g. `mean`, `sum`) and evaluates that on the DataFrame. In pandas these are spread across `pd.DataFrame.mean`, `pd.DataFrame.sum`. This will come up again when we look at `groupby`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Randomly sample rows with sample_n() and sample_frac()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# sample_n(flights, 10)\n", "flights.sample(n=10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# sample_frac(flights, 0.01)\n", "flights.sample(frac=.01)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Grouped operations " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# planes <- group_by(flights, tailnum)\n", "# delay <- summarise(planes,\n", "# count = n(),\n", "# dist = mean(distance, na.rm = TRUE),\n", "# delay = mean(arr_delay, na.rm = TRUE))\n", "# delay <- filter(delay, count > 20, dist < 2000)\n", "\n", "planes = flights.groupby(\"tailnum\")\n", "delay = (planes.agg({\"year\": \"count\",\n", " \"distance\": \"mean\",\n", " \"arr_delay\": \"mean\"})\n", " .rename(columns={\"distance\": \"dist\",\n", " \"arr_delay\": \"delay\",\n", " \"year\": \"count\"})\n", " .query(\"count > 20 & dist < 2000\"))\n", "delay" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For me, dplyr's ``n()`` looked is a bit starge at first, but it's already growing on me.\n", "\n", "I think pandas is more difficult for this particular example.\n", "There isn't as natural a way to mix column-agnostic aggregations (like ``count``) with column-specific aggregations like the other two. You end up writing could like `.agg{'year': 'count'}` which reads, \"I want the count of `year`\", even though you don't care about `year` specifically. You could just as easily have said `.agg('distance': 'count')`.\n", "Additionally assigning names can't be done as cleanly in pandas; you have to just follow it up with a ``rename`` like before." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We may as well reproduce the graph. It looks like `ggplots` `geom_smooth` is some kind of lowess smoother. We can either us [seaborn](http://stanford.edu/~mwaskom/software/seaborn/):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(12, 6))\n", "\n", "sns.regplot(\"dist\", \"delay\", data=delay, lowess=True, ax=ax,\n", " scatter_kws={'color': 'k', 'alpha': .5, 's': delay['count'] / 10}, ci=90,\n", " line_kws={'linewidth': 3});" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or using statsmodels directly for more control over the lowess, with an extremely lazy\n", "\"confidence interval\"." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import statsmodels.api as sm" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "smooth = sm.nonparametric.lowess(delay.delay, delay.dist, frac=1/8)\n", "ax = delay.plot(kind='scatter', x='dist', y = 'delay', figsize=(12, 6),\n", " color='k', alpha=.5, s=delay['count'] / 10)\n", "ax.plot(smooth[:, 0], smooth[:, 1], linewidth=3);\n", "std = smooth[:, 1].std()\n", "ax.fill_between(smooth[:, 0], smooth[:, 1] - std, smooth[:, 1] + std, alpha=.25);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# destinations <- group_by(flights, dest)\n", "# summarise(destinations,\n", "# planes = n_distinct(tailnum),\n", "# flights = n()\n", "# )\n", "\n", "destinations = flights.groupby('dest')\n", "destinations.agg({\n", " 'tailnum': lambda x: len(x.unique()),\n", " 'year': 'count'\n", " }).rename(columns={'tailnum': 'planes',\n", " 'year': 'flights'})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's a little know feature to `groupby.agg`: it accepts a dict of dicts mapping\n", "columns to `{name: aggfunc}` pairs. Here's the result:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "destinations = flights.groupby('dest')\n", "r = destinations.agg({'tailnum': {'planes': lambda x: len(x.unique())},\n", " 'year': {'flights': 'count'}})\n", "r" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result is a `MultiIndex` in the columns which can be a bit awkard to work with (you can drop a level with `r.columns.droplevel()`). Also the syntax going into the `.agg` may not be the clearest." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similar to how ``dplyr`` provides optimized C++ versions of most of the `summarise` functions, pandas uses [cython](http://cython.org) optimized versions for most of the `agg` methods." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# daily <- group_by(flights, year, month, day)\n", "# (per_day <- summarise(daily, flights = n()))\n", "\n", "daily = flights.groupby(['year', 'month', 'day'])\n", "per_day = daily['distance'].count()\n", "per_day" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# (per_month <- summarise(per_day, flights = sum(flights)))\n", "per_month = per_day.groupby(level=['year', 'month']).sum()\n", "per_month" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# (per_year <- summarise(per_month, flights = sum(flights)))\n", "per_year = per_month.sum()\n", "per_year" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I'm not sure how ``dplyr`` is handling the other columns, like `year`, in the last example. With pandas, it's clear that we're grouping by them since they're included in the groupby. For the last example, we didn't group by anything, so they aren't included in the result." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Chaining" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Any follower of Hadley's [twitter account](https://twitter.com/hadleywickham/) will know how much R users *love* the ``%>%`` (pipe) operator. And for good reason!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# flights %>%\n", "# group_by(year, month, day) %>%\n", "# select(arr_delay, dep_delay) %>%\n", "# summarise(\n", "# arr = mean(arr_delay, na.rm = TRUE),\n", "# dep = mean(dep_delay, na.rm = TRUE)\n", "# ) %>%\n", "# filter(arr > 30 | dep > 30)\n", "(\n", "flights.groupby(['year', 'month', 'day'])\n", " [['arr_delay', 'dep_delay']]\n", " .mean()\n", " .query('arr_delay > 30 | dep_delay > 30')\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A bit of soapboxing here if you'll indulge me.\n", "\n", "The example above is a bit contrived since it only uses methods on `DataFrame`. But what if you have some function to work into your pipeline that pandas hasn't (or won't) implement? In that case you're required to break up your pipeline by assigning your intermediate (probably uninteresting) DataFrame to a temporary variable you don't actually care about.\n", "\n", "`R` doesn't have this problem since the `%>%` operator works with any function that takes (and maybe returns) DataFrames.\n", "The python language doesn't have any notion of right to left function application (other than special cases like `__radd__` and `__rmul__`).\n", "It only allows the usual left to right `function(arguments)`, where you can think of the `()` as the \"call this function\" operator.\n", "\n", "Pandas wanted something like `%>%` and we did it in a farily pythonic way. The `pd.DataFrame.pipe` method takes a function and optionally some arguments, and calls that function with `self` (the DataFrame) as the first argument.\n", "\n", "So\n", "\n", "```R\n", "flights >%> my_function(my_argument=10)\n", "```\n", "\n", "becomes\n", "\n", "```python\n", "flights.pipe(my_function, my_argument=10)\n", "```\n", "\n", "We initially had grander visions for `.pipe`, but the wider python community didn't seem that interested." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Other Data Sources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pandas has tons [IO tools](http://pandas.pydata.org/pandas-docs/version/0.15.0/io.html) to help you get data in and out, including SQL databases via [SQLAlchemy](http://www.sqlalchemy.org)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I think pandas held up pretty well, considering this was a vignette written for dplyr. I found the degree of similarity more interesting than the differences. The most difficult task was renaming of columns within an operation; they had to be followed up with a call to ``rename`` *after* the operation, which isn't that burdensome honestly.\n", "\n", "More and more it looks like we're moving towards future where being a language or package partisan just doesn't make sense. Not when you can load up a [Jupyter](http://jupyter.org) (formerly IPython) notebook to call up a library written in R, and hand those results off to python or Julia or whatever for followup, before going back to R to make a cool [shiny](http://shiny.rstudio.com) web app.\n", "\n", "There will always be a place for your \"utility belt\" package like dplyr or pandas, but it wouldn't hurt to be familiar with both.\n", "\n", "If you want to contribute to pandas, we're always looking for help at https://github.com/pydata/pandas/.\n", "You can get ahold of me directly on [twitter](https://twitter.com/tomaugspurger)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.2" } }, "nbformat": 4, "nbformat_minor": 1 }