Select columns by filtering on column names with Clojure Incanter

Incanter is a pretty amazing library for working with data.  With a table of data, it's easy to select rows of data to work on by filtering on the data in each row.

For example:

(let [data (to-dataset [{:a 1 :b 2} {:a 3 :b 4}])]
        ($where {:a {:$gt 2}} data))

That will select all rows where given a row, its data under column :a is greater than 2.

But what if you want to filter the dataset to get rid of certain columns?  Say you only want column 0.  This can do that:

(let [data (to-dataset [{:a 1 :b 2} {:a 3 :b 4}])]
        ($ :all 0 data))

What if you want columns 0 and 2?  This can do that:

(let [data (to-dataset [{:a 1 :b 2 :c 3} {:a 3 :b 4 :c 5}])]
        ($ :all [0 2] data))

What if you want to select by column name, e.g. only selecting columns with name :a and :b?  This can do that:

(let [data (to-dataset [{:a 1 :b 2 :c 3} {:a 3 :b 4 :c 5}])]
        ($ :all [:a :b] data))

Here's the tricky one.  What if you want to select by column name, but you want to match the column name against a regular expression (say only names containing a vowel)?  The $where method only lets you select rows of data based on a query — it's a row-wise filtering operation.  We want column-wise filtering on column-names.

My first attempt involved taking the dataset, turning it into a clojure map, then filtering on the map's keys, and finally turning it back into an Incanter dataset.  But doing so ran into a problem in Incanter: to-dataset and to-map are not inverse functions of each other.

That is to say, there exists a file of data *filepath* that can be read into Incanter with read-dataset such that this does not work: (to-dataset (to-map (read-dataset *filepath*))).  For example, if the file is a CSV spreadsheet file with missing data in some cells (not nil, not 0, just no data).

Turns out the solution is much easier:

(let [data (to-dataset [{:a 1 :b 2 :c 3} {:a 3 :b 4 :c 5}])
        columns (filter #(re-find #"aeiou" (str %)) (:column-names data))]
        ($ columns data))

The idea is to create a list of column names of the columns you want, then use that list of column names with $ to select them out of the dataset.

Column-wise selection isn't as convenient as selecting row-wise, but this is one way that works.

Clojure Data Analysis Cookbook - a Book Review

Like yogthos, I was recently asked to review Clojure Data Analysis Cookbook.  With Incanter, data analysis has been one of the "selling points" of Clojure as a practical language.  A practical lisp for practical data analysis.

(Edit 2016: a second edition is available!)

The book is very example oriented, basically being a collection of code recipes for accomplishing apparently common tasks for data analysis.  It gives you recipes to go from taking raw data in the form of CSV, JSON, or whatever, to making an Incanter dataset, to doing analysis on those datasets (e.g. clustering the data by using a self-organizing map), to saving, viewing, or charting the resultant data.  Each recipe is accompanied by brief explanations, and cross-references to other related recipes in the book.

Each recipe is more or less self-contained, without much in building on top of previous recipes.  It makes the book more "random access".  It's less a book to read through cover to cover, and more of a handy reference to use by full-text searching for key terms, clicking on the relevant topic in the table of contents, or by looking up terms in the index.  It's definitely a book I'd rather have as a PDF ebook so that I can access it anywhere in the world, and so I can do full-text search in.

Having said that, you can tell whether a book was made to be seriously used as a reference or not by looking at its index.  There are 10 pages of indices, equivalent to about 3.2% of the number of pages previous to the index.  This counts as a book to be seriously used as a reference.

As a reference book, it's great for people who have already a familiarity with Clojure (and better yet, Incanter) in general.  If you don't know Clojure, this book won't teach it to you.  If you don't know Incanter, you can pick it up from this book if you're a fast learner (don't expect a lot of hand holding in learning Incanter though).

Similarly, I'd say you had best be familiar with how to do data analysis as a discipline in itself.  If you don't know whether to do clustering or regression, or whether to use a SOM or K-means, this book won't teach it to you.

Also, as a reference book, it is not comprehensive.  For example, as far as neural networks go, it only includes self-organizing maps.  There are no other kinds mentioned.  If you want another kind of neural network, you best know where to look for another Java or Clojure library.

Even with all those caveats, I'd still say this is a pretty decent book.  Why?  Because if you have some familiarity of Clojure, played around with Incanter for a bit to learn that library, have taken a class or two of data analysis in university, and aren't expecting a lot of hand holding from the book, then this book is a great guide to start you off on the road to doing data analysis with Clojure, Incanter, Weka, OpenCL, Cascalog, etc.

Serializing Records and Incanter Matrix in Clojure with print-dup

For most Clojure data structures, you can get a serialization by doing something like this:

(def x (some-data-structure 1 2 3))
(def x2 (binding [*print-dup* true]
          (prn-str x)))
;; now x2 is a string serialization of x
;; You can save x2 to disk or whatever
;; Then to read it back in:
(def x-clone (read-string x2))

(Note: you really shouldn't be programming in Clojure with a bunch of def'ed vars like that...)

Unfortunately, Clojure records does not support this functionality quite just yet. There is defrecord2 that implements this serialization functionality for records though, described in this discussion in the Clojure group.

I had a second problem though, since I had records storing Incanter matrices, which are Parallel Colt matrices, and they don't print-dup in a way that can be read back in with read (or read-string). So to solve both problems at the same time, we can just implement our own print-dup for the record we create with defrecord (print-dup is a multmethod).

Clojure Incanter Startup Basics

Assuming your incanter folder resides in ~/bin, we'll start up the Swank server that runs the Clojure code, start up emacs, and execute some code. Here's how.

In your terminal, do the following three:
cd ~/bin/incanter
bin/swank
emacs

Now in emacs, do the following two to play with the ants demo:
C-x C-f path/to/ants.clj
M-x slime-connect

Accept default Host: 127.0.0.1 with a return-key press; same for port. Now the REPL is running within emacs. Go back to the buffer with the ants.clj file open and page down to the very bottom.

To compile the file, do C-c C-k.

To interactively execute code, go to the end of the line of code you want to execute from ants.clj, say the line (send-off animator animation) then do C-c C-e

Not hard at all!

(Edit 2010-04-09: Weird! C-c C-e to execute the last sexp didn't work for a while here. Had to kill the file buffer and reopen the file. Now it works! Odd.

Oh I see, using Clojure Mode, a quick M-x clojure-enable-slime-on-existing-buffers solved the problem. Although the keybindings changed somewhat. C-x C-e to get the above behaviour, while C-c C-e lets you type in a quick one-line to execute.)