Category Archives: dataviz

Creating static data visualizations with D3.js and Node.js

D3.js is known for its great data viz capabilities for the web, including dynamic and interactive capability. But D3.js can also be used to generate static data visualizations via the command line, including CSS styling and/or stylesheets.

For example, if you want a command line solution for automated visualization creation, if you want to output an SVG into your make workflow, or if you need to prepare a visualization for print (for a research paper, journal article, a newspaper or magazine), a plain old non-interactive SVG can be very useful. The output we’re aiming for is an SVG you can load into Inkscape, Illustrator, or one that we can convert to an EPS, a PDF, render to a PNG or any other image file format you can think of.

Alternatives before diving in

Of course if you’ve already created your visualization on the web, you can directly save the SVG via copying the source code and CSS, or using a handy bookmarklet like the NYTimes SVG-crowbar. So the below is a bit overkill for a case where you just want to download your already-web-based data visualization.

Using Node.js to execute our JavaScript code

Let’s jump in. Luckily D3.js and Node.js make it possible to output an SVG from a JS source file, your data, and the command line. Node.js internally uses the Google V8 JS engine to execute JS code, and is often used as a server or for networking applications. Using Node.js to output SVG is a bit of a hack vs. creating D3 visualizations for the web. Regardless, we’ll be doing so due to the advantages mentioned above — automated generation of static SVGs is our goal. As a working example, I’ll be altering Mike Bostock’s area choropleth example to be used via Node.js to output the resulting SVG.

Tools needed (any platform, but Windows is a bit tougher):

  • Node.js and npm (the node package manager)
  • D3.js for Node (install via: “npm install d3”). Note that installing D3.js on Windows via npm can be a bit tough, you may or may not run into some dependency roadblocks that will require some googling, not recommended for the faint of heart
  • Terminal/Command Prompt depending on your platform (OSX/Linux/Windows)

First things first, for your Node.js script, you’ll need to require/import D3. This is similar to including d3.v3.min.js at the top of your HTML file.

d3 = require("d3");

To load json from our local filesystem, we’ll need the fs package, since d3.json won’t work as we’re not loading json via a HTTP request (see further down).

var fs = require("fs")

Next, we’ll need to include the client-side Topojson, Topojson.js. Download the latest Topojson.js here. Often Topojson is used via Node.js to simpify Geojson data (Geojson to Topojson for a smaller file size) via the server side API. However, it also can be used to convert the Topojson back to Geojson, via the client side API when rendering a map. Because the geo data we’re using here is Topojson, we’ll need the client side Topojson API. More information on the Topojson APIs is available here.

var vm = require('vm');

var includeInThisContext = function(path) {
    var code = fs.readFileSync(path);
    vm.runInThisContext(code, path);
}.bind(this);
includeInThisContext("topojson.js");

VM is a Node.js virtual machine. It imports the topojson.js in the context of this JS file, so that we can use the topojson client side API methods. Credit goes to this stackoverflow discussion here on including external .js files in Node.js.

Next, almost all the rest of the JS code is the same. However, one difference is that d3.json will need to be replaced by the node JSON parser. d3.json loads JSON via a HTTP request, however, when we’re running via the command line won’t be necessary (or possible, in our case).

var us = JSON.parse(fs.readFileSync("us.json", 'utf8'));

To include the CSS styling, we need to include the CSS within defs tags, within the SVG block.  In addition, for compatibility with Adobe Illustrator, or if our CSS includes > or < characters, we need to enclose the CSS within a CDATA block. This ensures that when we output the SVG, the CSS is attached, but not accidentally parsed.

var css_text = "

Because JavaScript isn’t a fan of multi-line strings,  we need to backslash (escape) the end of each line. Multiline strings in JS need escape characters at the end of each line, since there is no native support for multiline strings. Note that there are alternatives to escaping the end of each line, so feel free to explore other options if you have a larger style sheet.

Last but not least, we will output the svg to stdout, which we can write to an svg file.

console.log(d3.select('body').html());

This selects the html inside the body tag, which in our case is just the SVG block. From there, we can run our JS code (I saved the source file as “area_choropleth.js”) from the command line via:

node area_choropleth.js > area_choropleth.svg

which will direct the SVG text output from stdout into the file “area_choropleth.svg”. And we’re done!

All together now

Here’s the resulting SVG, rendered as a PNG:

area_choropleth

Here is the code in full below, which is also hosted with the us.json data from Mike Bostock as a gist on github:

//require modules
var d3 = require("d3");
var fs = require("fs");
var vm = require('vm');

//import topojson.js client side API
var includeInThisContext = function(path) {
 var code = fs.readFileSync(path);
 vm.runInThisContext(code, path);
}.bind(this);
includeInThisContext("topojson.js");

//SVG dimensions
var width = 960,
 height = 500;

//scale for county-population-based fill
var fill = d3.scale.log()
 .domain([10, 500])
 .range(["brown", "steelblue"]);

var path = d3.geo.path();

var svg = d3.select("body").append("svg")
 .attr('xmlns', 'http://www.w3.org/2000/svg')
 .attr("width", width)
 .attr("height", height);

//parse the US county population data JSON file
var us = JSON.parse(fs.readFileSync("us.json", 'utf8'));

//enter the counties
svg.append("g")
 .attr("class", "counties")
 .selectAll("path")
 .data(topojson.feature(us, us.objects.counties).features)
 .enter().append("path")
 .attr("d", path)
 .style("fill", function(d) { return fill(path.area(d)); });

//outline the states
svg.append("path")
 .datum(topojson.mesh(us, us.objects.states, function(a, b) { return a.id !== b.id; }))
 .attr("class", "states")
 .attr("d", path);

//add css stylesheet
var svg_style = svg.append("defs")
 .append('style')
 .attr('type','text/css');

//text of the CSS stylesheet below -- note the multi-line JS requires 
//escape characters "\" at the end of each line
var css_text = "

Summary

  • D3.js via node.js on the command line is certainly possible, although needs some extra massaging to ensure we have stylesheets embedded within the SVG, and that JSON/CSV/TSV data loads without issue.
  • CDATA blocks need to enclose the CSS stylesheet for proper rendering within Illustrator, and multiline CSS style sheets loaded as JS strings need end of lines to be escaped with “\”
  • For Topojson.js, the client side API source needs to be included separately via vm (or the Topojson.js methods need to be embeded directly in our JS code)
  • d3.json, d3.csv, d3.tsv etc… need to be converted to a local file reader via fs, since the former require HTTP requests to retrieve data

Questions, comments, feedback, have a better way?

Post a comment below, or message me on twitter @pykerl. Also thanks to Eric Dodge for some helpful suggestions.