Public Affairs Data Journalism

This is the homepage for Stanford COMM 273D, Public Affairs Data Journalism I, as taught in Fall 2014 by Hearst Professional in Residence, Dan Nguyen.

Quick Links

• Tutorials
• Recommended readings list
• Suggested datasets
• Syllabus for COMM 273D Fall 2014
• Github repo for padjo.org

Contact: Check the course syllabus for office hours. You can also email me at my stanford.edu address: (dun).

Core concepts

Our primary goal is to learn how to argue with and against data. To understand the business of our government, including the power it wields over – or yields to – our institutions, then we must understand data, the byproduct of that business, and often, its fuel.

We see data as a means of understanding and, when necessary, critiquing the "data-driven decisions" in public affairs. Our focus is on concepts rather than technology and mathematical problem solving over statistical methods. The core of our work is ultimately the same core of traditional reporting: the initiative to question, the independent research of facts, and the desire to inform the public.

Our three main strategies:

1. To systematically collect and count the data that, for various reasons, has not been officially collected, e.g. Homicide Watch and ProPublica's When Caregivers Harm
2. To efficiently analyze large – but otherwise benign – datasets to find evidence of something significant, e.g. the Sun Sentinel's Above the Law: Speeding Cops, Reuters' Water's Edge, Center for Investigative Reporting's On Shaky Ground
3. To join and compare datasets, e.g. all of the above

Techniques and tools

Knowing how to use the tools – and why they are needed – allows us to efficiently explore and analyze data ourselves, which is key to understanding data concepts in a meaningful way.

I've tried to keep the tools and programs as agnostic as possible:

Plain-text editor

There is no perfect data tool. So get used to importing and exporting data from spreadsheet to database to web application – and back. The common interface between all of these systems is just text.

This is why most of the data we'll encounter will be plaintext, i.e. delimiter-separated values, or, will be content (such as from PDFs) that we'll want to convert to plain text.

A plain-text editor (as opposed to a rich-text editor, such as Microsoft Word) can be the most effective way to explore and clean up a dataset before you attempt to analyze it with a spreadsheet or database.

We will cover a few text-based concepts: using Markdown to write web-and-human-friendly documents. And regular expressions, a kind of Find-and-Replace on steroids.

The best general purpose, cross-platform text editor is Sublime Text, which has a 30-day free trial. For Windows, there is the free Notepad++. For Mac, there is the free TextWrangler

Text-editor skills

• The purpose of delimiters
• Regular expressions

Spreadsheet

Spreadsheets are the most accessible way to explore, analyze, and visualize small to medium-sized (100,000 rows) datasets. They're also handy for just organizing and structuring your notes and thoughts (which can effortlessly be turned into "data" later). Modern spreadsheets typically have a layer of "data types", i.e., "July 4, 1776" is no longer plaintext, but a date, which allows the user to apply data-specific formulas, e.g. =WEEKDAY(some_cell_with_a_date) to get the day of week.

(However, the assumptions made by a spreadsheet can sometimes destroy data, which is why it's necessary to deal with data as just text).

In class, we'll be using Google Spreadsheets for their mostly-intuitive interface and online collaboration features. However, Microsoft Excel and the open-source LibreOffice can also be used.

Spreadsheet skills

• Sorting and filtering
• Data organization
• Pivot tables and aggregation
• Quick visualizations

Database

For most of our data analysis and exploration, a relational database will be overpowered and awkward. However, for the joining of datasets, i.e. two different tables that share the same column (such as an ID number), there is not a more direct way to do so than to express our desire than through Structured Query Language, i.e. SQL.

When you write SQL, you will technically be programming. However, to not get lost in the weeds, we'll only be covering a basic set of SQL commands, and will frequently be exporting data from a database into a spreadsheet.

We'll be accessing databases through graphical user interfaces. For all platforms, you can install the Firefox web browser and then, the SQLite Manager plugin. On Mac, the best free database GUI is Sequel Pro

Database skills

• Inner and outer joins
• Foreign key relationships

Optional tools:

• OpenRefine (Macs and PCs) - OpenRefine looks like a spreadsheet but has much narrower use case: the cleaning of data. But it's the best at what it does, and the difference between being average and good at data cleaning can easily make or break an entire project.
• Mou (for Macs) - A Markdown editor that lets you preview your work.
• Github desktop client - For when you tire of using the web interface.
• Google Fusion Tables - a hybrid of online database and visualization tool. Particularly useful for simple mapping of large datasets.
• ModeAnalytics - A kind-of "Github for MySQL", allowing collaboration and sharing of datasets and queries via the web browser.

Related coursework at Stanford

COMM 273D is the prerequisite for COMM 274D: Public Affairs Data Journalism II, taught in the winter by Cheryl Phillips.

I am also teaching the following two classes (open to undergraduates outside of the journalism program):

• COMM 113/213: Computational Methods in the Civic Sphere in the winter quarter
• COMM 177A/277A: Computational Journalism in the spring quarter.

How I see the relationship between the 3 courses I'm teaching:

• Public Affairs Data Journalism I covers the work and process of journalism, as well as the tools and concepts needed to efficiently understand data – data, that for the most part, I will have extracted and cleaned for class use.
• Computational Methods covers the problem-solving skills needed to collect and explore data for public accountability stories, as well as additional techniques for visualizing and publishing data journalism.
• Computational Journalism focuses on using computational methods and problem-solving to produce a project with civic impact, whether it be a web application or via other mediums.