Paying technical debt dues

The Texas Tribune didn’t just hire me to play third base on its softball team, although my performance in the absence of the injured photo editor definitely helped keep me in Austin for the past year.

My primary responsibility was to rebuild the government salaries website, an ambitious project to open the compensation information of Texas public workers by sending lots and lots of open records requests to countless state agencies.

Gender and ethnic salary analysis were new features of the rebuilt salary app.
Gender and ethnic salary analysis were new features of the rebuilt salary app.

Former Tribune developers built the original site in 2010 and was received much like other salary sites. It generated so much traffic and criticism from the individuals represented in the data that publisher Evan Smith had to write a blog post.

Why rebuild such a popular app? Technical debt.

The salary site was supposed to be updated when Tribune staff received information from their records requests, but the app fell woefully out of date because importing data was complex and time consuming. Tribune staff paid in the price in daily angry emails from state workers who were upset that anyone could find their outdated salary with a Google search.

Government Salaries v2.0 became my primary task in February. The MVP had three goals:

  1. Reduce technical debt
  2. Reduce angry emails
  3. Provide new analysis

Most of my work on the six-month project focused on importing and storing data. The new app used a rebuilt import process, an open government data schema championed by Travis Swicegood in his ONA talk and Docker for deployment. I looked to metrics, wireframes and user personas for design guidance and tested features as prototypes with the newsroom.

My paper prototypes looked rough.
My paper prototypes looked rough.

I was disappointed that I did not provide users transparency into the development process, a concern that was raised at the Hacking News Leadership conference in Austin. The status of the salary site is of great importance to state workers, and a development blog explaining our progress would have answered many emails and phone calls.

I moved to Los Angeles to be with my partner before the new salary app launched, but Ryan Murphy, Daniel Craigmile and Alex Duner did a great job carrying it to the finish line. When the app went live Thursday, I was both excited for users to finally see new data with gender and ethnic breakdowns and stunned to see months of work published.

Of course the project is not complete. In addition to updating the data, I hope my code and documentation help future developers add features and provide insights into Texas public agencies. I understand the relaunch generated more than 200 emails, and I wish those developers well in responding to them.

Novice skeptic considers race, technology, media

No matter how many Ferguson vines and live streams I watch, I haven’t experienced it. I’m not Mike Brown, I haven’t been tear gassed and armored vehicles aren’t driving my block.

Ferguson resonates with me, and modern technology lets me watch conflicts live from my apartment like never before. However, skepticism makes me question how I can convey and understand the experiences of others as a journalist and a news consumer.

I’ve been on philosophy binge. David Hume’s empiricism spoke to my work at The Texas Tribune, where I combed datasets for newsworthy information. I was already questioning the truthiness of my data, and Hume’s thoughts about knowledge coming from experience sync with the Bayesian statistical methods that were developed around the same time as Hume’s writings.

Recent readings also affirmed my experiences interviewing families with terminal illnesses for Inheritance of Hope. Each month for the past three years, I have written about parents who were told they have terminal diseases, but I still feel like my reports are incomplete. I can’t comprehend illness in the ways those families can.

I connected with Elaine Scarry’s claim that people are better at empathy than sharing actual physical feelings like pain. Physical feelings are tied to specific experiences, and Scarry channels Hume, who distinguishes “impressions” from “ideas” and claims that all ideas and facts are products of experiences.

Likewise, racial tension is rooted in past experiences. If I don’t have those experiences, how can I convey them as a journalist?

When Hume writes, “a man in a fit of anger is actuated in a very different manner from one who only thinks of that emotion,” I think of the Dr. Martin Luther King, Jr. quote, “a riot is the language of the unheard.”

I also recall the powerful response Angela Davis gave to a reporter who asked her about violence: “The person who’s asking that question has absolutely no idea what black people have gone through.” (4:52)

They’re telling me I need experience to understand.

Media gives me windows into other experiences. Black twitter creates discussions like #IfTheyGunnedMeDown, and activists can broadcast sights and sounds nearly instantly. Video and the definitive account it provides, which we learned from Rodney King, inspired petitions to require cameras for police officers after Mike Brown. I love the idea that technology can inform and empower.

Although media reflects these emotions and events as images and sound bites, I can’t take the place of a protester. I can’t actually have the experience with media, and that makes a difference. The history of racial segregation created what President Obama recently called a “gulf of mistrust,” and affects how we perceive events. The divides continue even in our connected lives with social media algorithms that create “filter bubbles” and distort information streams.

At least Antonio French, the alderman who has been at the Ferguson unrest nightly, can post his personal account to his Twitter stream. Without his tweets and videos, I would not have those glimpses from his perspective, even if they are brief glimpses.

However, I wonder what social media is unable to tell me about the black experience in St. Louis.

Note: My dive into skepticism started with a video of young adult writer John Green playing XBox.

Sampling static site generators

I’m coming clean: I’ve been writing on other blogs. It’s not that I don’t love this WordPress site I built as a college freshman with a homepage carousel and fixed-width containers, but some side projects presented opportunities to try generating websites with Jekyll and Yeoman.

I was inspired to start a datablog about about the legendary bearded pitcher Brian Wilson after reading Analyzing Baseball Data with R and wanted a website framework that would allow me to publish easily but also facilitate sharing the R code I used in my analysis.
Jekyll and GitHub fit my situation because I could deploy scripts and blogs to the same place. I suppose I could have used iPython notebooks or Jeremy Singer-Vine’s Reporter to embed my code more seamlessly in posts, but for now I link to the repo.
Originally I thought Jekyll’s use of markdown would block all the Javascript I planned to do with this project, but I’ve ended up embracing static charts and gifs, which are simple to embed in markdown.
I wanted to use the yeoman Backbone generator to get running with a single-page site that used my work examples as data models but into command-line setup struggles. After 15 frustrating minutes, I found my Node needed an update.
My first go with Yeoman wasn’t as snappy as I hoped, and my time allotment broke down like this:
  • updating node: 1 hour
  • fighting backbone models: 1/2 hour
  • writing my resume as json: 2 hours
  • design: 1/2 hour
  • deployment: 1/2 hour
Using Backbone to treat my resume as models meant I had to write my resume and work samples as json. I was willing to put up with the hassle because for some reason I like entering structured data by hand, but this is why lots of people have worked on ways to convert spreadsheets or google docs to structured data and I could have used those tools.
Deploying my yeoman-generated portfolio site was bittersweet as it was only once I put it on the interwebs that I realized the generator had included some libraries that broke Backbone on Internet Explorer. The generated site came with modernizr and had fallbacks to help IE users, but I would have rather handled IE myself. I could pick through the yeoman generator, find what breaks my site on IE and remove it, but I’m lazy.
Give me the grid
You gotta have a responsive grid. Yeoman generators come with one, Sass tools like Bourbon Neat can give you one. However in recent collaborations with my boy Frank Bi, we’ve used good ol’ Bootstrap since we both understand CSS and want avoid grunt/compiling Sass.

Hardware hacking with Arduino’s temperature sensor

Dad always kept it cool in the house, just like his native bay area. With a cold front moving through Texas this weekend, I thought I’d use my Arduino’s temperature sensor to track the overnight temperature of the unheated living room and compare its readings to the actual outdoor temperature.

I modified sparkfun’s temperature sensor example to log every minute, let the sensor run from 11 p.m. to 11 a.m. Friday night and plotted the indoor readings against the hourly outdoor recorded temperature.

Outdoor temperatures at the minute level would provide better comparison, but I was surprised to see how small the difference is.

Twilio was nice to hook my teammates and I up with open-source Arduino boards for best use of their SMS service on voting wait time our project for a hack in New York last summer. Tools like Arduino offer exciting opportunities for folks interested in sensor journalism, but I’ve found it tricky to find newsworthy applications of hardware sensors. Recording the overnight temperature of my apartment was a simple experiment, but I look forward to more.

Fortunately the Texas Tribune nerds let interns tinker in the newsroom.

Design for the story

I’ll never forget the meeting when Washington Post investigative editor Jeff Leen commissioned an interactive graphic for Cheryl Thompson’s series tracking homicide cases through D.C. courts. He enlisted developers, designers, data analysts and an aspiring intern to build an interactive for a story Thompson had already spent a year reporting.

“You can’t publish a story like this without giving readers access to the data,” I remember him saying.

I learned interactives aren’t just visual candy but vital to storytelling, and the decisions I make creating an interactive directly affect how a user receives the story.

I used to say “I work with data, but I’m not really a designer.” Even after taking classes on user experience, I’d concede design decisions to friends with experience in visual media. Digital journalism comprises many disciplines, and I was trying to say my skills skew more toward analysis than presentation, but my summer internship with The Wall Street Journal interactive graphics team changed my attitudes about design and storytelling.

Borderlands

First it was a template. Then a bar chart. But as news from Syria continued, pressure grew to get this story right. Joe Parkinson and Nour Malas had done some incredible conflict journalism for The Wall Street Journal from Syria and Turkey and compiled a series of videos documenting people whose lives were changed by the civil war near the border of those countries. In addition Storyful verified and geolocated YouTube videos from the Turkey-Syria border that provided additional context to the Journal’s reporting.

Many people were involved with the project – building data structures, designing prototypes, editing videos, translating text to Turkish – but the shared patience and care to do justice to Parkinson’s reporting amazed me. The Wall Street Journal is a leading video producer, and in this project videos captured the human story of the civil war. The design had to let the videos shine.

America Shops: U.S. Retail Sales

It started as the typical interactive graphics intern assignment: javascript-ify an old Flash graphic. In addition to the technical challenges of making an interactive graphic update with new data, the design required a presentation to allow both financial data wonks and average readers a tool to explain an important monthly report.

I learned so much from prototyping with Kurt Wilberding, and Phil Izzo and Pat Minczeski contributed their expertise with economic data to each iteration. It was featured on Wonkblog as “Graph of the Day,” so we were at least successful at designing for the nerds!

I’ve been lucky to work incredible reporters and be reminded that what I can offer with technology must enhance journalism.