Data-Driven Product Development

Data Driven Products Now!

Here’s a video of this talk. In this blog post, I will be using some screenshots from the above presentation.

In this presentation deck, former Etsy developer Dan McKinley outlines two very important ideas:

  1. Data-driven strategies of project management
  2. Data-driven tactics of product management

First, the Agile-esque process of iterative development using prototyping and A/B testing at key milestones is an interesting approach. This is difficult to do in both small and large companies due to different reasons. In small companies, the resources required for the discipline of this kind of development is too big. At large companies, a single developer or product manager would not have enough control to apply this process in a realistic way, especially given constraints from other teams, QA, designers, and management.

Screenshot 2015-12-17 00.11.13.png


This is a great ideal to shoot for. But personally, I don’t know how plausible it is. It requires buy-in from everyone involved. McKinley even briefly mentions this when he talks about discussions with his designer, in which he promises to polish up the prototype in the second phase (“Refinement”) of development.

The second great concept in this talk is the tactical use of simple arithmetic and statistics to make decisions about products and features. While the previous idea’s concern is the quality of a particular product in development, this idea pertains to the nitty-gritty of picking which products to develop in the first place.

Screenshot 2015-12-17 00.15.25.png

This back-of-the-envelope calculation, he outlines, has saved him from spending weeks or months in design, development, testing, deployment, and analysis. This idea seems fundamental and possibly obvious to experienced product managers. However, it is tempting to bypass the rigor this level of analysis requires when everyone involved is swept up in the excitement of a cool new feature.

While these two ideas a vital enough in themselves, this presentation gives us this diagram as icing on the cake:

Screenshot 2015-12-17 00.21.54.png

These charts demonstrate the change in priorities that must happen as a company grows. There are two reasons for this necessity:

  1. Risk mitigation – reducing the number of moonshot ideas being implemented
  2. The absolute importance of using data to make product decisions

And, as McKinley mentions at the beginning of the talk, a dangerous pitfall is to mis-categorize these three by forming an opinion and then finding some pieces of data to back it up. The approach should have more scientific rigor: it should start with a dispassionate hypothesis based on prior data, and through the process of prototyping and A/B testing, the hypothesis should either stand and be refined, or easily discarded after a falsification.

Is Software Engineering Really Engineering?

Programmers: Stop Calling Yourselves Engineers

This article raises some valid points about whether or not software engineering really is engineering.

While software does have some overlap with traditional engineering disciplines, I believe that it’s fundamentally different because it usually doesn’t involve physical materials. Software is similar to mathematics in that it’s only one step removed from pure ideas. As Fred Brooks wrote in Mythical Man-Month:

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures…

This allows for practices like Agile development and continuous deployment. Combined with the rise of scripting languages in the past few years (NodeJS and Python powering backends, for example), it has become easier to deploy poorly-engineered software to production. That’s not necessarily Agile’s or scripting languages’ fault. It’s just a consequence of a diminished need for rigor.

Test-driven development, extreme programming, and other approaches reintroduces some of the rigor, but I don’t believe that’s enough. These processes are difficult to faithfully implement in a real-world setting, which waters down their effectiveness.

I believe the key to solving this issue is a change in attitude on the part of software engineers. Even though it’s easy to give into poor quality production software and loose standards for unit testing and integration testing, software engineering teams and their managers need to take a stronger stance on shipping quality products. The emphasis on design in the current Apple-dominated consumer tech world is a great model to emulate. We should be as insistent on good software as we are on good design. In the end, shipping high-quality software is rewarding for both users and engineers.