What could possibly go wrong?

Failure modes in software development

Tadej Štajner, TVBeat / @tadejtadej / tadej@tvbeat.com

#vblatu

About me

Developer at TVBeat
PhD student at Jožef Stefan International Postgraduate School.
Solving hard problems with data

Why this topic?

What happens to things that you are working on?

First I'll do a mental exercise. What were the results of my work in the last few years, from my perspective. It doesn't matter when it happened, nor where, nor if it was your fault. Here's mine. Even though this chart show miserable failure, all contracts were completed, deliverables done, most even on time. What this shows is how I perceived the result to look like. This talk will be about thinking about how to make this pie have strawberries and more blueberries when I bake it again 10 years from now. We'll, to do this exercise, we first need to do a short breakdown.

Politically sacked. No advice for this. If you feel that's your thing to solve, stop studying computer science and go into management consulting. It's like game of thrones, but with more cruelty and less metal weapons.
Misguided target. You can see yourself going very far very quickly running in circles. This is a product management problem in the end, which you won't learn about in CS, but small PM decision have huge impact.
Failure to learn. There's a huge class of problems that stem from the fact that small mistakes repeat often enough, and no one bothers to prevent them.
Failure to operate: when your program works great in powerpoint, production is often less rosy.

Failure to learn

Rapid feedback

So, you're setting out to generate knowledge. Good for you. While movies show knowledge being generated by scientist when stuff explodes or colourful gases start leaking, the reality that it's a much more boring process. While it may look like a hamster wheel at the first glance, it's about reducing uncertainty. You start with a complex system that you need to understand. For climate scientists, this is a planet. For sociologists, these are societies. For social media marketers, these are teenage girls on instagram. And if any one of these wants to do their job, they have to understand that system. So, science is the process of figuring out that system, one piece at a time, until it becomes an elegant model that can predict things. While the narrative always hinges on one eureka moment, it's the rapid feedback that's actually doing the work. A common stereotype of science is that from the outside, things are moving nowhere, and suddenly, eureka. That's not really how things work. Complex products are build by building step by step. Complex scientific problems are solved by reducing the unknown step by step. How does this look like in code? The first feature that you get in most software projects that are used in science is an REPL. The ability to iterate quickly AND have high performance code is what gives you that. And it's more important than you thing. When I was doing my code in C++, got a 20-50% perf boost compared to python numeric packages. But with compile times well above a minute even after tricks, I typically forgot what the build was meant to test for. I call this the coffee run rule. If your iteration takes longer than it takes you to walk down the hallway to make coffee, you're going to be slow and high on caffeine. http://www.azarask.in/blog/post/the-wrong-problem/

http://www.azarask.in/blog/post/the-wrong-problem/

Prototypes

Own your core

part of your system that you spend the most time on

Measurement

We're not here to just learn.

Summing up - learning

Quick iterations
Prototypes
Own critical parts
Measure

Failures in operation

Sytems behaviour

how does your system respond to load?

Sytems behaviour

is there a cliff?

Keep the system in the happy zone
Make the system stable by design

Things will happen

hardware failure is a part of normal operation

distributed computational systems
distributed databases

Babysitting

Summing up - systems

Distributed systems
The babysitting trap

Failure to communicate

Abstractions

if you use this interface, an implementation can solve your problem

One magic button

Leaky abstractions

Leaking abstraction is when a system presents you with one interface, but you actually spend your time at a whole different level of abstraction. For instance, Apache Storm, an otherwise excellent component, lures you in with its promise of a very elegant model of Spouts and Bolts. Porting your code to that abstraction is relatively easy. However, operationally, you then spend time tuning the infrastructure so that it's always in its optimum operational envelope. You quickly learn that when data travels from one component to another, it traverses four buffer queues, which flush only when full. In short, this means tweaking buffer sizes to your throughput, and if your stream varies a lot, you need sophisticated dynamic balancing. In short, we were sold an elegant interface, and got a nonlinear optimization problem with tens of variables.

Leaky abstraction

Lying abstraction

feature nominally exist, but with ridiculous performance envelope

Summing up - communication & abstraction

Fake simplicity
Leaky abstractions
Lying abstractions

Failure to observe

what axis is your problem hard at?

Performance

Scale

Complexity

Reliability

Politics

Summing up

Failures to learn
Failures to operate
Failures to abstract
Failures to observe

Thanks

Tadej Štajner, TVBeat / @tadejtadej / tadej@tvbeat.com

http://tdj.si/failuremodes/ (press S for notes)