This was a session at Sol 2012.
Ben: We all want to change the world for the better in various ways, but I think we should seriously consider the possible negative side effects if we were to succeed. The invention of the coal-fired steam engine succeeded in creating a world with lots of things that we take for granted that help us to live good lives, while simultaneously locking us into a path dependence. And, as we realized only later, the path led to the climate change and the potential destruction of both all the benefits provided by the steam engine and some of the quality of life we had before that. Now it's obviously hard to argue that they could have forseeen all of this. In fact we probably couldn't have even forseen half of the things we did with the coal fired steam engine itself and its descendant technologies that it made possible. So there are two extreme ways to react to this situation. One way is to say that 'if you have a great idea for changing the world you should just do it because there is no way to tell what could go wrong but there is at least some benefit.' The other extreme is that we should stop all development of technology and other forms of innovation altogether, and just implement some lifestyle that seems like the best we can come up with in perpetuity. Neither of these seems like a good enough answer.
The Environmental Protection Agency exists for the purpose of retroactively saying, 'We've discovered that these things are harmful, let's stop doing them.' That is at least some sort of middle ground. It's the same kind of attitude that the police have toward crime. You can't stop them from committing the crime in the first place. You can only punish them retroactively and hope that other criminals will learn from the example. In the case of crime you can imagine society could change so that people just don't feel like committing crimes. But in the case of unintended consequences, there is no society that could train people to avoid actions that have unintended consquences. We know what the laws are and how to avoid breaking them, but, by definition, we don't know what unintended consequences will come from our actions.
Brandon: Most stable systems have feedback. Trying to predict by dead reckoning is a fool's errand; don't invest in it. You can do preemptive feedback early on, when thinking about a project and observing issues that have come up with similar projects in the past. You can also "fire bullets before you fire cannonballs": put a small amount of investment in your initial direction, thus allowing you to more easily change direction later if something goes wrong (and you get feedback).
- Ben: So you can also continue having feedback for years and years on some big project, which is kind of like what the EPA does.