Systems

Hello!

This week’s reading seemed to converge on a theme of “thinking and acting in systems”. This is, perhaps, the root challenge of our times.

• Using self-driving cars as an example, Marianne Bellotti writes about safety culture, failure in complex systems, and how they intersect with waves hands AI. This may become my go-to introductory piece on the subject—short enough to read quickly, profound enough to forever change how you think. Key points:
  • Human error is not the actual root cause we think it is: it comes from our desire to see a human accountable; we stop incident review processes when we find someone who’s deviated from what’s expected / documented for them, but this ignores that it’s entirely natural, and often preferable, to deviate from those expectations.
  • Self-driving cars currently use a “human delegates to AI” model: humans supervise AI action, and are expected to take over when needed. In reality, it’s often far too late for a human to take over by the time something’s about to go wrong—but the human at the wheel will still be blamed.
  • A safer approach (if we’re to keep both humans and AI involved) would be “AI delegates to human”: this forces the creator of the AI system to clearly define its limits, to only allow its use where it’s truly safe to do so—in part, because it makes the creator more directly liable for the decisions the system makes. These incentives matter!
• On the topic of AI, Simon Willison’s been sharing his observations as he tries out some of the interactive AI systems that’ve cropped up: image generation and narrative description of code. His writing offers an accessible overview of these systems—and their more profound implications—and introduces central concepts like “prompt engineering”. Most importantly, they provide tangible examples of what it’s like to think in the grammar (language?) and “mindspace” of these novel, relatively opaque systems.
• Mita Williams’s University of Winds from this week (issue 290), delves into systems thinking on a few axes. Williams shares a link to a piece on understanding and defining energy security—this prompted me to dive into Low-tech Magazine, including its novel solar-hosted website and its list of “low-tech solutions” to life’s necessities.
• Paul Wells uses RCMP testimony to the Mass Casualty Commission to surface tensions in communicating in complex systems—namely, how those doing the communicating seek control over that act, versus the messiness of dealing with interlocutors like journalists. The major change here, from the perspective of RCMP communications staff, is the advent of social media, and the direct connection it offers, versus the indirect communication of press conferences and interviews.

Unrelated to systems thinking (at least on the surface), I like the idea of East Coast skeptics and West Coast skeptics: the difference being in how you react to realizing that you can’t know everything.

All the best for the week ahead!

Lucas