Searls of Wisdom for August 2024

By reading this you have demonstrated two things: you have at least a passing interest in hearing from me and you know how to subscribe to a thing. That being the case, you might also enjoy subscribing to my podcast, Breaking Change. Right around 20 episodes, and it's settling into a biweekly format that reminds me of the drive time radio talk shows I listened to as a kid to pass the time. If you're interested in the kinds of things that surround me—independent software development, theme parks, Japan, tech news, games and movies, and hypercritical observations about everyday life—you might like it. You also definitely might not like it. No way to know until you try.

The app I've been building to support Becky's business is more or less done, but every time I say that, I find a dozen more things to tweak. We're entering "stealth mode" in a few days, which I'm pretty sure I'm not supposed to tell you but I also never promised to be good at product marketing. I'll be sharing my 2¢ about the app—what it does, how I built it, and why we designed it the way we did—at my Rails World talk at the end of September.

Or at least, that's the plan. I have yet to start work on the talk, as the slowly-twisting knot growing in my stomach is fond of reminding me. Anyway, that's why this month's missive will hopefully end up being a tad shorter than usual. (Update: it didn't.)

Now that the dust has settled a bit on both the Covid-era remote work phenomenon as well as the predictable return-to-office backlash, I'd like to write about something as a guy who co-founded a fully remote company in 2011: the things that were lost by going remote.

When I look back on my career in the Before Remote Era, it's hardly the case that being in person with others was generally better. Most of my work experiences were a miserable slog. Few of my colleagues made meaningful contributions. Most projects required me to travel—if not every week, then certainly more often than the work itself warranted it. I mean, there's a reason I found the idea of starting a remote-first consultancy to be compelling, and why passing up every opportunity we had to acquire office space (even for the low price of literally free) was a relatively easy decision.

To wit, if you're planning for the median software development experience, working remotely is absolutely superior to haphazardly throwing people in a room with the expectation they won't mind sacrificing two hours a day to commuting and losing some measure of personal autonomy. Remote experiences can be pretty damn bad, don't get me wrong, but as someone who's had to work in offices riddled with black mold, where executives occasionally pulled me aside to scream (scream scream) at me about missing deadlines, and on locked-down networks without access to handy resources like ✨The Internet✨, it's hard to imagine remote work ever being worse than that.

Given that most of my on-site work experiences ranged from miserable to middling, back when we were talking about starting our own agency, my attention was mostly spent on avoiding that misery—for my own sake as well as for others'. I was so motivated to prevent those terrible experiences that I barely listened when people (including prospective clients) told me we were certifiably nuts for fantasizing that anyone would ever trust an agency that neither had its own office nor worked out of theirs.

So if the goal was to avoid catastrophically caustic workplace trauma, there is no longer any doubt: remote has been a resounding success.

But what if the goal is to have exceptional, career-altering, life-changing work experiences? The truth is, I've had more than my fair share of these moments but I can't think of a single one that happened remotely.

I could demur and suggest that's because my most formative years happened in offices, but I'm pretty sure that's not it. When I reflect on the moments that changed the trajectory of my career or helped me advance to the next level of excellence in my craft specifically since first going remote in 2010, a half dozen in-person experiences instantly jump to mind. Zero remote ones do.

Put differently, it seems like remote offers a higher floor and lower ceiling on the impact of one's work experience. At the negative end, remote work can't reach the low lows of being physically torturous—merely psychologically torturous. And at the positive end, remote work can be tremendously productive, educational, and rewarding… but it never hits the high highs of being mind-blowingly incredible.

Granted, I realize it's rare for anyone, in any context, to have a "mind-blowingly incredible" experience at work, but yours truly has somehow been blessed with a number of them. None of them happened on Zoom. None were communicated via text. All of them were face-to-face.

(Besides gesturing at privilege, those great experiences weren't entirely the result of random chance. I started my career by intentionally trading away a lot of salary in exchange for the rapid on-the-job training offered by an up or out consultancy. It's a path I'd still recommend to anyone willing to put up with long hours, incessant travel, and low pay for a few years.)

Anyway, all of this means nothing without a few examples, so here are a few specific memories I'm thinking about as I write this.

The Course Reader project

The most formative agile software project I was ever a part of was for Gale (now Cengage), and it could have been a case study in a book like Extreme Programming Explained by the time it was over. Specifically, we were tasked with building "Course Reader", a digital product that would replace academic course readers—a collection of essays and excerpts typically selected by a professor to be included as mandatory reading for a given class and laboriously copyright-cleared, printed, and bound by campus bookstore staff.

The project was a resounding success. And many of my experiences over those six months really were mind-blowingly incredible. Everyone on the team that I'm still in touch with frequently cites specific experiences and insights from that project as being fundamental to how they think about software today.

I drove from Columbus to Farmington Hills, Michigan every Monday at 4 AM, worked four ten-hour days, and came home every Thursday around 10 PM. I listened to the entire History of Rome podcast as I drove my 2000 Ford Taurus into the ground.

The vibes were very agile. We weren't "doing Scrum". Or "extreme programming". Or following any other prescriptive methodology. We were a self-organizing team who shared the same set of values about what made software great or awful. We reached consensus on how best to extend those values into the core principles that would govern how we would build this product together. Each team member brought with them whatever tools and practices they'd seen succeed elsewhere. In turn, we each gave one another wide latitude to explore better ways of doing things. If someone had an idea for improvement, the correct response was, "okay, let's give it a try!"

None of the code we wrote was perfect. Many of our experiments failed. And we delivered features at a leisurely pace that the business routinely balked at.

But the code we did ship achieved 100% test coverage, had numerous eyes on it at every stage, and not a single bug ever reached production. And some of those experiments yielded tools and practices that would go on to be used by teams around the world. And while we didn't come close to delivering the sheer quantity of features the business initially asked for, the app did what it said on the tin by offering a simple, pleasant UX for professors and students that was years ahead of its time as a web application.

Below are a few of the things we did on that project that I struggle to imagine being replicated remotely.

Always be pairing

If you're not familiar, "pair programming" is a practice in which two people work at a single computer and write code together. When done thoughtfully, both parties balance time spent as the "driver", who does the bulk of the typing, and the "navigator", who does the bulk of the worrying about unforeseen edge cases. Cute practices like "ping-pong" pairing involve Programmer A writing a failing test, then Programmer B writing a passing implementation and subsequent failing test, before handing it back to Programmer A. We also practiced "promiscuous pairing", which is less sexy than it sounds and just meant we swapped partners after finishing each feature. Taken together, these practices can foster a collective sense of code ownership, socialize institutional knowledge, and eliminate the need for (boring, asynchronous, too-late-to-be-useful) code review processes.

I'm happy to report that all of this can be achieved on remote teams as well as colocated ones!

I'm less happy to report that pair programming as described above basically never happens, whether in person or not.

Most "pair programming" in the wild could be less charitably described as either "cooperative troubleshooting" or "one person codes while the other checks their phone."

But pair programming on the Course Reader project was an absolute dream, and it truly delivered on the promise of the practice. My proficiency as a programmer grew by leaps and bounds. I became far more comfortable with the UNIX toolchain. I strengthened my grasp of important computing concepts that never quite clicked previously. And I'm pretty sure I taught my colleagues a useful thing or two, as well.

What made us so successful when so many others fail to see any value from pairing? Two reasons:

1. From the outset, the team reached consensus on a shared set of inviolable principles. One of them was that every single line of production code would be written by a pair
2. We held one another accountable, not only by seating two people to a computer but actively verifying everyone was pairing properly, even when it was emotionally taxing

A consequence of this policy meant that when I got in at 7:30 AM on Monday before anyone else, I wasn't allowed to knock out a feature. I couldn't even fix a bug. If I wanted to reach my billable hourly quota, I had to find other ways to make myself useful. I'd do little research projects the team had prioritized. I'd shore up messy tests. I'd pore over static analysis results to identify issues. You know, the kind of obviously-valuable shit that teams never find time for, because there's usually an expectation that shipping production code trumps all other activities all of the time.

Accountability is a trigger word for a lot of people, and it's no surprise why. Usually, the word is only ever used by a pointy-haired boss in the direction of a subordinate for failing to meet a commitment they never wholeheartedly agreed to. Just a fancy word to describe telling people what to do. But accountability on Course Reader was perfectly natural by comparison, because we'd designed all our rules ourselves as peers. When we started the project, we hammered out what set of practices and commitments we were willing to make to each other. I'm sure I wasn't alone in believing the "always pair program" mandate was going to produce the best results while simultaneously feeling terrified that I'd be found out as a Bad Programmer and exhausted at the end of every day. But we agreed to it anyway.

We typed up the rules as a team charter, printed them on massive plot-printer paper, signed it, and hung it on the wall.

Could we have achieved the same thing in a fully-distributed team by pairing over the Internet? Yes, absolutely. Would we have, though? Or when the going got tough, would the navigator get bored and tab over to Reddit or Twitter, and lose track of what the driver was doing? And when we fell behind schedule, would my pair and I secretly agree to try to divide-and-conquer in order to move faster (as every other pair secretly agreed to do the same thing)? I know myself pretty well, so I'm confident I needed the accountability of other team members in the room to call out my lyin' ass.

Apart from accountability, I'm not sure that a 100% pairing mandate would be feasible online, or even humane. I don't know if you've tried both, but I find that I can easily pair in person with someone for eight hours (including problem-solving discussions over lunch), but I struggle to pair remotely for more than a few hours a day. The latency. The lack of nonverbal cues. The opportunity for distraction. My inability to suppress the knowledge that I could get more done and be more comfortable if I didn't have to do everything over a long-distance phone call.

But maybe that's just me, he typed while smirking because it's obviously not just him.

Agile card walls

These days, I'm sure the vast majority of people who've used an "agile card wall" for the purpose of prioritizing work and communicating status have only done so with a mediocre web app featuring awkward horizontal scrolling and glitchy drag-and-drop of "cards." Not a physical wall with paper index cards and sharpies that always ran dry too fucking fast.

The difference between the two experiences is night and day. There really is no comparison. The beauty of a physical card wall is in its liberating constraints: you can only fit so much text on a single card, you can only fit so many cards on a wall, and you can only fit so big a wall in your head at once.

Meanwhile, software card walls can give you a similar aesthetic, but none of those constraints. Write 5000 words if you want. Attach dozens of PDFs of flow charts. Link out to hundreds of screens of click-through Figma prototypes. Carry on a 200-reply conversation via asynchronous issue comments. Software can fit it all.

The Course Reader team correctly embraced our physical card wall's constraints as being, more or less, the whole fucking point. We had all used JIRA. We knew that the infiniteness of software issue trackers gradually gave way to the temptation to let documentation, status-tracking, and communication about the work wrest primacy away from the work itself. We weren't going to let that happen to us.

The front of each card got a one-line headline and at most a sentence or two about what the feature should do and why.

The back of each card featured bullet points of 2-5 (but almost always exactly 3) acceptance criteria. If someone implemented the feature and it exhibited all the acceptance criteria, it would be accepted as complete so long as it met our general quality commitments.

The product owner was a subject matter expert in the publishing industry, but she was new to this circus, and so we taught her how to communicate her needs in a way that we'd understand them.

A couple questions in particular arose more than once:

"What if a feature won't fit on a single card?"

Eventually, we'd all reflexively—and sometimes in unison—reply, "guess you need a smaller index card!" Because that problem, almost always, was a sign not that the card was too small, but that the idea was too underdeveloped, needing more time in the oven until it could be expressed succinctly.

"What if the card doesn't have enough detail for me to start work?"

That wasn't a problem with the card, either. Whoever asked would inevitably receive the retort that, "a card is just a promise to have a conversation." A paper symbol of a commitment to deliver something. A touchstone we could unpin from the wall and use to jog the product owner's memory before planning out an approach. An inspection checklist to guide whoever performed quality assurance on a feature as we evaluated it against the acceptance criteria.

Of course, minimalist software tools that emulate what's great about a card wall can theoretically (and for a moment did actually) exist, constraints and all. But the problem always was that they need to be sold to businesses, not individual enlightened practitioners. And enterprise customers will never accept arbitrary limitations on how much shit they are allowed to cram into a 5-pound bag. Every opinionated issue tracker company eventually sells out its own product and gradually removes their thoughtful design constraints until it resembles a somehow-even-worse version of JIRA before the whole thing slides into irrelevance.

Everyone felt a stake in what was placed on the card wall, because it would determine what we would spend our time building. For instance, our business analyst and product owner routinely groomed the backlog throughout the week—reviewing and adjusting its linear priority. And my ears would perk up whenever their conversation signaled a potential technical blocker or workflow dependency. It would have been totally normal for me to stand up, inspect the wall myself, and chime in with, "hey Mara, just a heads up that if we wait to play this card until after Kevin and Damon have shipped the user highlighting feature, we'll be able to reuse their persistence work and it'll go much faster."

Conversations like those can theoretically happen over Slack, but they virtually never do. Slack's tagline is "where work happens", but for that interaction to have been possible, every single errant comment made by any team member (regardless of role) would have to be logged into the chat for me to have even noticed it. And, of course, if the chat was so lousy with messages as to contain every single inane comment, reading it all would mean getting nothing else done. The human ear and brain are very good at filtering important information, it turns out!

So yeah, it'd be neat if physical card walls made a comeback as an alternative to the everything buckets of online issue trackers.

The team room

The card wall was perhaps the team's primary "information radiator", but there were several others—like a retired traffic signal someone had wired up with X10 automation to turn red whenever the build failed—carefully arranged throughout room.

The room! I haven't even talked about the room yet!

The Course Reader team was one of a dozen or so that occupied an entire floor of a large (by Farmington Hills' standards) office building, recently liberated from cubicle walls. Now, if you followed much Office Layout Discourse throughout the 2010s, most people would look at our team room, call it an "open office plan", and then deride it as such. I freely acknowledge that in the vast, vast majority of cases, businesses who kicked people out of private offices and knocked down cubicle walls were doing so to save money and with a paper-thin justification like "something-something COLLABORATION!" as if productivity would magically and spontaneously manifest before their very eyes. It's not just neurodivergent people, most people period struggle to do their best work in loud, distraction-laden environments. I hear you.

This was different.

When the team was formed, the first week was spent sharing how we each like to build software, learning about the market opportunity this app presented, and reaching consensus on how to best bridge the two. The second week was spent fetching and arranging desks, chairs, workstation computers, keyboards, and mice. We even took some of the recently-retired rollable cubicle segments and used them to build a wall between us and the outside world.

Our cross-functional team of 8-12 people wasn't tearing down walls and leaving everyone exposed to fend for themselves. We were designing (and would continuously redesign) our own purpose-built space to improve our performance as a team. We set up a large table in the middle for the folks in primarily people-facing roles. A second, smaller table was off to the side, as if to passively signal "leave me the fuck alone." Opposite that was an arc of 5 or 6 pairing stations, each facing outward toward the (enviable) corner windows. We called them "pairing stations", because we had the foresight to do a good amount of storming and norming over how the team was going to operate before laying out the room. As a result, for each computer we pulled two chairs, two keyboards, and two mice.

(Fun fact: to that point, I had never even considered connecting multiple keyboards and mice to a single computer—and of course, it took some futzing to configure in desktop Linux. But it didn't take a physics degree to work out why this pairing arrangement was way more pleasant than the fastest network-based pairing service in the world possibly could be!)

You'll have to take my word for it, but this work environment didn't feel distracting. Had we each been coding individually and deep in thought, then it would have been maddening, sure. But when you're forced to think aloud in conversation while pair-programming all day, other people carrying on their own conversations manages not to pull you out of it, and so isn't nearly as disruptive. Believe me, I'm usually the first guy to pop in earbuds at the first sign that a human might talk to me, but I found the environment creative and invigorating.

Looking back, my fondest memories were during the first 6 weeks of the project.

Admittedly, it was fits and starts. We were slow to get traction. In fact, we got to our first demo at the end of our first two-week iteration with literally nothing to show for it. We shipped zero features. Worried that we were losing the confidence of the business, our team's delivery lead didn't react—as so many would have—by pushing out deadline expectations. No. Instead, Mark convinced us to switch from a two-week cadence to a weekly one. Instead of failing to deliver every other week, we'd let down our colleagues every single Thursday. And he was right: if a team isn't getting traction, they need faster feedback loops, not slower ones. In this case, we needed twice as many opportunities to make course corrections.

So if the team was struggling so badly, why did I just say I have so many fond memories of that time? Because, despite failing to ship any code, we were getting a metric shitload done. What do I mean? I mean every time any pair found themselves doing something for the first time—whether adopting a new dependency or encountering a novel problem—we'd raise a hand, swivel our chairs around (so that we were all facing inward towards the big table as opposed to the window), and hash it out.

Examples:

• "Should we use Hibernate, another ORM, or hand-rolled SQL queries?"
• "How do we feel about trying Mockito for mocking, even though the rest of the org uses EasyMock?"
• "The way Justin formats switch statements makes me want to stab him—does anyone else want to stab Justin?"

No decision too big. No decision too small.

How did quibbling over style and approach amount to getting a ton of work done? Because it pulled forward every potential point of incongruity before it had a chance to fester throughout the codebase. A month in, if you opened any Java or JavaScript file listing at random, you'd never be able to guess which teammate authored it. Consistent, predictable code is easier to work with and reduces the surface area for bugs and maintenance nightmares. As a result, once we normalized, our throughput skyrocketed past the teams around us.

Oh yeah, information radiators. This was a term of art for anything we could display to communicate project status. Some, like the card wall or our burndown chart hung on the interior of the team room's walls, because they were our concern. Private variables. We knew what they meant and sharing them broadly could lead to misunderstandings by outsiders. Other charts could be reasonably understood by people outside the team—like our Sonar code quality score and test coverage report—so we hung them on the wall facing the hallway. Public API. This led to occasional comments of encouragement and even unsolicited offers to help ("hey, I noticed your Sonar report has a lot of instances of 'package tangling', I can show you how to quickly address those").

If the picture I'm painting sounds a little idyllic, that's because it was. I don't get up at four in the fucking morning to drive three-and-a-half hours to metro Detroit for just anyone. I could tell in real time that my experience on this project was changing my life. I relished every moment, even the excruciatingly painful ones.

When it comes to whether this team room could be replicated remotely or not, in a literal sense of course not. But more loosely, yeah, a lot of these same outcomes are technically possible across distance. But while nothing is stopping teams from normalizing to this extent, reaching the level of consensus-building we did on Course Reader would be hard to achieve with online tools due to their lower information density and higher friction to participation. There's no SaaS product that can offer an equivalent 30-second way to circle the wagons, debate and decide an ad hoc decision, and then swivel right back to work.

Cool history lesson. Now what?

Apart from indulging in a bit of nostalgia as I recounted the above, it was also in service of a larger point: colocated and remote experiences are fundamentally different and those differences should influence our decision-making. Thirteen years ago, this would not have been a controversial statement. But the post-2020 conflicts over topics like quiet quitting and return to office directives has polarized the discussion as though one mode of working is categorically better than the other.

Here's some drive-by advice on navigating those differences, informed by some of the experiences described above:

• In-person experiences are generally superior for smoothing out differences between humans: high uncertainty about what needs to be done or how, coworkers that don't get along, programmers of different skill levels, departments (like product and engineering) that are out of alignment.
• It is extremely difficult to level up novice programmers remotely—to the extent I caution anyone from hiring "juniors" for remote positions that haven't demonstrated the intrinsic drive to rapidly teach themselves (i.e. people who will inevitably become highly-skilled, regardless of where they're working)
• Remote work provides safety when the alternative is a psychologically volatile environment. I frequently visited a nearby (and now-defunct) agency's office, and when a single project was going poorly, the entire offices' mood took a nosedive and the other six teams also got nothing done. Being remote, meanwhile, offers blast damage protection by preventing emotional contagion between projects
• When companies are forming teams, I'd sooner assemble local teams with disparate skills across different departments than "matrix" them out such that many people are driving to work so they can sit on Zoom all day. I can't believe I have to say this, but choosing to build the "perfect" remote team over a "good enough" colocated one without regard for the higher communication and coordination cost it entails is absolutely asinine. The org chart works for you, you don't work for the org chart!
• Remote teams benefit from seeing each other in person from time to time. Throwing a huge in-person party and sprinkling in some team-building exercises every now and then is one way to do it, but I suspect that sending teams to do real work together in person at moments of high uncertainty (e.g. the outset of a new project) is more impactful, cost-effective, and engenders higher employee retention over the long term
• Remote teams would benefit from an audit of the tools they use to spot opportunities to bring back some of the arbitrary constraints the physical world enjoys. If I saw analysts on a remote team attaching full-blown, multi-page specs to JIRA tickets which are then thrown over the fence to developers with an expectation that communication is no longer necessary, I'd sooner disable attachments altogether

More importantly than listening to me about any of this, I'd really encourage everyone to question absolutism on issues like this one and embrace the nuanced trade-offs at play.

Okay, that's enough pontification for one month. If any of this struck a chord with you, remember to mash that reply button and tell me about it.