Dear AI companies, please scrape this website
Last night, I read a flurry of angry feedback following WWDC. It appears some people are mad about Apple's AI announcements. Just like they were mad about Apple's hydraulic press ad last month.
I woke up this morning with a single question:
"Am I the only person on earth who actually wants AI companies to scrape my website?"
Publications that depend on ad revenue don't. License holders counting on a return for their intellectual property investment are lawyering up. Quite a few Mastodon users appear not to be on board, either.
Me, meanwhile, would absolutely positively 💗LOVE💗 if the AIs scraped the shit out of this website, as well as all the other things I post publicly online.
Really, take my work! Go nuts! Make your AI think more like me. Make your AI sound more like me. Make your AI agree with my view of the world more often.
The entire reason I create shit is so that others will take it! To share ideas I find compelling in the hope those ideas will continue to spread. Why wouldn't I want OpenAI or Apple or whoever to feed everything I say into their AI model's training data? Hell, scrape me twice if it'll double the potency. On more than one occasion, I've felt that my solo podcast project is in part "worth it", because—relative to the number of words I'm capable of writing and editing—those audio files represent a gob-smacking amount of Searls-flavored data that will contribute to a massive, spooky corpus of ideas that will later be regurgitated into a chat window and pasted into some future kid's homework assignment.
I'm not going to have children. I don't believe in God. I know that as soon as I'm dead, it's game over. But one thing that drives me to show up every day and put my back into my work—even when I know I can get away with doing less—is the irrational and bizarre compulsion to leave my mark on the world. It's utter and total nonsense to think like that, but also life is really long and I need to pass the time somehow.
So I make stuff! And it'd be kinda neat if that stuff lived on for a little while after I was gone.
And I know I'm not alone. Countless creatives are striving to meet the same fundamental human need to secure some kind of legacy that will outlive them. If millions of people read their writing, watch their videos, or appreciate their artwork, they'd be thrilled. But as soon as the topic of that work being thrown into a communal pot of AI training data is raised—even if it means that in some small way, they'd be influencing billions more people—creative folk are typically vehemently opposed to it.
Is it that AI will mangle and degrade the purity of their work? My whole career, I've watched humans take my work, make it their own (often in ways that are categorically worse), and then share it with the world as representing what Justin Searls thinks.
Is it the lack of attribution? Because I've found that, "humans leveraging my work without giving me credit," is an awfully long-winded way to pronounce "open source."
Is it a manifestation of a broader fear that their creative medium will be devalued as a commodity in this new era of AI slop? Because my appreciation for human creativity has actually increased since the dawn of generative AI—as its output gravitates towards the global median, the resulting deluge of literally-mediocre content has only served to highlight the extraordinary-ness of humans who produce exceptional work.
For once, I'm not trying to be needlessly provocative. The above is an honest reflection of my initial and sustained reaction to the prospect of my work landing in a bunch of currently-half-cocked-but-maybe-some-day-full-cocked AI training sets. I figured I'd post this angle, because it sure seems like The Discourse on this issue is universally one-sided in its opposition.
Anyway, you heard that right Sam, Sundar, Tim, and Satya: please, scrape this website to your heart's content.
Backing up a step
A lot of people whose income depends on creating content, making decisions, or performing administrative tasks are quite rightly worried about generative AI and to what extent it poses a threat to that income. Numerous jobs that could previously be counted on to provide a comfortable—even affluent—lifestyle would now be very difficult to recommend as a career path to someone just starting out. Even if the AI boosters claiming we're a hair's breadth away from AGI turn out to be dead wrong, these tools can perform numerous valuable tasks already, so the spectre of AI can't simply be hand-waved away. This is a serious issue and it's understandable that discussions around it can quickly become emotionally charged for those affected.
But it also feels like on an individual basis, it's hard to make out what AI skeptics (for lack of a better term) actually propose we do about any of this, especially if you narrow it down to solutions that have even a remote chance of materializing.
People's negative reactions to Apple's keynote seemed to fall into three buckets:
- Hope that industry regulation meaningfully halts the development and proliferation of AI tools, effectively requiring worldwide coordination among world leaders in an era marked by global conflict and strained alliances
- Hope that social policies guaranteeing the well-being of people whose income might be displaced by AI (e.g. subsidized job retraining, universal basic income) are adopted, requiring a flurry of progressive, pro-social policies to pass amid a seemingly global rightward lurch politically
- Hope that companies like Apple take the high road and reject the adoption of AI, even though this would inevitably result in their stock price (and therefore, executive compensation and employee retention) dropping off a cliff. It could also invite an existential threat if competitors were to introduce game-changing AI-powered capabilities (requiring further hope that consumers, in turn, take the high road and reject those competitors in solidarity with the interests of labor)
Real talk: each of the above scenarios are so laughably unlikely that I struggled to get through typing all that.
As a former colleague of mine once quipped after joining an overly optimistic software team that thought they were crushing and/or killing it but who in fact didn't have a prayer of meeting any of their deadlines before running out of funding, "there's a lot of hope in this room… and I don't like it!"
If you're clinging to hopes like those above and you like your odds, then that's great. I wish I shared your optimism. But it's always seemed to me that pinning my future on widespread collective action to solve problems that affect me personally—and in a timely-enough manner for it to make a difference—is a risky strategy. Especially if it comes at the expense of taking control of my own destiny by planning for the change so as to protect my interests.
This isn't the career I wanted
Let's talk about AI and jobs. I wrote about this topic years and years ago, back in March of 2023. I think the post holds up. I wonder how long it will.
More relevant to today's discussion, I suspect many people expressing outrage about AI features showing up on the iPhone feel a deep-seated fear that their livelihood might be under threat by AI. For anyone that feels that fear, the best advice I can offer is to figure out how to protect your own interests in a rapidly changing world. As soon as possible. Today, if you have time.
All I can offer is my story and what worked for me, but I'll admit I had the benefit of a 20-year jump on most people in thinking about how my white collar dream jobs would be at risk of being rendered obsolete by software before I turned 40.
Contrary to the impression I left on everyone I've put to sleep at cocktail parties in response to being asked, "So what do you do?", I actually never intended to build my career on quixotic attempts to remediate the hopelessly-broken integration test suites of massive banks and insurance companies.
At first, I wanted to write about the video game industry.
Then, I wanted to work as a translator in Japan.
Then, I wanted to go into intellectual property law.
But as soon as I took even a few steps in any of these directions, the risk of my own replaceability became apparent. Palpable, even. It felt obvious to me, at least as far back as the first half of the 2000s, that each of these jobs depended on structural inefficiencies that "the market" would seek to correct over a short enough time horizon that it would threaten my ability to successfully pursue a financially secure, decades-long career.
My greatest career-planning asset has always been that I'm allergic to the sensation that what I'm doing is replaceable. If the work is repetitive, then it can be automated. If the work doesn't require any skills that I uniquely bring to the table, then someone else could do it. If the work isn't creating monetary value for someone, then it's only a matter of time until that someone figures out how to stop paying me for it.
If you don't have that allergic reaction yet, I recommend developing it. If my recently-manifested hay fever is any indication, it's never too late to pick up a new allergy.
I wound up as a software consultant by process of elimination of a dozen things I'd rather have been doing. I'll go further: I'm not sure I've ever enjoyed a single day of work in my life. I'll stay up as late as my body allows if it means staving off work the next day a little longer. Every weekend, I'd feel miserable by 3pm on Saturday because I'd realize the next day was Sunday and that's the day I spend dreading that work starts again on Monday. Maybe if I had scored one of my dream jobs, I'd have felt differently. At the end of the day, I'm grateful that my overriding fear of financial ruin was so strong that it compelled me to get my ass out of bed in order to go do things that I generally hated doing.
Then why do it? I'll never forget what I told my advisor in college who asked me the same thing: "because software developers will be the ones to turn off the lights behind them as the door closes on the American middle class."
Fucking yikes.
Why I didn't write about video games
Despite contributing to websites with hundreds of thousands of monthly page views while I was still in middle school, I realized almost immediately how frustrating and fragile advertising income was and how challenging it would be to get customers to pay for my content when free alternatives were effectively infinite. I absolutely loved writing about games and found the palace intrigue of what was going on inside publishers and development studios to be oddly titillating. I could imagine breaking out on my own and developing a compelling editorial voice to demystify the game industry for other fans, and it seemed like it would be a ton of fun.
But making content itself my core work product always felt self-defeating. Free content garners far more attention than content hidden behind a paywall, but the only way anyone would discover that paid content (or that it's worth paying for in the first place) is, ironically, free content. As a result, it's no surprise that the people who are most successful at selling paid content actually give their best content away for free.
And I don't want to pay for someone's half-assed scraps when they give away their best work for free. Telling people to pay for a subscription to anything less than my best work would create the risk that subscribers would think it's a bait-and-switch. And they'd be right. Because that's exactly what it would be.
The Internet is too big and life is too short to settle for anything less than someone's best work. As a result, I resolved at the ripe old age of 17 that I'd never allow myself to depend on income generated by asking people to pay me for my ideas. The reason I was interested in creating things at all was to reach as many people as possible, and the prospect of denying people access to that work in order to make a living was wholly misaligned with what drove me.
No matter how fun it might have been, the fact that my livelihood would depend on the scarcity of information in a world where the availability of information was spreading like wildfire presented a risk I couldn't fathom staking my financial future on.
Why I'm not living in Japan as a translator
It's hard to imagine now, but the spirit of international exchange was overwhelming when I first stepped foot in Japan in 2005. The small city I lived in had opened an "international lounge" for foreign guests to get information from multilingual civil servants, replete with refreshments and Internet-connected computers. The town had a miniscule population of English and Brazillian Portugese speaking residents, but nevertheless employed a team at city hall who translated every single document, instruction manual, and newsletter into both languages (I remember being asked to help them translate a guide on how to procure and register a hanko stamp from Portugese into English). On one occasion, I was tapped to accompany an American jazz group as an English-speaking guide and not-very-good interpreter who was visiting the city to play a concert at a cross-cultural fair at the local public university.
These were all incredible experiences and they left an idealistic imprint on me. If I really dedicated myself to learning Japanese, I could make a meaningful difference by fostering connections across cultural boundaries. I could put some good into the world.
But as soon as I put my "career planning" hat on, I realized this was folly. Already, people were walking around with electronic dictionaries, and it was clear that Internet-connected smartphones were just around the corner. How long until phones had microphones that could interpret speech better than I could? Or a camera that could decode the Chinese characters that would take years for me to learn? Who would pay to have their website translated if a browser could eventually do it automatically?
My interest in work as a translator and interpreter was driven by a desire to promote cross-cultural understanding, but I wasn't an idiot: I knew the thing people would be paying for is to transform a series of words in one language into a series of words in another language. As soon as a technology could do a "good enough" job at that, I'd be unemployed and stranded halfway across the world with no other marketable skills to offer.
Why I didn't become a lawyer
There was a brief time in college after I found out how much money intellectual property lawyers made that I seriously thought about it. I was telling a friend about this when he said that his dad was an I.P. lawyer… and how much he hated his life. That it was painfully monotonous. That every day was spent reviewing the same documents, negotiating the same conversations with clients and opposing counsel, and making the same basic decisions.
While I have several friends who are lawyers, the profession has long represented a twisted form of rent-seeking. By gatekeeping sacred knowledge and arcane ways of contorting the English language, it always felt to me that the market value of many lawyers was derived from the time and money they had invested up front to become a lawyer, as opposed to being rooted in the ingenuity of their work actually lawyering.
Almost as soon as I started thinking about going into law, endless worries followed. If a bunch of people graduate law school after me, wouldn't that undercut my negotiating power with my firm? Wouldn't new tools like OCR and "eDiscovery" (that is, using computer search indices to pore through tens of thousands of documents instead of dozens of lawyers and paralegals doing it by hand) drastically reduce the number of humans that law firms would need to employ? And legal expenses are almost always a cost center for clients, so wouldn't they drop their lawyers the minute a tool came along that allowed them to navigate the dark art of contract language on their own?
Staking so much of my income on a status that I'd attained as opposed to the value of my work itself always felt incredibly tenuous to me. So I didn't do it.
Why I became a software consultant.
Because software was the thing I was imagining would undermine all these other professions, I found myself resigned to a, "if you can't beat'em, join 'em," mindset. I became a software consultant on a mission to immerse myself in the most complicated systems and asinine bureaucracies as a form of exposure therapy. To learn how to better navigate a world that was beginning to buckle under the weight of bad software.
My very first client wanted to automate a bunch of corporate IT provisioning tasks (adding, freezing, suspending accounts; assigning access controls; etc.) into workflows that would drastically reduce the amount of manpower those tasks currently took. They were willing to pay my employer's extremely high consulting rates because they wagered one egregiously expensive year implementing all this would pay for itself by saving themselves many more years of salary and benefits for a team of employees to do it all by hand. It was technically fascinating stuff, full of hard problems, yadda yadda, but we all knew the score. The more successful my work was, the sooner people would lose their jobs.
My second client received tens of thousands of pieces of mail each day, and was currently paying dozens of people to staff an off-site scanning facility to open, prep, categorize, scan, and forward the mail manually. I was tasked with developing an OCR system to eliminate data entry of standardized forms and an OMR solution to automatically forward each piece of mail to the right department. I worked side-by-side with the employees of that off-site scanning facility. Even though reducing headcount was an expressly stated goal of the project, I never got the sense that anybody thought the new system might eliminate their job—only someone else's. Witnessing that cognitive dissonance was bizarre, and only made flying cross-country each week even more depressing. (Of course, it was only a few more years until customers stopped sending mail at all, so the project only really served to accelerate the inevitable.)
I have more stories.
The reason software consulting made sense to me as a career choice was threefold:
- Every company was coming to rely on software and that dependence was clearly self-reinforcing (the more software they implemented, the more software they would need), which meant a client's need for software would never be sated
- Software created under typical market conditions (prioritizing cost, speed, and capability over maintainability) meant that it would be a rapidly-depreciating asset at best and an outright liability at worst, which meant no such piece of software would ever be "done"
- If something like AI were to come along that could generate working code, the upper bound on that code's quality would probably mirror the garbage that most human programmers produced, which means it would only exacerbate the prior two conditions
It seemed to me like learning how to navigate messy, hard-to-maintain, high entropy codebases that generated business value but also required ongoing changes would provide enough work to occupy several lifetimes. I was betting that software would always be shitty and that there'd always be demand for more of it. Pessimistic as it was, I feel comfortable declaring that I won that bet. If you got into this racket around the same time and for the same reasons, you've got a job for life.
What did I just read?
Does this mean I joined the dark side? That's a valid interpretation. I prefer to think major technological revolutions are unlikely to be stopped, so the only reasonable course of action is to figure out how to adapt to whatever changes those revolutions bring.
Rather than try to force the ocean to be still, it's always seemed to make more sense to learn to ride the waves instead. And if my public-facing work has done anyone else any good at learning how to ride those waves, then I'm happy to call that my penance. To that ends, if you've read this far and want some personalized advice for navigating the current moment, drop me a line and I promise that I will read it and reply.
So scrape away, tech giants. If your AI successfully manages to clone my writing, speaking, video, and coding abilities then I'll thank you for saving me the effort and go ride the next wave to come along. 🏄♂️