2 reactions 2019-09-15
If you clicked on this title with an open mind, you probably don’t exist, but welcome anyway.
- Massive Community
- Moving Target, Moving Goalposts
- No One is in Charge
- We Can’t Take Ownership of Our Tools
- We Treat Toolmakers Like Tools
- It doesn’t all suck
- Other references
JS’ culture derives very much from its original design decisions. Being easy to get up and running opened doors for many programmers, but with it come all the flaws, right down to undefined property access not causing a runtime error and causing the famous ”
Which leads to…
As the largest non-spreadsheet programming language on Earth, you are guaranteed to have near-religious schisms in everything from code style to software delivery to testing approaches. One of the chief tools is even called Babel, for crying out loud! (This is deliciously ironic given the story it references, on many layers)
A huge community is usually a plus, but people also forget it can have downsides. Having to make everybody happy when it literally means EVERYBODY is a very different and sometimes conflicting proposition from “do one thing well”.
The simplest way to make people happy is to add config after config, so that’s what we do. However this way lies config hell and still doesn’t help tools interoperate. I like Jared Palmer’s approach of paying attention to composition and the 80% usecase. More people should also try tools like
patch-package for maintaining low cost forks.
A huge community is great, until it leads to…
We don’t agree on whether JS libraries should work in both the browser and in Node.
We don’t agree on whether JS tools should transpile their own dependencies.
We don’t agree on how to deal with legacy browsers. As time goes on, this will become less and less of an issue, but we’re still in the thick of it. Create-React-App has had ongoing debates about differential serving for 2 years.
Ultimately, we don’t agree on the problems our tools are meant to solve. At one extreme, this manifests as beginners asking what’s the difference between A and B (which are frequently mentioned as alternatives) and their maintainers replying that A and B are “orthogonal”. At the other extreme, this causes experts in A (who care a lot about one thing) to yell at experts in B (who are trying to do something else).
The moment we even come close to agreeing on something, goalposts move, and something else comes up to disagree on. We rather tear each other down on small things we disagree on, more than support and remind each other about the big things we all want, causing burnout.
Which brings me to…
Yes, we will have a new frontend framework every so often (not every day, nor every month, stop being disingenuous, it isn’t funny making jokes about dimissing people’s work without even looking at it). Yes, we have 3-4 different bundlers that all do different things. Yes we have file formats nobody uses (yet?). Yes, we download 3-liner packages 93 million times a month.
This is extremely wasteful and duplicative and insecure and exhausting and hard to learn.
We know. It’s also the best way to ensure we have tried and tested ideas before committing them forever to some monolithic thing everyone has to use. If some tool goes bad, we have another waiting in the wings. If the alternative thing is materially better, the incumbent will initially ignore it but eventually try to catch up. It may fail to do so.
Which in fact means that we don’t bother checking if we’re using the right tool, and can’t because our tools use thousands of tools…
I’ll break the writing style of this column so far by relating a recent incident. I can do this because I don’t have an editor tut-tutting at me. If I lose you, though, let me know.
By default, all
npm installs assign packages with the caret (
^) which matches minor and major versions of a package, assuming that packages follow semver. However, semver is a social contract - trivially, if any users rely on bugs, then a bugfix is a breaking change, so semver relies on the definition of a bug, which has the very technical definition of “software not working as we thought we intended”. More importantly, semver isn’t enforced in any meaningful way when published, so it can’t be relied upon when downloading. Yet that is exactly what we do when we use the caret by default.
event-stream happened, and a package was removed even though it was on the register, so all JS tooling and apps built on that tooling started failing. Package management security is above my paygrade, but the reason it had so much impact was because of a single character:
flatmap-stream depended on
event-stream, and that is in
fs-events, which everyone depends on, so Vue, React, Angular, and probably everyone else went down. How could a supposedly competitive, massive, varied ecosystem have a single point of failure? I panicked a bit, but knew how to fix it. I imagine a lot of beginners were turned off that day. I imagine some people (like me) had demos to show people my software and ran into this at the first try and immediately got a bad impression like it was my fault.
It kind of is, but it kind of isn’t. npm could fix my scenario above by making default installations pinned versions (apparently the Go ecosystem does this), but that doesn’t address the real root of the problem - I didn’t know what I was using. That day was the first day I even heard of
Here is where I get heretical. I’m supposed to tell you it’s my fault. That implies it’s your fault too, if you’ve had it happen to you. I originally titled this section “We Don’t Take Ownership of our Tools”.
I changed my mind.
A lot of prominent developers like to say that you should treat
node_modules like your own code, but that just doesn’t ring true, and makes it sound like it’s your fault. It’s near impossible (Brian from Begin comes to mind as an exception) to own your entire dependency chain all the way down. Thomas Young was the Last Man Who Knew Everything and he died in 1829.
Rich Harris, He who Kicks Hornet Nests for Fun, also pointed out the flaws of the small modules approach. In particular, he picks on discoverability and software integrity: ”Everyone has an idea about how to make it easier to find needles in the haystack, but no-one bothers to ask what all this hay is doing here in the first place.” He was largely ignored - how do you stop an absolutely massive community in its tracks?
Even if you’re the model developer, you are not an island. You will work with people who are not model developers, and who don’t own their tools. Therefore, it is a statistical eventuality that you will start using tools you don’t know you use. I call this the Hashimoto lemma, after one of Mitchell Hashimoto’s reponses on Why Multicloud.
Basically, if our entire toolchain/productivity strategy relies on owning every dependency just to be reliable, we should probably explore other ideas. (To be clear: I don’t have one. Just thinking out loud.)
You don’t have to know everything, but that doesn’t mean not caring about the makers behind the tools you use. But this is a real problem, and it is an epidemic.
This asymmetry means that a vast other-ness applies when users of tools interact with makers of tools. And because of the relative knowledge difference, as well as just general support goodwill, it’s the makers of the tools who often have to bend over backwards to help users diagnose and even fill out basic bug reports.
What’s worse is, there is a rational explanation for all of this. There really is value in putting together a bunch of things you didn’t make and selling it as a cohesive package. It really is wise to be more cautious in some tech choices. But for goodness sake don’t stop trying things out and contributing back. And don’t ever feel smug about treating toolmakers like tools.
It’s endemic that more junior engineers work on tools while senior engineers tend to sit back and think about bigger picture things. However I wish we could pass knowledge down better. It’s too glib to say “back in my day we knew how to do software right! Kids these days just don’t know how to code!” (Usually leading to some gatekeeping on bootcamps and degrees) It’s unrealistic to expect senior people don’t move beyond just thinking about tools, because there is so much more to software production and management than tools. But I wish we had better ways to pass down and communicate lessons learned, instead of being snarky on social media.
Of course, nobody’s going to change their behavior just because I wrote this, so that leads toolmakers to say… f*** you, pay me.
🗣️As an aside, 97-99% is a fun recurring number in anything to do with the commons. It corresponds to the ratio between consumers and creators of content. And also the amount of DNA we share with chimps.
The problems with open source maintainership are well documented. There are many, many, many failed experiments with getting funding for open source. (To be clear, a lot of them were bad ideas to begin with) After one of the most recent attempts, Feross concluded that “An open source maintainer is a startup founder but with none of the upside.” This rings very true, although occasionally open source maintainers do eventually create startups around their work.
Solutions I do dislike:
- Advertising business models (on docs and in my console)
- Begging on Patreon
- Begging on Github
- any form of financial contribution where developers pay out of their own pockets instead of their employers (if applicable)
- Bounty systems like Gitcoin
Solutions I do like:
- Training (aka React Training)
- Managed Service (Laravel Forge, Meteor Galaxy)
- I highly recommend Joseph Jacks’ thinking on “Open Core” and OSS business models. However, do note that all of these are forms of trading off increasingly less open cooperation and software for more money.
Meditations on economics of open business models aside, expecting extrinsic motivation to beat intrinsic motivation is a bad bet. But we can’t help it, we’re addicted to it. Dan Pink’s Drive puts it best:
With the important caveat that basic needs are taken care of, I do fundamentally agree with DHH that intrinsic motivation is the best way to create sustainable open source. You may be financially successful with the OSS business model you pick, but if you are driven by external incentives, then at some point down the line your external incentives lead you to make choices and compromises that aren’t to do with solving the problem you set out to solve.
📺Do yourself a favor and watch or read DHH’s RailsConf 2019 keynote on Open Source Beyond the Market to understand.
As to counter the individual points I made:
- Community: We are getting better at establishing Codes of Conduct, GitHub implemented issue templates and reactions to prompt people to work together better. We are constantly learning about better composition and abstraction patterns in the highest stress-test of these patterns there has ever been
- Moving Target: Increasing adoption of evergreen browsers are helping to offer a stable target. Although cross-browser inconsistencies still necessitate abstraction layers, their differences are far smaller and have less impact on tool choice then ever before.
- No One in Charge: This is a feature, not a bug.
- Ownership: I don’t have ideas here.
- Funding: Although I’ve explained my reservations, there are plenty of initiatives to help address open source funding and they are all needed. I would follow closely Devon Zuegel of GitHub and also plug my friend Travis Fischer if you are interested in this.
- Seth Godin - Why is software so bad?