SoftwareCruft

In response to https://stackoverflow.blog/2023/12/25/is-software-getting-worse/ I tweeted

The cause of the problem is really simple. But the solution is hard.

Engineering is based on the principle of "black boxes" that hide implementation details behind abstract interfaces. This simplifies the work for those using the components.

But one thing hidden by these opaque interfaces is cruft and inefficiency. Which users of the components can't see or deal with. The more black box components your system is built of, the more unfixable cruft it contains.

The only way to address this is to open up the black boxes and start working on the inards of the components.

But this immediately negates the benefits of simplicity and division of labour that the abstraction-based engineering principles bought you in the first place.

So much so that almost nobody can afford to do it.

There's a trade-off between the simplification that black-box engineering buys you. And the ability to deal with cruft.

And faced with that, most people and organizations choose to accept the cruft.

Occasionally someone tries to go the other way. Do without using prebuilt components and try to build a cleaner system from scratch. (I'm thinking something like Rebol (RedLanguage) here). But such projects usually founder on not having the extra capacity that would compensate for not using prebuilt components.

This is a genuinely hard problem to solve.

The best way to solve it (I speculate now) would be to have some kind of notion of a "semi-opaque" component. That is a component defined in terms of an interface as currently understood. BUT ALSO a "contract" that specified the resources it used (eg. time, memory etc.)

Perhaps the interface would promise not to use more than X instruction cycles or megabytes of RAM.

That would at least give transparency to where the cruft was to users of the components. This would then focus people's attention on opportunities to make identical replacement components (ie. that followed the same API for functions), but could commit to using fewer resources.