[trustable-software] What is Trustable About ?

[Dan Shearer]

On Tue, 2 Apr 2019 at 13:48, Paul Sherwood
<paul.sherwood at codethink.co.uk> wrote:
> For your specific example, I expect that we can agree that SQLite is
> excellent software which can be trusted to perform reliably in a wide
> range of scenarios. I think that gap/issue in this specific case is not
> about the software - it's about ensuring an architectural approach to
> avoid hazards due to SQLIte's 'corruptible' design. However I do believe
> our community can help to address it (even if only by triggering smarter
> folks in other communities to run with the ball).

This is a great case study of the human problem being the greatest
block to trustable software, and there are many more case studies of
this kind. It's worth following the links here.

Alas no Paul, I do not think SQLite is excellent software in the
slightest, and not just because it frequently corrupts its database.

Six years ago (!) the SQLite parser was ported to use LMDB (
https://github.com/LMDB/sqlightning ) underneath instead of its
venerable, unreliable BTree code, state of the art perhaps 25 years
ago. LMDB offers MVCC guarantees (
https://en.wikipedia.org/wiki/Multiversion_concurrency_control ) with
cursors, memory maps it's database and  and was written with
efficiency in mind, so it is very significantly faster, between 20%
and 2000% depending on the measure. It also scales to much larger use
cases. Another measure of efficiency is that LMDB is 5k LOC and will
fit in L1 cache on any moderately recent CPU whereas that is not the
case for the backend of SQLite despite it being much less functional.

So why don't a large proportion of the billions of SQLite users in the
world use SQLightning instead, since it comes with reliability and
accountability built in, and modern computer science in its design?
SQLite did eventually learn to do some memory mapped I/O, and there
have been some other modest improvements. But basically SQLite remains
insecure and outdated by deliberate choice. It could have been
licensing perhaps, SQLite being as close to no license as it is
legally possible to get and SQLightning being a mildly different (but
entirely compatible) MIT-type license. SQLightning isn't a stunningly
community-oriented open source project, in fact at times it has seemed
like the original authors just don't really care too much about it!

I have come up with some reasons about what SQLightning did
incorrectly, but basically any any one of the thousands of companies
who are SQLite users had wanted a safer, faster, more scaleable
product, it would have taken very little effort. It would take very
little expense to fix this problem. SQLite is inside tens of billions
in devices, robots, entertainment systems, building management systems
and so on and so on. I'm not providing links for these assertions, by
all means call me out if you think they are unreasonable.

SQLite represents an active choice to use demonstrably worse software;
to ensure that the most-used database in the world is unreliable and
in a word, less trustable. And it could still be fixed.

--
Dan Shearer
dan at shearer.org