SQLite: The Database at the Edge of the Network with Dr. Richard Hipp

I stumbled on this video presentation by Dr Hipp recently (may have been from another HN comment). I really enjoyed it. Probably because he seems so enthusiastic and passionate. SQLite: The Database at the Edge of the Network with Dr. Richard Hipp [VIDEO]

This point is echoed by the creator of SQLite, Dr Richard Hipp, in his excellent talk "SQLite: The Database at the Edge of the Network". [1] "SQLite doesn't compete with client-server databases, it competes with fopen" (I highly recommend to watch it, if only to get a sense of the smart and funny energy that Dr Richard Hipp puts off!) [1]: [VIDEO]

> Hmmmm, there's a better talk by D. Richard Hipp around, but I'm not finding it atm. In the better talk he explains they (SQLite) are planning to be around for the 30 year timeframe you mention. Something to do with an airplane manufacturer looking for long term guarantees, but I don't remember the details. Here's the video[1] linked to from a tweet[2] linked to above.[3] [1]: [VIDEO] [2]: [LINK] [3]: [LINK]

This talk is really interesting too [VIDEO] big industry projects (like the airline mentioned) probably contribute

Here's Dr. Richard Hipp mentioning that Airbus requested sqlite support for the lifetime of the airframe: [VIDEO]

They actually have to test to that degree to follow aviation standards (DO-178b [0]) because they're used in aviation equipment. Dr. Hipp said he started really following it when Android came out and included SQLite and suddenly there were 200M mobile SQLite users finding edge cases: [VIDEO] Lightly edited transcript here: > It made a huge difference. That that was when Android was just kicking off. In fact Android might not have been publicly announced, but we had been called in to help with getting Android going with SQLite. [Actually], they had been publicly announced and there were a bunch of Android phones out and we were getting flooded with problems coming in from Android. > I mean it worked great in the lab it worked great in all the testing and then [...] you give it to 200 million people and let them start clicking on their phone all day and suddenly bugs come up. And this is a big problem for us. > So I started doing following this DO-178b process and it took a good solid year to get us there. Good solid year of 12 hour days, six days a week, I mean we really really pushed but we got it there. And you know, once we got SQLite to the point where it was at that DO-178b level, standard, we still get bugs but you know they're very manageable. They're infrequent and they don't affect nearly as many people. > So it's been a huge huge thing. If you're writing an application deal ones, you know a website, a DO-178b/a is way overkill, okay? It's just because it's very expen

I've been reading through many of the other pages on the SQLite website. I didn't realize that SQLite is developed by a company (Hwaci) and funded by selling support, licenses, and testing services. I had always assumed it was an open source project backed by a non-profit, similar to the Apache Software Foundation. This is another very interesting example of an open-source business. I would be interested to learn more about how the Hwaci company operates (revenue, number of employees, etc.). I find this very interesting: > We are a 100% engineering company. There is no sales staff. Our goal is to provide outstanding service and honest advice without spin or sales-talk. They list some "$8K-50K/year" and "$85k" price tags directly on their "Pro Support" webpage. These would usually be behind a "Schedule a Call" or "Get Quote" button. I've been thinking about doing something similar with my own on-premise licenses and support contracts. I'm not very good at sales and I don't really want to hire a sales team, so I'd be interested to know how this worked out for them. I also liked this sentence, which is very similar to Basecamp's philosophy (and both companies were started around the same time - 1999 vs 2000): > Hwaci intends to continue operating in its current form, and at roughly its current size until at least the year 2050. It's interesting to think that SQLite could have raised money and grown into a billion-dollar public company with thousands of employees. I'm going to li

DuckDB looks very interesting and I'm quite excited to examine it more closely in the next few days! I just wanted to add to the discussion that an unchanging file format, or at least a backwards compatible one, is a key feature of sqlite. See for example Richard Hipp's comments here [1] (I think he also mentioned earlier in the talk that the file format has become a limiting factor now in terms of some of the refactoring that they can do). The file format therefore seems likely to be a major factor in the long term success of this project and I am glad to see that you are taking your time before settling on any architecture here. Given that you are targeting the data science and analytics space, what are your plans for integration with arrow and the feather file format? From a purely user/developer perspective, arrow's aim of shared memory data structures across different analytics tools, seems like a great goal. I know Wes McKinney and Ursa Labs have also spent quite some time at the file storage part of this, see for example the Feather V2 announcement [2]. What are your thoughts on the tradeoffs they considered and how do you see the requirements of DuckDB in relation to theirs? From the Carnegie Mellon DuckDB talk [3], I saw that you already have a zero-copy reader to the pandas memory data structures, so the vision I have is that DuckDB could be the universal SQL interface to arrow datasets which can then also be shared with more complex ML models. Is that something tha