Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic

Have you seen "Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic"? [VIDEO]

Well unums in particular (especially the new type 3 "posits" ones) are very interesting and practical for hardware implementation. John Gustafson, the creator of unums, gave the first talk on type 3 unums at Stanford earlier this month, and there are already multiple hardware implementation efforts (one of which I am indirectly overseeing). While many people have recently hyped up low precision floating point for things like machine learning, Posits provide greater dynamic range AND greater accuracy for very few bits (as low as four, but with good ML starting 8 to 10 bits). On top of all of that, it costs less area and power in silicon than IEEE hardware. Check out the talk: [VIDEO]

I'm very surprised nobody has brought up John Gustafson's Unum work. His latest iteration is the posit, which he presented at Stanford earlier this month[0][1] EDIT: apologies to CalChris[2]. What I should have said is discussed Gustafson's Unum work. [0] [VIDEO] (relevant question at 1h22m37s) [1] [LINK] [2] [LINK]

You may like: [VIDEO] It turns out the math is slightly harder but it's faster than IEEE FP in hardware, probably because of less conditionals in the spec. There is better performance in terms of numerical accuracy at the cost of substantially more difficult error analysis. Disclaimer: I'm in the video.

For anyone interested in this sort of thing you'd probably also be interested in John Gustafson's work on unums/posits. Here's a nice introductory presentation [VIDEO]

Are these the Neuromorphic chips [1] Jeff Hawkins of Numenta [2] has been talking about? [1] Neuromorphic Chips [LINK] [2] Numenta papers/videos [LINK] It would also be good to see a major chip manufacturer or cloud provider that makes its own chips (Google/IBM) get serious about graph processing chips [3,4] and moving beyond floating point [5]. [3] Novel Graph Processor Architecture [LINK] [4] Novel Graph Processor Architecture, Prototype System, and Results [LINK] [5] Stanford Seminar: Beyond Floating Point: Next Generation Computer Arithmetic [VIDEO]

> Points near the origin have dark colors, with the color assigned to a complex number z approaching black as z approaches 0. Points far from the origin are light, with the color of z approaching white as |z| approaches infinity. Every complex number has a different color in this picture, so a complex number can be uniquely specified by giving its color. So I was wondering if this approach could also be useful in the context of floats and integers. The implementation becomes simpler in many ways (we only have a finite number of possible values to consider). For example, start with a unique mapping of 8-bit integers to a continuous colour ramp. I suppose the easiest option would be to start in the top left, then go from left to right, top to bottom, resulting in a 16x16 image. If we then take a formula that also returns an 8-bit value, map every possible input to an output, and show the results as an image, it should make it easy to see where we have an integer overflow, for example (if we see sudden jumps from very bright to dark or the other way around). Then I realised: I have seen this in the wild! It's used all over the place in the demo-scene. For example: [LINK] [LINK] Aside, this also reminds me of Gustafson's closure plots for comparing his number encoding to standard floating points, although it's not really the same thing: [VIDEO]

so I'm the author of one of the first usable unum implementations. We're actually moving to something called sigmoid numbers which are even better. The lecture doesn't cover "valid mode" but that's more like the "unum". At the end of the video, there's a demonstration where I show some very interesting results concerning machine learning. [VIDEO] You can try out sigmoid numbers in julia here: [LINK] Also, I am working on some verilog implementations, so we may have this in hardware rather soon. A few things about sigmoid numbers: 1) they have fixed sizes, so you don't have to worry about keeping track of things (which was a pain). 2) they have a property I call "isomorphic structure" - which means that zero-padding does not affect the value, and truncation gives you the nearest value at a lower bit resolution. 3) they really do perform better than floating points in several applications. There's a higher bit-per-bit entropy for 'most things you want to do', so the information content is richer. The distribution of the numbers is "more human" - that is to say at a certain point you stop caring about the details around the number, and more care that it's 'really big' or 'really small'; and the distribution reflects that.