Regarding the JVM as a strict superset, there are a few features I'm not sure fit that model.
Isolated garbage collection. Limiting GC to a single process/thread. It's my understanding that this is only possible with individual heaps, prohibiting shared memory. I see this as a polarizing tradeoff (with pros and cons). I don't think the JVM does this, and therefore isn't a superset.
Preemptive scheduling and soft real time. It's my understanding that Quasar bridges this gap by looking for natural points in execution to insert a pause/continue (oversimplified I'm sure). What are the guarantees? Erlang provides fairly strong guarantees about fair scheduling, making it very suitable for soft real-time systems. Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?
> It's my understanding that this is only possible with individual heaps, prohibiting shared memory.
Yes, but the only advantage is a simpler GC algorithm. HotSpot's GCs are so advanced (let alone HotSpot descendents like Zing) that they give you similar behavior even without isolation, and they include collection of shared data structures, too.
> Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?
Quasar does full preemptive scheduling but not time-sliced based preemption. This is not because it can't -- as a matter of fact, early versions of Quasar did time-sliced based preemption, but it turned out to provide no benefit whatsoever. In fact, Erlang's behavior is a limitation. The reason is that Erlang has only one type of thread -- the process, or the user-mode thread -- and therefore has to handle any type of thread, including those that are computation heavy. Thing is, work-stealing is a great scheduling mechanism for transaction-serving threads (that block very often), but not so good for computational threads. Quasar lets you choose: fibers for transactions stuff, and plain threads for long-running computations, both are abstracted into what we call a "strand".
I'm skeptical that Hotspots GC can provide the same behavior, but I'm not familiar enough to challenge that claim. My skepticism arises from most JVM benchmarks around latency (for example, a webserver) having a significant standard deviation, which I had always attributed to an overly broad GC.
I have a chip on my shoulder about limited concurrency abstractions. I've been burned too many times by Akka and thread pools and Scala Parallel collections etc.. I've always appreciated the fact that while I won't get the most performance out of BEAM, there's very little I can do wrong that will break my application.
I'm going to have to spend some time with Quasar. If your claims hold up, then Clojure plus Quasar seems like fantastic platform.
The JDK has some of the best implementations of lock-free data structures anywhere, and they are actually all implemented in Java (the JVM exposes direct access to memory fences and CAS). You really never have to drop down to C unless you want to interface with some OS-specific code.
The JVM is simply much more general-purpose. Just like your machine and OS themselves can support Erlang, so can the JVM.
In Armstrong Thesis there are explicit assertion about why thread-based JVM is a flawed design.
My point was that there is a big difference between a VM for a functional and imperative language (immutability makes it and GC a lot simpler). JVM's selling points, that it is "general" and is "already here" are strong, but in my opinion, its praise is more religion-like repetition of dogmas and marketing memes.
I would like to assert that Erlang designers have made better design decisions (to avoid threading and mutation whatsoever) so Erlang's VM is much simpler, more stable (JVM crashes often, wastes memory - memory used vs. memory served ratio, and GC behavior tend to become unpredictable for long-running process) and good-enough.
So is this a sort of Greenspun's tenth rule situation? Any sufficiently advanced distributed systems contains a buggy implementation of Erlang?
More seriously, is that what you are suggesting? By rewriting (or heavily reconfiguring) the GC, adding a scheduler, etc you end up with all the benefits of Erlang while maintaining the multi-paradigm platform of the JVM? Sounds amazing. Give it to me with out of the box sane configuration and I'm sold.
It sounds like it's not quite the case, especially around distributed systems. But perhaps that's just the intended design because clustering machines is sort of out of style in the age of RESTful microservices.
> Any sufficiently advanced distributed systems contains a buggy implementation of Erlang?
Why won't you say that about machine code? The JVM simply operates at a lower level than Erlang, hence it can be used to implement Erlang, and, it turns out that doing so might gives better results than current Erlang VMs implemented in C, because there are man-decades (or centuries) of shared functionality that's already in the JVM.
> By rewriting (or heavily reconfiguring) the GC, adding a scheduler, etc
You don't need to rewrite or heavily configure the GC (besides, which one? There are plenty of JVM GCs) -- just to use it as Erlang does, or one that is pauseless no matter how you use it (two implementations, one commercial and one open-source under development at Red Hat). Also, you don't need to add a scheduler. One of the world's best implementations of a work-stealing scheduler, also with about a decade of effort behind it by Doug Lea, is packaged with the JDK. See here[1] to get a feel for the effort behind it.
> It sounds like it's not quite the case, especially around distributed systems.
Erlang doesn't add anything not already found in high-quality, battle-tested libraries like JGroups[2] when it comes to clustering (actually, JGroups is more powerful than Erlang's clustering). The JVM ecosystem is so huge that it's very hard to come up with something (other than syntax) that doesn't already have a solid implementation out there.
I'd really like to retrofit Erjang to use Quasar rather than Kilim (Quasar is just better maintained, more battle-tested and probably faster).
I really believe that running Erlang on the JVM is a win for everyone. The Erlang community simply doesn't have a tenth of the resources that are put into OpenJDK[3]. Why not enjoy the benefits?
Greenspun's tenth rule is tongue-in-cheek and usually followed by something like "...including Erlang itself".
I get it. I see the huge benefits in a ground-up design for Erlang in the JVM. (Said design should definitely use existing GC, scheduler, etc.).
However, while what you are describing is technically the JVM, it's sort of a Ship of Theseus (with the GC replaced, with static binary manipulation by Quasar, etc.). This isn't a bad thing, but it's not what I'd normally associate with the JVM and its ecosystem.
I've got a bit of a "chip on my shoulder" regarding what I see as a pretty terrible experience with Akka. I feel strongly that preemption makes a huge difference in usability. With Akka, there are too many caveats relating to things you can and cannot do. Rather than simplifying my life, Akka gives me a whole bunch of additional ways I can break my application. While it could be said that "being a professional engineer" means solidly understanding my tools, I don't feel that is a scalable way to build development teams.
I'd love an out-of-the-box "erlang mode" switch (or some sort of launcher) that would give me these safe behaviors. Something along the lines of Percona for MySQL.
Reach out to me directly with email (in profile). This has been a fascinating discussion and I'm very interested in learning more.
But you don't need to replace the GC. HotSpot alone comes with a choice of 3 (or 4) GCs, and a fifth will join them. As of Java 9, HotSpot will support pluggable JITs (written in Java!). Quasar's bytecode manipulation is common practice among a large number of JVM tools, including one of the most popular Maven plugins, many of the most popular JVM profilers, and Spring. It isn't some exotic technique.
And don't forget -- the JVM is a spec with many implementations (including some for hard-realtime systems), although I'm talking of OpenJDK's HotSpot here.
I totally agree about preemption (and Akka, obviously). Blocking a thread (userspace or kernel) is the essential abstraction of imperative programming (or, more accurately, suspending a continuation) of that is the essence of imperative programming (even functional imperative programming, like Erlang and Clojure).
Isolated garbage collection. Limiting GC to a single process/thread. It's my understanding that this is only possible with individual heaps, prohibiting shared memory. I see this as a polarizing tradeoff (with pros and cons). I don't think the JVM does this, and therefore isn't a superset.
Preemptive scheduling and soft real time. It's my understanding that Quasar bridges this gap by looking for natural points in execution to insert a pause/continue (oversimplified I'm sure). What are the guarantees? Erlang provides fairly strong guarantees about fair scheduling, making it very suitable for soft real-time systems. Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?