Comments on as simple as possible, but no simpler: Box2D as a Measure of Runtime Performance

I'm coming to this late, but something for fut...

2012-03-15T05:35:04.138-04:00

I'm coming to this late, but something for future blog posts...

It would have been a lot easier to read the graphs if you had picked a color for a given implementation, and then kept that color between the various graphs.

Eg, you used Blue for the C code in graphs 1 and 2, but then re-used Blue in the third graph for Box2d-web (safari nightly). At first glance, I thought the second graph was just a zoom-in on the bottom 3 lines, not a zoom-in on the bottom 4 lines with one omitted.

I'm guessing these were just the default colors for graphs, but it would have made it easier to read.

I wonder if ChipmunkJS would go faster -- it's a hand port, with more JS-specific optimisations than box2dweb.

Here is the comparison I was talking about in my l...

2012-02-27T16:37:58.536-05:00

Here is the comparison I was talking about in my last comment:

JS Linked vs. C++ Compiled

The JS linked version is the one running a version of the benchmark written in JavaScript dynamically linked to the emscripten library output.

The C++ compiled version runs a C++ version of the benchmark which was compiled along with the box2d library by emscripten.

On my machine the JavaScript version is either faster on the whole or exhibits a better garbage collection/loop tracing profile. Also worth noting is how different the performance is between V8 and SpiderMonkey.

2012-02-27T16:36:56.532-05:00

This comment has been removed by the author.

@Nathan: I just merged your pull request, and am g...

2012-02-27T10:56:01.444-05:00

@Nathan: I just merged your pull request, and am getting something on the order of 90ms/f on the same MacBook Pro I used for the other benchmarks. I've asked Alon to have a look to make sure I'm not screwing anything up.

Luis, Alon included a JS version of the bench2d te...

2012-02-25T16:47:18.958-05:00

Luis, Alon included a JS version of the bench2d test code against the JS bindings. Take a look at: https://github.com/kripken/box2d.js/blob/master/test.js

It is quite simple to run that test, but it isn't quite an even-handed comparison to Joel's existing Emscripten version–those numbers measured the performance of a transpiled LLVM BC/C++ benchmark.

However, I feel like this is a much more interesting approach: it is effectively generating a dynamically-linkable JS library.

I'm very curious as to the performance impact of using JS to interface with the lib as opposed to transpiled JS linking.

(However, I really don't feel like massaging all of the tests to run on my machine so that I can have consistent values. So Joel, can you simply run test.js on the same machine you've run all of the others on? Pretty please?)

Would love to see how azakai's new one compare...

2012-02-22T12:34:57.730-05:00

Would love to see how azakai's new one compares to the others.

http://mozakai.blogspot.com/2012/02/box2djs-box2d-on-web-is-getting-faster.html

Jeff: I'm pretty sure the author is about to d...

2012-01-02T22:14:35.238-05:00

Jeff: I'm pretty sure the author is about to do just that, but feel free to take a crack at it. I suspect it will still be behind V8 by a pretty wide margin at this point (still very early days for the optimizing Dart VM). But the existence of the benchmark should give the team something to work from in their optimization efforts. I hear they really like to make things fast :)

Hey Joel, I was doing some searching today and f...

2011-12-29T13:25:25.282-05:00

Hey Joel,

I was doing some searching today and found

http://code.google.com/p/dartbox2d/

I haven't tried it yet, but was wondering if you were interested in seeing how dart vm preformed with this aswell.

@jpvincent: I don't have much spare time to do...

2011-12-20T11:15:39.169-05:00

@jpvincent: I don't have much spare time to do so, but it shouldn't be hard to just check out the repository and stick it in a local HTTP server. The Mandreel and Emscripten compiled output is included in the repo, and you can run the Javascript version from source.

Do be aware that they all seem to have problems on IE9/10. Box2d-web appears to use some of the newer get/set Javascript syntax, and Mandreel/Emscripten really need TypedArrays to perform reasonably.

Is there a way you provide the benchmark as a stan...

2011-12-20T02:15:35.468-05:00

Is there a way you provide the benchmark as a standalone page ? I would like to compare JS engines on windows 8

@Osvaldo > But JS is wildly more difficult, an...

2011-12-17T15:55:48.676-05:00

@Osvaldo

> But JS is wildly more difficult, anything can change anytime, so optimized code will always be shock-full of guard code, and general bloat like fast- plus slow-paths, deopt metadata, instrumentation etc.

Not necessarily, analysis methods like SpiderMonkey TI can detect what things cannot change and do not need guards at all.

In theory anything can change, but it usually won't, especially in high-performance code, and that behavior is detectable both statically and dynamically. Again, this isn't just theoretical, it's going into a stable browser next week (FF9).

- azakai

@azakai: In theory, none; but in theory Java can a...

2011-12-16T21:24:21.530-05:00

@azakai: In theory, none; but in theory Java can also be as fast as 100%, which often happens but not always. Compiler optimization is not magic, for one thing most of the time it can reduce overheads but not completely remove them. Method dispatch is a good example, you can devirtualize and even inline but you may need to insert some guard code to bail out when optimistic assumptions are violated. Now in a language like Java, the JIT can often avoid that guard code, or use classloading for that, because the static-typed typesystem and the rigid class structure are critically helpful to the compiler. But JS is wildly more difficult, anything can change anytime, so optimized code will always be shock-full of guard code, and general bloat like fast- plus slow-paths, deopt metadata, instrumentation etc. And the very best optimization is only achievable when optimizing huge basic blocks built by very aggressive inlining; which is great for benchmarks that spend 90% of the time in a few "hot spots", but doesn't map nicely to a next-gen app with a MLOC of JS code and a mostly flat profile.

@Osvaldo: > OTOH, Java has some intrinsic effi...

2011-12-16T13:49:19.056-05:00

@Osvaldo:

> OTOH, Java has some intrinsic efficiency aspects that JS will hardly ever match, enabling most classic compiler optimizations and them some

What in your opinion is a classic compiler optimization that *cannot* be done in an SSA-optimizing JS JIT like CrankShaft?

@Isaac: Thanks for the link. I just noticed that J...

2011-12-16T12:21:06.954-05:00

@Isaac: Thanks for the link. I just noticed that JBox2D has a flag to turn on LUT-based transcendentals. Turning this on yields better performance -- just under 2.5x native.

I remain stunned that there's no built-in fast-sin implementation in Java.

@Osvaldo: Yes -- i was using "dynamic VM"...

2011-12-16T12:18:04.448-05:00

@Osvaldo: Yes -- i was using "dynamic VM" as rough shorthand for "a safe virtual machine with garbage collection and all that good stuff". I doubt it will ever be possible for Javascript VMs to match JVM or CLR performance, but it's still a useful target, even if it's an asymptote.

I brought this up primarily because I think it would be simply unrealistic to expect Javascript VMs to ever reach native performance (except perhaps on very narrow synthetic benchmarks). JVM/CLR performance is a bit more reasonable to expect.

@Joel Webber -Damn, somehow I had no idea that...

2011-12-16T11:58:00.812-05:00
@Joel Webber -Damn, somehow I had no idea that this was the case on the JVM!

This came up a few years ago in the benchmarks game, see the FastMath workaround in this Java partial-sums program.

"I suspect the JVM's performance can be t...

2011-12-16T11:45:51.155-05:00
"I suspect the JVM's performance can be taken as a rough lower-bound on what one can expect from any dynamic VM. Thus, Javascript VM implementors can take the JVM's performance as a reasonable target to shoot for."

The limitations of Java performance come not much from its dynamic (JIT) aspect, but from other things like simple typesystem (.NET's CLR is much better here and still managed), memory model (safe, GC'd, no control over layout / locality etc.), limited use of some HW facilities from CPU's vector units to GPU's programmable shaders, and portability (btw this is the reason for the expensive transcendental functions: the spec demands well-defined results independent from OS/CPU/FPU).

OTOH, Java has some intrinsic efficiency aspects that JS will hardly ever match, enabling most classic compiler optimizations and them some. IMHO Java will always be a very distant target for JS, almost as much as native code. Hoping though that Dart can fix at least a big part of this gap.

@gsnedders: 11.60. Just updated and re-ran the tes...

2011-12-16T10:49:58.235-05:00
@gsnedders: 11.60. Just updated and re-ran the tests to be sure. Results are pretty consistent.

What version of Opera are you testing there? 11.60...

2011-12-16T10:22:52.462-05:00
What version of Opera are you testing there? 11.60? There were some fairly major changes made between 11.50 and 11.60.

@Markus: That's actually what I meant -- JSCor...

2011-12-16T10:20:40.985-05:00
@Markus: That's actually what I meant -- JSCore (Safari) has a lower variance, *possibly* because those spikes in the V8 numbers are GC pauses. But I haven't yet verified whether they're actually GC pauses, or something else entirely.

@Unknown: Note as above -- if you or anyone else h...

2011-12-16T10:07:36.608-05:00
@Unknown: Note as above -- if you or anyone else has time to setup a Flash version of the test, that would be greatly appreciated.

w.r.t. Alchemy -- Mandreel actually has a Flash target as well. I haven't set it up yet, but given its performance on V8, I'm guessing it would show the AS3 VM in the best light. I'll see if I can find time to add it to the mix.

@Toby: Damn, somehow I had no idea that this was t...

2011-12-16T10:06:07.807-05:00
@Toby: Damn, somehow I had no idea that this was the case on the JVM! That sucks very much badly; I'll find some reasonable fast-transcendental implementations and update the numbers in a followup edit.

-1 to pedantry. If I wanted slow arithmetic I would have asked for it :(

@David Masover: Sorry, wasn't clear about this...

2011-12-16T10:04:41.071-05:00
@David Masover: Sorry, wasn't clear about this -- it's the standard JRE 1.6 on my MacBook Pro, which means that it's always --server (there's no client VM on the Mac, and I've not mucked about with OpenJDK).

So it is running Hotspot, and I gave it 64 frames of warmup time, which seems to be enough to make Hotspot happy (I gave all the Javascript VMs the same warmup time; the native/NaCl variants don't really need to warm up at all).

@René: I'd like to do that as well, but am not...

2011-12-16T10:02:26.352-05:00
@René: I'd like to do that as well, but am not currently setup to do so (and need to get back to my "day job"!). As noted above, if anyone has the time to deal with .NET or Flash, please feel free to do so and send a pull request to the github repo.