Wednesday, March 16, 2011

Is JavaScript Faster Than C?

There's been a lot great work happening in the VM performance space over the last few years. The problems of performance are beginning to be well understood as even dynamic languages begin to challenge the incumbents. This article reviews a project which aims to bring more empirical testing to the language/runtime performance debate. Rather than argue about the theorectical, we let the code speak for itself. You may be suprised by what it says.

Stop!


The author of this article is well aware of its provocative nature. This is just one test and readers should take care when extrapolating these results to dissimilar use cases. The C code provided by the project being reviewed follows an Object Oriented style. It is certainly possible to optimize the C code further, for instance by using a more procedural style with less memory allocation. But then a similar approach should be used by the other versions (languages) presented as well. I whole heartedly encourge you to submit pull requests with these new versions in separate directories based on the style/technique used.

My intention is to challenge readers to question their assumptions regarding language/runtime performance. I also hope to highlight the fact that server side JavaScript has become a viable platform choice when performance is a consideration. That said, this is one test and I can only encourage more work in this area. If you're interested in such things, check out The Computer Language Benchmarks Game. Let's discuss results rather than argue fanatically about predictions.

The Problem



Solving 137846 games of the 15-Hole Peg Soltaire by calculating every possible move using an Object Oriented style. Solutions are provided in C, C#, Java, JavaScript, PHP, Python and Ruby (more submissions encouraged!). Most of the code was written by Jonathan Fuerth his presentation on Java/HotSpot performance. I've updated the JavaScript so that it uses prototypical method inheritance. You can find my 'fork' on github.

The Code


    git clone git://github.com/gflarity/peg-performance.git


Pre-requisites


This was all developed and tested under OS X however there shouldn't be any platform dependencies. You'll need a number of things for the test itself. I've put the version I'm using in brackets: gcc (4.2.1), Mono (2.10.1), Java (1.6.0_24), node.js (0.4.2), PHP (5.3.3), Python (2.7.1), Ruby (1.8.7).

Run the tests!


Note this will take a few minutes as some platforms are definitely faster than others.
    cd src/main
make test

For each of the languages the test is run 10 times. I throw out the best and worst result then compute an average. A chart is created to show the results.

Results


Results can be viewed by opening .reports/report.html.

Here is the graph produced by this test on my MacBook Air:


Analysis


Here's what I find interesting about these results:

0) The JavaScript implementation is even faster if you load the browser version in Chrome 10. It clocks in around 2300ms on my mba.

1) HotSpot Java is F.A.S.T., FAST. I'd really like to see a C++ version for comparison in this test. But given the ecosystem out there and the fact that it's automatically cross platform, I'd have hard time justifying the use of C++ for many things these days... Well, maybe a kickass JavaScript engine ;)

2) I'd also like to see the results of this test if this problem was solved in a non object oriented style. That said, there are distinct advantages to using OOP.

3) Much respect to the Mono project. While I'm sure Microsoft's .NET runtime approaches the HotSpot VM in terms of performance, Mono is hanging in there.

4) JavaScript is a dynamically typed language. We must consider this when comparing its performance vs a statically typed Java. While some might argue that static typing is 'safer', performance is its only virtue in my opinion. (Edit: Static typing also helps IDEs with code complete and other useful features, this is also a virtue :) Typically I trade reduced performance for increased development efficiency every time I choose Perl, Python or Ruby. So despite its quirks, it appears that JavaScript can offer the best of both worlds in terms of performance and development efficiency. Not bad for the language that's in every browser and hence is the most widely deployed language/runtime in the world.

18 Comments:

At March 16, 2011 at 9:52 AM , Blogger Arseny Kapoulkine said...

Congratulations! You've successfully proven that Java is, like, 4x times faster than C. A milestone to speak of, well done! It's a pity that most of the desktop software is written in C/C++, even JavaScript would've been a better choice!

 
At March 16, 2011 at 10:44 AM , Blogger Geoff Flarity said...

Well, this only one test, and C is not known for being an Object Oriented language. Still it's something to think about.

 
At March 16, 2011 at 3:18 PM , Blogger elephantum said...

I've optimized a bit your python solution. Now it takes around 5s instead of 30s to run.

You can take a look if you want: https://github.com/elephantum/peg-performance/tree/master/src/main/python

 
At March 17, 2011 at 7:03 AM , Blogger Geoff Flarity said...

Thanks, it's actually not my solution, but jfeurth's. I think I'm going to add an anything goes directory and then merge your stuff there.

The main directory should remain object oriented and the problem should be solved the same way for each language. Some of changes actually moves away from being object oriented and changes how the problem is solved. Ie using a generator to remove the memory pressure of keeping the lists of moves, and using hashes to lookup pegs in the game state.

But thanks, I'm learning a lot :)

 
At March 19, 2011 at 8:51 AM , Blogger softboysxp said...

I implemented a C++ version and it's just slightly faster than Node on my Snow Leopard iMac

https://github.com/softboysxp/peg-performance

( I did turn on optimizations, as IMO the node engine was built with O2 or O3, it's fair to turn them on for the C/C++ version)

Another interesting observation is that on my Lion MacBook Pro, which has LLVM gcc/g++ by default, the C++ version's performance is greatly improved (probably due to optimization) and is almost 3x faster than node (again, node was built with LLVM gcc)

 
At March 19, 2011 at 10:35 AM , Blogger Geoff Flarity said...

Sweet! WIll checkout the C++ version and try to merge it.

 
At March 19, 2011 at 10:56 AM , Blogger Geoff Flarity said...

It's been a long time since I looked at C++, but it seems your version isn't freeing the memory you allocate?

Java and JavaScript have GC, so this comes for free. If I can find some time this week I'll look into adding some delete statements.

 
At March 19, 2011 at 1:58 PM , Blogger softboysxp said...

I intentionally avoided object creation with new in my implementation. so the objects should be allocated on the local stack instead of the heap and should be automatically destroyed when they go out of scope.

 
At March 19, 2011 at 3:35 PM , Blogger elephantum said...

@softboysxp by creating objects on stack you introduce massive memory copying, are you sure that this is faster than dealing with pointers and avoiding copying?

 
At March 20, 2011 at 12:40 AM , Blogger softboysxp said...

@elephantum thanks for the tip, I guess I need to find out about that, :) my current logic is to follow the Java version as close as possible for the comparison.

 
At March 22, 2011 at 2:04 PM , Blogger Isaac Gouy said...

Which GC does the Mono program use? --gc=sgen?

What's the C compiler commandline? Which commandline flags did you use?

Which JVM flags did you use?

 
At March 31, 2011 at 6:40 AM , Blogger Geoff Flarity said...

@isaac:

Take a close look at the legend ;)

Otherwise, try looking at the source.

 
At August 26, 2011 at 11:56 PM , Blogger Jacob Whitley said...

Hi,
In a production environment, the amount of RAM each session uses is quite big (there is a MIME encoder/decoder which eats something like 10x the size of the message and we receive 10M+ messages), which makes it a problem.
Thanks

 
At August 26, 2011 at 11:57 PM , Blogger Jacob Whitley said...

Javascript Countdown Timer

 
At October 14, 2011 at 10:37 PM , Blogger madhuri said...

Ofcourse, Javascript had many advantages in advance compared to C.U have made a best post on its importance.
web design company

 
At May 23, 2012 at 11:23 AM , OpenID fluffy said...

I just did a trivial port of the Java version to C++11, the only change being that I use a std::unordered_set (nee std::ext::hash_set) instead of a List for the gamestate contents. On my system, the Java version runs in 672ms, and this C++ version runs in 621.

I have a git repo for this at http://beesbuzz.biz/git/peg-performance.git

 
At May 23, 2012 at 11:27 AM , OpenID fluffy said...

Also, why would you not want to use hashes to store the game state? It's only fair to make use of associative containers that the language provides, as that is what real programmers will be using for arbitrary object lookup. Why hobble the solution by doing an O(n) lookup needlessly?

 
At May 24, 2013 at 1:31 PM , Blogger sanjay said...

Appreciation for nice Updates, I found something new and folks can get useful info about BEST ONLINE TRAINING

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home