Install Graphite On A SmartMachine (SmartOS Zone)

I've been hearing a lot of good things about Graphite lately. So I thought I'd give it a try on a SmartOS Zone (SmartMachine?). I had one heck of a time installing it. Here's how I did it.

First, for some reason I had to use the 32bit version SmartOS 1.4.7. If you try the 64bit version, the first error you get:

Checking for header Python.h             : Could not find the python development headers

But this appears to be a red herring and the real issue is related to 32 vs 64bit issues. Investigating this helped me understand why people don't like waf. Just use the 32bit zone and safe some pain.


Install all the dependencies that are available via pkgin:

pkgin install apache gcc-compiler scmgit python27 pkg-config cairo xproto renderproto-0.11 kbproto py27-setuptools py27-django py27-twisted ap22-py27-wsgi

I ran into a problem where it said apache was missing when I tried to download it. This was fixed by doing 'pkgin -f update'. The '-f' was necessary.

Dependency - pycairo

Install pycairo, we need py2cairo as its for python 2.7:

git clone git://git.cairographics.org/git/py2cairo
cd py2cairo
PREFIX="/opt/local" ./waf configure
./waf build 
./waf install

Install Graphite components 'by hand'. Download, untar and run 'python setup.py install' in each of the resulting directories:

http://pypi.python.org/pypi/carbon
http://pypi.python.org/pypi/graphite-web
http://pypi.python.org/pypi/whisper

Install django-tagging by hand. Download, untar and run 'python setup.py install'.
http://code.google.com/p/django-tagging/downloads/list


Inside graphite-web run:

./check-dependencies.py

You might see warning, but there shouldn't see any FATAL messages.

(Under Construction) You still need to:
- copy and edit the apache config
- copy the .wsgi example into place
- copy the carbon.conf.example and the storage-schemas.conf.example
- patch the settings.py file so that it doesn't prepend the third party path

How Node.js Multiprocess Load Balancing Works

As of version 0.6.0 of node, load multiple process load balancing is available for node. The concept of forking and child processes isn't new to me. Yet, it wasn't obvious to me how this was implemented at first. It's quite easy to use however:

var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  // Fork workers.
  for (var i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('death', function(worker) {
    console.log('worker ' + worker.pid + ' died');
  });
} else {
  // Worker processes have a http server.
  http.Server(function(req, res) {
    res.writeHead(200);
    res.end("hello world\n");
  }).listen(8000);
}

This is quite a beautiful api, but it hides some clever details. For instance, how is possible for all child processes to each open up port 8000 and start listening?

The answer is that the cluster module is just a wrapper around child_process.fork and net.Server.listen is aware of the cluster module. Specifically, cluster.fork uses child_process.fork to create child processes with special variable set in their environments to indicate cluster was used. Specifically process.env.NODE_WORKER_ID is non-zero for such children.

 envCopy['NODE_WORKER_ID'] = id;
 var worker = fork(workerFilename, workerArgs, { env: envCopy });

Then net.Server.listen checks to see if process.env.NODE_WORKER_ID is set. If so, then the current process is a child created cluster. Instead of trying to start accepting connections on this port, a file handle is requested from the parent process:

//inside net.js
require('cluster')._getServer(address, port, addressType, function(handle) {

self._handle = handle;

self._listen2(address, port, addressType);

});

//inside cluster.js
cluster._getServer = function(address, port, addressType, cb) {
  assert(cluster.isWorker);

  queryMaster({
    cmd: "queryServer",
    address: address,
    port: port,
    addressType: addressType
  }, function(msg, handle) {
    cb(handle);
  });
};

Finally, this handle is listened on instead of creating a new handle. At which point you have another process that is listening on the same port. Quite clever, though I think the beauty of the api comes at the cost of some not so hideous hacks inside the api.

Why KVM on Illumos matters - ZFS

When Joyent first announced they had ported KVM to Illumos, the value proposition didn't occur to me right away. Both KVM and Xen under Linux have been around for while and you if want/need to run a Linux guest, why not use a linux Host? One reason is a feature of ZFS known as a zvol.

To understand what a zvol is, you must first understand what a zpool is. In short its an abstraction on top of one or many disks turning it all into a logical unit on which filesystems can be created. Multiple disks can be used for both redunancy and performance. In particular, you can create a zpool that uses a pair of super fast state of the art SSDs for the ZIL, or intent log. This makes random writes blazingly fast. You can add a couple of fast SSDs as a read caching layer (l2arc). Finally, use those boring old -but relatively high capacity- magnetic disks for raw storage. The result is a hybrid storage system that provides performance and redundancy at a terrific price.

The zpool is where the zfs filesystems live. So once you create a pool you can create a bunch of different file systems in there. You can also create and export block devices from a zpool, these are called zvols. These block devices can now be passed to qemu running on top of kvm inside Illumos, providing the underlying VM's with potentially blazingly fast redundant storage. A Linux VM will use this block device as if were a raw disk.

More generally, by using a zvol for a guest disk you can use all the ZFS features that are below the zfs filesystem layer. For instance it's possible to snapshot a live/attached zvol then export that snapshot to a file which can be used to create backups and clones. The implications for cloud/cluster management are clear.

Here's some other posts on zvols I found interesting:

http://www.markround.com/archives/37-ZFS-as-a-volume-manager.html

http://blogs.oracle.com/scottdickson/entry/fun_with_zvols_-_ufs

OpenSource Drobo replacement - Update

After over two years of faithful service, I've upgraded my OpenSource Drobo replacement. I thought it fitting to write an update post as my new setup is even better though mostly the same.

For perspective, not long after I setup this system, Apple discontinued work on the ZFS port to OS X. Thankfully the work they had done was solid. During the last 2 years I lost 2 drives and was easily able to replace them and get back to full speed within 24 hours of purchasing a new disk.

My new system consists of the same Mediasonic 4 bay ProBox headed by a Zotac ATOM based NetTop I purchased for around $300. You can buy a similar model now without ram or disk for under $150. I'd go for this one if I could go back in time. You can use a cheap USB stick and two gigs of ram for under $100 these days.

For the new head operating system I tried a couple of different Open Solaris/Illumos based distrobutions such as Smart OS, Open Indiana and Nextenta Core Platform. Unfortunately but unsurprisingly, these didn't seem to like my cheap hardware. I finally stumbled upon FreeNas, a FreeBSD based NAS appliance distribution complete with Web UI . Because it's FreeBSD based, support for ZFS is included. It's also designed to run from a USB stick in order to free up space for another drive which is a great idea.

Migrating to FreeNAS was a breeze once I learned to trust the Web UI. I still had to manually chown and chmod everything from a shell. Chown fixed the ownership for sharing everything properly in the FreeNas way. Chmod was required to fix an ACL problem with the filesystem caused my the OS X port. I also elected to upgrade to version 15 of ZFS which took all of two seconds (literally). It's possible I could have easily accomplished all this in the FreeNAS UI but old habits die hard.

Setuping up sharing was as easy as following the directions in the documentation. In short, create user, add user to group, configure the volume to be owned by that group with appropriate perms, create share, start/restart CIFS service. All in all it took about an hour!

Here's an update to the caveat's section from first post:

- you don't really need to know how to use a command line to get up and running with FreeNAS unless you're migrating coming from OS X ZFS
-esata does improve performance, at least it seems to though I've done no rigorous tests
-reboot the system, everything will come back online
- the green drive problem doesn't seem to be an issue anymore

Mobile HTML5/JavaScript Apps Part II

Challenges Throttling Adoption

In my previous article I outlined 10 reasons why the HTML5/JavaScript platform was the future of mobile app development. Adopting HTML5/JavaScript as a mobile development platform isn’t without it’s challenges however. In this article I outline the problems that are throttling the adoption of this platform.

The App Store Model

Presently there is no standard format for bundling HTML5/JavaScript apps for inclusion into app stores. This is disadvantageous for a number of reasons. Foremost, app stores help your customer’s find your app. They provide a platform integrated place where people can rank and recommend your app. It can help you grow your business.

Secondly, an app store also provides an integrated payment processing solution, often through cell phone billing. Sure they all take their cut and you could go the ad-supported route instead. But having a choice of models is one advantage the native app developers have. The seamless experience of well designed apps store is important.

Offline Support

Your customer’s expect to keep working when they enter the subway or fly on airplane. Native applications are bundled in to discrete packages and so capability typically comes for free. Traditional web applications host application logic remotely. As such, logic is transferred across the internet each time the app gets started or it even worse it remains on the server entirely.

In either case your customer is at the mercy of their mobile network which is bad for two reasons. The first is that the user experience will probably be slow or unusable, at least occasionally. The second is this architectural model causes a lot more data to be transferred. Mobile bandwidth isn’t cheap now that carriers are moving to plans with quotas.

HTML5 does provide a offline web app solution in the form of offline manifests. Anyone who’s ever tried to use it knows how rough around the edges it still is. Eventually they’ll smooth out those edges, but the in the meantime it’s painful to use.


The Solutions

In my next post I’ll describe the new technologies that are emerging to solve these problems. A perfect storm is converging to make HTML5/JavaScript the dominant platform for mobile applications. The future is bright and open source software is leading the way.

Mobile HTML5/JavaScript Apps Part I

I'm convinced that HTML5+JavaScript is the future of mobile development. That’s not to say there aren’t challenges throttling the adoption of this technology in the mobile space. But suffice it to say we feel the advantages of JavaScript are so compelling that critical mass has been achieved.

In my next article I’ll highlight the challenges facing mobile HTML5/JavaScript adoption. But first, here are the top 10 reasons why I believes HTML5+JavaScript is the future of mobile app development:

1) iOS (Apple)

2) Android ( Google )

3) BlackBerryOS ( RIM )

4) Windows Phone 7 ( Microsoft )

5) WebOS ( HP formerly Palm )

The first three mobile platforms each have close to a third of the market share. Yet their ‘native’ application platforms are all different. All of these devices have modern HTML5 browsers.

As a Mobile App Developer, you want to target as many different platforms and customers as you possibly can. At the same time, your app needs to look GOOD and it needs to be consistent across platforms. Finding the talent to target all these native platforms would be expensive.

6) JavaScript Is The Most Widely Available Development Platform, Ever.

There are over 2,000,000,000 internet users in the world now. Each uses a web browser, and that browser runs JavaScript. Eventually HTML5/JavaScript will work on every phone, laptop and desktop in the world.

7) You Can Still Sell HTML5/JavaScript Apps Through Regular App Stores

Great things are happening at fellow Canadian startup Nitobi. They’ve created an open source HTML5 app platform that allows you to author native applications with web technologies and get access to APIs and app stores. With PhoneGap it’s possible to write your application in HTML5/JavaScript, package it in a native wrapper, then distribute it through traditional app store channels.

8) JavaScript Is F A S T, Fast.

Google, Apple, Microsoft and Mozilla are all throwing cash and talent at making JavaScript faster. Techniques like JIT Compilation and Inline Caching are raising the bar on JavaScript performance. JavaScript is already fast and it will only get faster.

9) There Are *A Lot* Of JavaScript Developers Out There And The Number Is Growing

Chances are if you do Web development in PHP, Ruby, Python, Perl, Java etc, you’ve also implemented client side logic in JavaScript. That is, almost every web developer out there has at least dabbled in JavaScript. As the Web as a platform continues to gain traction, so does JavaScript. There’s a large pool of talent waiting to code in this language.

10) Windows 8

Even Microsoft is committed to supporting HTML/JavaScript as a core development technology. In a recent press release they describe Windows 8 as a re-imagining of the operating system. Here’s the part that stands out:

“Windows 8 apps use the power of HTML5, tapping into the native capabilities of Windows using standard JavaScript and HTML to deliver new kinds of experiences. These new Windows 8 apps are full-screen and touch-optimized, and they easily integrate with the capabilities of the new Windows user interface.”

No where in the press release do they mention .NET or Silverlight, technologies that Microsoft developed which compete directly with HTML5. So it seems that Microsoft is acknowledging that HTML5/JavaScript is platform to back if they’re going to retain developer mind-share.

A Fragmented Future United By Web Technologies

This post is in reponse to Peter Bright’s article on arstechnica.

Microsoft shocked many of it’s developer community when at D9 it announced HTML5/JavaScript would form the basis of Window’s 8 developer platform. Peter argues that saying nothing about the future of these technologies can only hurt Microsoft. However, I’m not sure Microsoft has much to lose and quite a bit to gain by spinning it this way. Here’s why.

Microsoft may rule the desktop now, but the future is mobile and they know it. Windows Phone 7 may be great and it’s apps may be excellent but they’re playing some serious catch up to iOS, Android and even Blackberry. Remember, Microsoft makes more off Android royalties than it does on Windows Phone 7. So while it’s true that Microsoft risks alienating their developer community, at present course that community will become smaller and more marginalized as time goes on anyways.

The truth is that if Microsoft wants be more than niche player in the mobile/tablet market, it needs to embrace HTML5. It’s just to far behind to expect to act as if it were Apple and try to force a proprietary platform. You can look towards the Financial Times launch of an HTML5 ‘iPhone’ app as model for the future here. Developing multiple high quality native applications is expensive and becoming increasingly unneccessary.

Now microsoft hasn’t said they won’t support their traditional platforms. No, far from it. Chances are there’ll be support for Silverlight as well. So while windows developers might be afraid right now, they’ve invested too much hold a serious grudge. Once Microsoft finally placates their fears they’ll suck it up and leverage the skills they’ve invested so much in.

By focusing the spotlight on their ‘choice’ of the only truly open cross-device app platform, Microsoft can been seen to be taking Windows in a bold new direction. Perhaps even buying themselves some creditability from the much larger web development community, many of whom still hold a grudge over IE 6. Microsoft has much to gain by spinning this decision as a bold and re-invention rather than reactionary concession. If that means temporarily upsetting their development community. They’ll get over it.