newLISP - Lisp for the masses

There is a popular phrase among Lisp hackers: Plant a tree, write a book, create your own dialect of lisp. Although there aren't many popular Lisps out there and even the mainstream Common Lisp has never been massively used, it seems that just like in case of various Linux distributions, often more means simply better. A good example of such success story is Clojure, and here comes another candidate to take the lead.
newLISP is a modern dialect of Lisp, designed by Lutz Mueller to be (as he says himself) "quick to learn and to get the job done". I must say that this sentence couldn't be more true - solving Euler Problem 10 (finding the sum of all primes below 2 million) after just two days of fiddling with newLISP took me less than 3 minutes, including designing, writing, testing and executing the following code:

(println (apply + (filter (fn (n) (= 1 (length (factor n)))) (sequence 2 2000000))))

In spite of being an interpreted language, programs created with newLISP run amazingly fast. The code above is a purely brute force solution, yet it executes in less than 10 seconds on Core 2 Duo 1.66GHz.
However, simplicity comes for a price. If you try to use a more sophisticated approach, like classic sieve of Eratosthenes, you might get a bit surprised:

(define (sieve seq out)
  (let ((n (first seq)))
    (setf seq (filter (fn (x) (!= 0 (mod x n))) seq))
    (push n out)
    (if (not seq) out (sieve seq out))))
(print (apply + (sieve (sequence 2 2000000))))

In this code function sieve, although properly tail recursive, will make newLISP to quickly run out of stack or (if you provide enough space for the stack) of all available memory. It's because newLISP does no tail recursion optimization. If for some reason you cannot live with it, you can still use Common Lisp for implementing such recursions:

(defun range (min max) (loop for i from min to max collect i))
(defun sieve (seq &optional out)
  (let ((n (car seq)))
    (setf seq (delete-if #'(lambda (x) (= 0 (mod x n))) seq))
    (push n out)
    (if (not seq) out (sieve seq out))))
(print (apply #'+ (sieve (range 2 2000000))))

As you can see, the code of function sieve is very similar, so it's fairly easy to switch to newLISP if you already have some Common Lisp background. Differences to other Lisp dialects are well documented, as well as the language itself. Documentation is another strength of newLISP: you can learn how to solve different real life problems using newLISP code patterns or browse through many interesting code snippets.
What I personally like about newLISP in comparison to other Lisps is its really tiny footprint. You can create a standalone executable containing an embedded newLISP engine which is 200kB small, using two simple steps:

(load "/usr/share/newlisp/util/link.lsp")
(link "/usr/bin/newlisp" "mycode.bin" "mycode.lsp")

Despite being so small, newLISP provides a surprising amount of functionalities out of the box: regular expressions, TCP/IP networking (including FTP and HTTP protocols), database access (through external libraries), OpenGL, XML and XML-RPC handling, matrices, statistics (including Bayesian formulas), unicode support and a set of C/C++ modules that extend its abilities even more.
newLISP also supports parallel processing through Cilk-like API, and distributed computing through a built-in function net-eval.
newLISP is definitely not a New Common Lisp, and in some points (such as the beforementioned tail recursion) is still inferior. But newLISP is a perfect example that in the IT industry sometimes worse is better.

Web Sockets: a new era for the Web

The Web Sockets API is a part of HTML 5 specification and is to the Web what TCP is to the IP protocol. It allows full-duplex, bidirectional communication between the server and the client browser - no more polling, no more busy waiting, and no more problems with keep-alive HTTP connections. Web Sockets allow to leverage Web applications to an absolutely new level, where they can finally operate like any other network software, without crippled overlays like AJAX or Comet. With Web Sockets you can push data to the client just as you would do it with XMPP.
One of the first browsers to support Web Sockets is Google Chrome. And, of course, one of the first languages natively supporting Web Sockets is Go. Here is a simple server application which sends time information to the browser in one second intervals:

package main

import (
  "http"
  "io"
  "strconv"
  "time"
  "websocket"
)

func ClockServer(ws *websocket.Conn) {
  ch := time.Tick(100000000)
  t1 := <- ch
  t := t1 / 1000000000
  for {
    t2 := <-ch
    td := (t2 - t1) / 1000000000
    if td != t {
      io.WriteString(ws, strconv.Itoa64(td))
      t = td
    }
  }
}

func main() {
  http.Handle("/clock", websocket.Handler(ClockServer))
  err := http.ListenAndServe(":12345", nil)
  if err != nil {
    panic("Error: ", err.String())
  }
}

When you compile and start the server, create a web page with the following content and save it to your disk:

<html>
<head>
<title>WebSocket</title>
<script type="text/javascript">
function init() {
  var tick = document.getElementById("tick");
  if ("WebSocket" in window) {
    var ws = new WebSocket("ws://localhost:12345/clock");
    ws.onmessage = function (evt) {
      tick.innerHTML = evt.data;
    };
  } else {
    tick.innerHTML = "The browser doesn't support WebSocket.";
  }
}
</script>
</head>
<body onload="init()">
<span>Seconds elapsed: </span><span id="tick">0</span>
</body>
</html>

If you open the page in Google Chrome, you will see seconds ticking. There is no AJAX, no long-polling or any other fancy stuff - the server pushes data directly to the client. If you open more Chrome instances, you will see that each of them has its own connection with the server with its own clock (however, beware opening many connections in the same window, but in different tabs - this can occasionally crash your browser).
You can also use Erlang to make use of Web Sockets technology. Joe Armstrong has recently posted an article on his blog with full source code of both Web Sockets client and server.
Have fun!

Go memcache client package

Recently I needed to access Memcached from Go. I couldn't find a suitable package anywhere on the web, so I created one. Gomemcache provides basic operations to store, retrieve and delete data using memcache text protocol. You can download the package from its Github repository.

Edit: Gomemcache is now distributed under the terms of LGPL license with static linking exception. It means that you can link it statically with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you meet the terms and conditions of LGPL for the package itself. Originally GNU Lesser General Public License does not allow unproblematic static linking with proprietary source code.

Go Programming Language Resources

Go is a fairly new programming language, so at the moment it is hard to find interesting projects associated with it. Go Programming Language Resources is a web site that tries to gather them in one place. It also contains links to mailing lists, discussion groups and IRC archives, as well as Go ports to different operating systems. You can also find there a few interesting development tools and syntax highlighting for the most popular programmer's editors. If you are interested in Go, this site is definitely worth adding to your bookmarks.

Go - a new programming language from Google

Go is a new programming language developed at Google, which according to its FAQ "was born out of frustration with existing languages and environments for systems programming". Some people ask if the world needs another programming language, but those who know that among Go authors are Ken Thompson and Rob Pike, famous Unix hackers, usually don't. If there is a language that has a chance to replace plain C in system programming, Go is a perfect candidate. It features a syntax derived from the C tree (which makes learning curve fairly easy for most of the programmers), fast compilation to native machine code, and fast execution of compiled binaries. Additionally, Go provides a built-in garbage collector and language constructs that simplify parallel programming, especially the concept of goroutines, which are regular program functions executed concurrently. Goroutines can communicate with each other and the main thread through channels, that can also be used for synchronization purposes.
I have prepared a few simple programs to compare Go with C in terms of speed and to play with concurrent programming in Go. First, let's have a look at a typical recursive Fibonacci example. Below is a C version:

#include <stdio.h>
#include <stdlib.h>

int fib(int n) {
  if (n < 2) {
    return(n);
  }
  return(fib(n-2) + fib(n-1));
}

int main(int argc, char *argv[]) {
  int n = atoi(argv[1]);
  printf("%d\n", fib(n));
  return(0);
}

and here is a Go version:

package main

import (
  "flag"
  "fmt"
)

var f = flag.Int("f", 1, "Fibonacci number")

func fib(n int) int {
  if n < 2 {
    return n
  }
  return fib(n-2) + fib(n-1)
}

func main() {
  flag.Parse()
  fmt.Println(fib(*f))
}

A quick test shows that a single threaded Go program is actually faster than the C one:

$ gcc -O2 -o fib fib.c
$ time ./fib 40
102334155

real 0m1.987s
user 0m1.980s
sys 0m0.004s

$ 8g fib.go; 8l fib.8
$ time ./8.out -f=40
102334155

real 0m1.934s
user 0m1.932s
sys 0m0.004s

I also prepared a program in Go to calculate a sum of subsequent Fibonacci sequences up to a given number in parallel. It uses run function as a goroutine to calculate each sequence independently and a shared channel ch to gather the results, that are finally summed up (so we don't care about the order in which they appear in the channel):

package main

import (
  "flag"
  "fmt"
  "runtime"
)

var n = flag.Int("n", 1, "Number of CPUs to use")
var f = flag.Int("f", 1, "Fibonacci number")

func fib(n int) int {
  if n < 2 {
    return n
  }
  return fib(n-2) + fib(n-1)
}

func run(n int, ch chan int) {
  ch <- fib(n)
}

func main() {
  flag.Parse()
  runtime.GOMAXPROCS(*n)
  ch := make(chan int)
  for i := 0; i <= *f; i++ {
    go run(i, ch)
  }
  sum := 0
  for i := 0; i <= *f; i++ {
    sum += <-ch
  }
  fmt.Println(sum)
}

The program takes additional parameter -n to indicate the number of CPU cores to use. According to Go runtime package documentation the call to GOMAXPROCS is temporary and will go away when the scheduler improves. Until then you have to remember to use this call, otherwise your application will use only one CPU core by default.

I ran the program for 40 Fibonacci sequences on dual Intel Xeon L5420 2.50GHz using from single up to all available CPU cores. The execution time improved most dramatically between -n=1 (5.073s) and -n=2 (3.141s), than it gradually slowed down from -n=3 (2.574s) to -n=8 (2.013s).

What I like about Go is that it gives a set of powerful tools into programmer's hands, but at the same time does not try to hide the complexity of parallel programming behind bloated libraries or awkward language constructs. It nicely follows the KISS principle and borrows some good ideas from Unix design (like channels, which work similar to Unix pipelines). If you think seriously about future system programming, I think Go is definitely a language worth learning. Not only because it's Google ;-)

MagLev public alpha

Gemstone has just announced an alpha version of their concurrent Ruby engine, MagLev, available for download. MagLev is based on Gemstone's Smalltalk virtual machine and supports 64-bit Linux, Mac OS X and Solaris x86 operating systems. There are no plans for 32-bit version of MagLev.
MagLev does not support Rails yet, but so does not Fabio Kung's JMagLev. However, the advantage of MagLev over Fabio's machine is that Gemstone is determined to create an enterprise-class product, and JMagLev was just a demonstration of the power of Terracotta and does not seem to be developed any further. It seems that the next step for Gemstone will be to implement Rails functionality and allow RoR applications to run in a clustered enviornment, just as Grails ones can run on Terracotta.

Erlang project crawler

Today I received an email from Erlang Training and Consulting Ltd. - the owner of popular Erlang Community Site Trapexit - announcing its own Erlang open source project crawler. Crawler gathers information on open source Erlang projects from a number of code repositories such as GitHub, Bitbucket, SourceForge and Google Code. At the time when I am writing this post it includes information on 1228 projects. The number may not be impressive, but it is good to have information about the most interesting open source Erlang projects gathered in one place.