Thursday, June 3, 2010

newLISP - Lisp for the masses

There is a popular phrase among Lisp hackers: Plant a tree, write a book, create your own dialect of lisp. Although there aren't many popular Lisps out there and even the mainstream Common Lisp has never been massively used, it seems that just like in case of various Linux distributions, often more means simply better. A good example of such success story is Clojure, and here comes another candidate to take the lead.
newLISP is a modern dialect of Lisp, designed by Lutz Mueller to be (as he says himself) "quick to learn and to get the job done". I must say that this sentence couldn't be more true - solving Euler Problem 10 (finding the sum of all primes below 2 million) after just two days of fiddling with newLISP took me less than 3 minutes, including designing, writing, testing and executing the following code:
(println (apply + (filter (fn (n) (= 1 (length (factor n)))) (sequence 2 2000000))))
In spite of being an interpreted language, programs created with newLISP run amazingly fast. The code above is a purely brute force solution, yet it executes in less than 10 seconds on Core 2 Duo 1.66GHz.
However, simplicity comes for a price. If you try to use a more sophisticated approach, like classic sieve of Eratosthenes, you might get a bit surprised:
(define (sieve seq out)
  (let ((n (first seq)))
    (setf seq (filter (fn (x) (!= 0 (mod x n))) seq))
    (push n out)
    (if (not seq) out (sieve seq out))))
(print (apply + (sieve (sequence 2 2000000))))
In this code function sieve, although properly tail recursive, will make newLISP to quickly run out of stack or (if you provide enough space for the stack) of all available memory. It's because newLISP does no tail recursion optimization. If for some reason you cannot live with it, you can still use Common Lisp for implementing such recursions:
(defun range (min max) (loop for i from min to max collect i))
(defun sieve (seq &optional out)
  (let ((n (car seq)))
    (setf seq (delete-if #'(lambda (x) (= 0 (mod x n))) seq))
    (push n out)
    (if (not seq) out (sieve seq out))))
(print (apply #'+ (sieve (range 2 2000000))))
As you can see, the code of function sieve is very similar, so it's fairly easy to switch to newLISP if you already have some Common Lisp background. Differences to other Lisp dialects are well documented, as well as the language itself. Documentation is another strength of newLISP: you can learn how to solve different real life problems using newLISP code patterns or browse through many interesting code snippets.
What I personally like about newLISP in comparison to other Lisps is its really tiny footprint. You can create a standalone executable containing an embedded newLISP engine which is 200kB small, using two simple steps:
(load "/usr/share/newlisp/util/link.lsp")
(link "/usr/bin/newlisp" "mycode.bin" "mycode.lsp")
Despite being so small, newLISP provides a surprising amount of functionalities out of the box: regular expressions, TCP/IP networking (including FTP and HTTP protocols), database access (through external libraries), OpenGL, XML and XML-RPC handling, matrices, statistics (including Bayesian formulas), unicode support and a set of C/C++ modules that extend its abilities even more.
newLISP also supports parallel processing through Cilk-like API, and distributed computing through a built-in function net-eval.
newLISP is definitely not a New Common Lisp, and in some points (such as the beforementioned tail recursion) is still inferior. But newLISP is a perfect example that in the IT industry sometimes worse is better.

4 comments:

Kazimir Majorinc said...

Nice review article. I hope if you do not mind if I add one thing I consider to be important advantage of Newlisp, and you didn't mentioned: Newlisp insists on "code=data" idea more than most non-experimental, actively maintained Lisp dialects.

(1) Because of few reasons, Newlisp make much more use of eval than most other Lisp dialects. One of the reason is that its eval is fast, only Eligis Open Lisp and Picolisp have even faster one.

(2) Newlisp has dynamic, and not lexical scope. Dynamic scope works better with eval. For example,

(let((x 1))(eval 'x))

is legal in dynamic scope, and returns "x not defined" error in lexical scope. Emacs Lisp, Picolisp, and Newlisp support dynamic scope, CL supports both, other dialects only lexical scope.

(3) Newlisp macros are actually FEXPR's; i.e. something like first-class, runtime defined macro. Unlike macros, FEXPR can be assigned as values, applied, mapped and used as data during runtime. FEXPRS existed in early implementations of Lisp, but were discontinued because supposedly make some compiler optimizations impossible. That claim is, I believe, misleading, but it is irrelevant for interpreted languages on the first place. No other current Lisp dialect support fexprs. (Newlisp also supports reader macros and some traditional macros - but through library.)

(4) Unlike macros, FEXPRS cooperate nice with eval, so if program combines (fexprs or macros) and eval, it is likely that it will be much faster in Newlisp than in other Lisp dialects.

(5) Functions and macros (i.e. FEXPRS) in Newlisp are *expressions*, not the results of the evaluation of expressions. So, these can be (including anonymous) analysed and mutated during runtime.

References: [1], [2].

--

Bill Six said...

"If for some reason you cannot live with it, you can still use Common Lisp for implementing such recursions"

I don't get this sentence. Are you saying the Common Lisp standard mandates tail call optimization?

kklis said...

Not the ANSI standard itself, but I don't know any ANSI compilant implementation that doesn't support tail recursive call optimization.
The same is with ANSI C, which does not define single line comments, only block comments. Single line comments are C++ feature, yet practically all C compilers support it.

Unknown said...

« You can create a standalone executable containing an embedded newLISP engine which is 200kB small, using two simple steps:

(load "/usr/share/newlisp/util/link.lsp")
(link "/usr/bin/newlisp" "mycode.bin" "mycode.lsp") »


Now, it's : newlisp -x filename.lsp filename(.exe) in command line