The Urn Logo

Latest posts

Version 0.6.1 released

Normally I start these update posts with some comment about how much has changed, or how long it’s been since the last one. Sadly, I can’t think of anything to say so let’s just jump right into some key changes:

Method improvements

Methods are a really powerful way of writing functions which can handle all sorts of data types. That being said, they are not without their problems and so we’re always working to improve them. One issue is the large increase in code size that using methods results in. In Urn 0.6.1, we’ve tried to lower that overhead, resulting in even smaller files!

Alongside with that, we’ve also added the ability to specify wrappers within methods. Every call to your method will go through this delegate, allowing one to provide specialist logic. This is used by the eq? method to check using the == operator before trying anything else.

(defgeneric eq? (x y)
  "Compare values for equality deeply."
  :delegate (lambda (x y)
              (if (= x y)
                true
                (myself x y))))

Code generation and optimisation tweaks

Interestingly enough, method delegates exposed several places where code generation could be further improved. One such improvement was forcing bindings to be generated as local definitions rather than directly called functions. This makes code more readable and potentially allows further improvements as statements can be emitted more sensibly. For instance, consider the following code:

(print!
  (with (foo (if (empty? '()) 1 2))
    (+ 1 foo)))

here’s how it would be compiled on 0.6.0:

return print1(1 + (function(foo)
        return foo
end)((function()
        if empty_3f_1({tag="list", n=0}) then
                return 1
        else
                return 2
        end
end)()))

and here’s the equivalent code on 0.6.1:

return print1(1 + (function()
        local foo
        if empty_3f_1({tag="list", n=0}) then
                foo = 1
        else
                foo = 2
        end
        return foo
end)())

Another interesting optimisation we now apply is lowering “deferrable” variables. For instance, consider this definition:

(with (foo '())
  (when (empty? '())
    (debug foo)))

The definition of foo is only needed if then when block is executed, and so we’re able to “push” the definition down:

(when (empty? '())
  (with (foo '())
    (debug foo)))

Whilst this code doesn’t appear much in practice (or at least in this overly simplistic form), it’s nice to catch the few occasions where it does.

Version 0.6.0 released

Wow, it’s been a while since the last update. During that time, we’ve been doing a lot of work on the standard library. So let’s dig right in an see what’s changed.

Standard library restructuring

Urn’s standard library has always been a bit of a mess, with features scattered everywhere. Whilst some chaos is inevitable due to having to bootstrap the library, there was still lots of improvement. Sadly, deciding on a good approach proved more complicated than expected.

For Urn 0.6 we finally went about this restructuring, splitting Urn into several modules such as core (for bootstrapping the stdlib), data (data manipulation libraries) and test (used for writing tests). It’s worth noting that some programs may need changing to work with the new layout, but rest assured we won’t be changing it again any time soon.

Extended math library

Lisp has always had a history of numerical computing, and Urn does not disappoint. 0.6 adds support for rationals, complex numbers, vectors and matrices. It’s now that much simpler to do your maths homework in Urn!

In order for these features to work fluidly together, we’ve added a collection of generic arithmetic methods, such as n+ and n* which operate on arbitrary data structures. These methods, along with rational literals, bring a lot to the table.

> (import math/vector ())
> (n* (vector 1 2 3) 2/3)
out = [2/3 4/3 2/1]

String formatting changes

hydraz has done a lot of work on developing more powerful ways to format strings. Whilst string/format is useful, it can still be tedious to use. Thankfully the data/format library accepts format strings which neatly integrate with pretty printing, as well as allowing keyword arguments and interpolation of symbols in the current scope. I’d really recommend checking out the documentation for more information.

> (format nil "0x{foo%x}" :foo 123)
out = "0x7b"

Internal changes

Whilst the obvious changes have occurred within the libraries, the compiler’s internals have also changed. Aside from improved codegen, line mappings are now substantially more accurate. Consequently code coverage and error messages are more informative - always a perk when debugging!

Version 0.5.1 released

We’ve just released another Urn update - 0.5.1. This is a relatively small update, mostly composed of bug fixes and performance improvements. That’s not to say there aren’t some useful new features though!

REPL improvements

The Urn REPL is a great tool for testing and prototyping code. It’s something we regularly use, and so are always eager to improve it.

One task we often find ourselves doing is loading up the REPL, importing a module and testing a small feature in it. This process has now been streamlined: when using the --repl flag, any file provided on the command line will be automatically imported into the current scope.

We’ve also added a :view command, which allows you to preview a symbol’s definition. This saves you switching between source code and the REPL, trying to remember how you got that awesome feature to work.

Shows the result of :view command

Optimised optimiser

The Urn compiler has never been the fastest beast, with the optimiser being the worst offender. Whilst the optimiser is great at reducing the size (and speed) of generated code, it also takes considerable time to run. Even worse, much of the optimiser’s time was just spent in iterating over every node - not even running the various passes! Something had to be done.

Urn 0.5.1 introduces a new framework for optimisations, doing something so obvious I’m surprised we weren’t doing it from the start. Instead of running each pass sequentially, we have one unified visitor object which will call each pass for every node it hits. Not only does this substantially reduce the visitor overhead, it reduces the number of times we have to run a pass, resulting it in an even bigger performance increase! We’ve also made some improvements to definition and usage tracking, which ensures that the information is more up-to-date, and so the optimiser can make smarter decisions.

These various improvements have resulted in a 2-3x performance increase in the optimiser (depending on Lua version and implementation). In fact, it now takes less time to optimise the compiler than it does to load it. There is evidently room for performance improvements elsewhere!

Version 0.5.0 released

Updates my dears, and not a moment too soon. It’s been 100 commits since our last release, and each commit is packed with lispy goodness. OK, maybe not every commit. In this time, we’ve reached 1000 commits which, whilst a rather arbitrary milestone, it still a milestone we can celebrate. There’s been more changes than I can cover here, so I’d really recommend checking out the release nodes for the whole picture. Anyway, here’s a small summary of what’s new:

Multimethods (aka defgeneric)

Whilst Urn prides itself on the extensible power that macros offer, the extensibility of the standard library has historically been lacking. There are several functions which do a long if-elseif chain of type comparisons, with a final metatable lookup if all else fails. This is obviously undesirable and so something needed to be done. Enter multimethods.

If you’ve used an object oriented language such as C++ or Java then you will have used methods before, and so multimethods will not be an entirely alien concept. However, instead of dispatching based on the type of the first argument, it uses all arguments to determine which function to call. This makes multimethods a much more powerful solution.

In order to demonstrate their power, we’ll create a generic merge method. This’ll attempt to merge two objects together and return the resulting object. First we define a new method with defgeneric, providing our argument names and a docstring:

(defgeneric merge (x y)
  "Merge X and Y together, returning a new object.")

We can now call merge, but we’ll just get an error. After all, we haven’t defined any implementations yet. This is done with defmethod, specifying the argument types and an implementation. Let’s do a couple of simple ones:

(defmethod (merge string string) (x y) (.. x y))
(defmethod (merge list list) (x y) (append x y))

Now let’s use these:

> (merge "foo" "bar")
out = "foobar"
> (merge '(1 2 3) '(4 5 6))
out = (1 2 3 4 5 6)

Of course, if you’ve got a fancy structure which can be merged, its trivial to add support for that too. We’ve made great use of this feature in the standard library, converting pretty and eq? to use it.

I’d really recommend reading hydraz’s blog for more information about this feature. He’s the one who wrote it, and goes into much more detail about the implementation and rational behind it. Now’s when I’d normally insert a cynical comment about no-one getting this far, but it’s late and there are plenty more features to cover…

Fancy structures

One of the gems of Lua is it’s table. This simple data structure forms the base of many of Urn’s standard types. However, using passing around tables and indexing them via strings does rather take the idea of abstraction and smash it on the floor. We’ve written the urn/struct library to provide a more intuitive way to create your own data structures.

This library provides the defstruct macro. This provides a small DSL which allows you to declare fields, constructors and more. This will, in turn, generate the appropriate constructor, getters, setters and pattern matching utilities. Sounds great? Well let’s jump in and declare the struct staple - a point.

(defstruct point
  (fields
    (immutable x)
    (immutable y)))

(defmethod (pretty point) (p)
  (string/format "(x=%d y=%d)" (point-x p) (point-y p)))

This structure definition will create a constructor (make-point) and a type predicate (point?). Each field definition will specify an argument to the constructor and a getter. If the field is marked as mutable, then a setter will also be generated. We also chuck in a pretty implementation for good measure.

Now we can start constructing points, indexing their fields and printing them:

> (point 1 2)
out = (x=1 y=2)
> (point-x (make-point 1 2))
out = 1
> (point-y (make-point 1 2))
out = 2

If you don’t like these function names, then customising them is trivial. One can also make particular fields hidden package-local, if you have implementation specific fields. See the documentation for more information.

Testing, coverage and stability

All these new features a awfully nice, if we don’t have any guarantees they actually work correctly. That’s why we’ve spent a lot of time getting our testing infrastructure up to date. I won’t go into the details as frankly, it isn’t that interesting but here’s a couple of highlights:

Example testing

The Urn documentation is full of example usage for various functions, complete with inputs and outputs. These examples are perfect for a trivial test case, to ensure the basic functionality is there. To make best use of these examples, we’ve written a compiler plugin which extracts codeblocks from docstrings and runs them, verifying the output is as expected.

Whilst I’m not proposing doing away with normal tests - after all, you don’t want to include every edge case in an example - they are a small step in making sure Urn is fully tested. Furthermore, it allows us to ensure all examples are up to date, and provides an incentive to write more documentation - always a plus.

Code coverage

Tests are great for seeing what works, but it’s much harder to find what doesn’t work. One way to help with this is to determine what the test suite doesn’t check. And so, of course, we’ve written a small code coverage utility for Urn. It generates files in the same format as LuaCov, and so provides some interpobility with other Lua-based tools. To get started, just run your code with the --profile=coverage switch.

Codegen and optimiser improvements

Normally I’d bang on about the most exciting optimisation changes this release. However, for better or for worse, there haven’t been many major changes. Instead, there’s been a lot of work on stabilising the various systems, resulting in a less buggy and more consistent experience. Whilst it’s far from perfect, we’re one step closer to not blowing up in your face the whole time.

Version 0.4.4 released

Oh boy, it’s another Urn update! I know, I can hardly supress my excitment. There’s been almost 50 commits since the last release, and a whole host of features inside those commits. So let’s get started.

Goodbye setf!, hello lenses!

A while ago we introduced the collections library and the joy that is lenses. I won’t re-iterate what is said there, but in essence it provides a powerful and composable way to query and update objects (as well as making immutable copies of them). We’ve merged this feature into the main standard library, replacing the rather hacky setf!.

String interpolation

If you need to concatinate a load of strings together, there aren’t many nice ways. Sure, format strings help a bit but there is nothing quite as great as string interpolation! Prefixing a string with the dollar ($) sign (or calling the $ macro) allows you to embed variables directly in code:

> (let [(foo "some string")
.      (bar '(1 2 3))]
. $"Interpolating ${foo} and ~{bar}")

"Interpolating some string (1 2 3)"

Currently Urn’s implementation is very basic, just allowing variables (it’s implemented as a macro after all), but we’ve plans to extend it.

More code-generation improvements

Every release I witter on about the various code generation improvements we’ve made, and this time is no exception. I’ve been running the generated code through luacheck to try to find some places where we’re generating pretty poor Lua code. We’ve made a lot of progress in this area, cutting the number of warnings from 469 to 104 (which equates to a warning on 1.2% of lines).

This isn’t just a meaningless statistic though - the end result means more compact and “readable” code. Thanks to various enhancements, we’ve cut 300 lines from the previous release.