Latest posts

How time flies. It’s been almost three months since the last Urn release, and a whole year since the first one. I can’t say there’s one “super exciting” feature in this update, but instead there’s been many small improvements to the standard library and internals.

Configurable assertions

Often when I write Urn (or any other language for that matter), I find myself needing debug assertions. Namely, assertions which can be enabled at development time or disabled when I need to eck out every last bit of performance. Urn now has built-in support for this, thanks to the demand and desire macros.

Both of these macros take a value which must be upheld and, optionally, an error message which will be displayed if it is not.

> (defun negate (a)
.   (demand (number? a) "a must be a number")
.   (* a -1))

> (negate false)
[ERROR] demand not met: (number? a) (a must be a number).

stack traceback:
  lib/core/demand:29-31: in global 'demand-failure'
  <stdin>:2: in global 'negate'
  <stdin>:1 in main chunk

The difference between the two is when the assertions are enabled. demand is enabled by default, being disabled by passing -flax-checks to the compiler. Conversely, desire is disabled by default, with -fstrict-checks enabling such assertions.

Utilising a similar mechanic, one can also use -fstrict-structs to add typechecks to struct getters and setters. This will ensure the target is a struct of the expected type, preventing you from passing modifying unknown values.

You can enable all “strict” checks with the -fstrict flag, and disable all normal checks with -flax. Note that -flax arguments take precedence over their strict equivalents.

Rewritten native bindings

Interacting with Lua has always been a bit of a tricky thing with Urn. Whilst things have changed several times over the various Urn releases, we’ve never really hit a “sweet spot”. That being said, the latest redesign is substantially closer than we’ve got before.

Previously, one would define a .meta.lua file, which defined how various native definitions worked: whether they were some piece of Lua syntax, or a binding to an arbitrary Lua expression: whether they were some piece of Lua syntax, or a binding to an arbitrary expression. Whilst we are still sticking with this concept, all of these declarations have moved inline with the main define-native file.

This makes things a little cleaner (as we no longer need multiple files) and allows for dynamically generating definitions at compile time.

Compiler cleanup

As the Urn compiler is almost as old as Urn itself, most of it was written without access to some of the more recent features. One of the ways this is most apparent is how raw Lua tables and indexes are used for every structure, meaning you have no clue what (.> x :name) is referring to. Additionally, it makes it awfully easy to misspell field names, or set the correct field on entirely the wrong object.

This Urn release makes great strides in “sturdying-up” the compiler, converting many of the data structures to use defstruct. Not only does this provide better type safety, it also makes it much easier to read and write documentation on specific fields.

REPL reloading

One of the workflows I often rely on is loading files into the REPL, checking functions work as expected, and then modifying the files if not. Sadly, each time you change the file, you need to restart the REPL, meaning you have to start from scratch. This release adds the :reload (or :r) command to the REPL. This will scan all loaded modules, determine which ones have changed, recompile them. This means you can add or remove features without ever having to leave the comfort of Urn.

We’ve also rewritten the online REPL, using lua.vm.js to run the Urn compiler in the browser. This should make things more responsive (and removes the need to rely on friends’ servers). Many thanks to RawGit for their GitHub CDN service, which is used to fetch the standard library.

Do note that this is just a snapshot of the changes included in this release of Urn. Do check the full changelog or peruse the docs for more information.

Normally I start these update posts with some comment about how much has changed, or how long it’s been since the last one. Sadly, I can’t think of anything to say so let’s just jump right into some key changes:

Method improvements

Methods are a really powerful way of writing functions which can handle all sorts of data types. That being said, they are not without their problems and so we’re always working to improve them. One issue is the large increase in code size that using methods results in. In Urn 0.6.1, we’ve tried to lower that overhead, resulting in even smaller files!

Alongside with that, we’ve also added the ability to specify wrappers within methods. Every call to your method will go through this delegate, allowing one to provide specialist logic. This is used by the eq? method to check using the == operator before trying anything else.

(defgeneric eq? (x y)
  "Compare values for equality deeply."
  :delegate (lambda (x y)
              (if (= x y)
                true
                (myself x y))))

Code generation and optimisation tweaks

Interestingly enough, method delegates exposed several places where code generation could be further improved. One such improvement was forcing bindings to be generated as local definitions rather than directly called functions. This makes code more readable and potentially allows further improvements as statements can be emitted more sensibly. For instance, consider the following code:

(print!
  (with (foo (if (empty? '()) 1 2))
    (+ 1 foo)))

here’s how it would be compiled on 0.6.0:

return print1(1 + (function(foo)
        return foo
end)((function()
        if empty_3f_1({tag="list", n=0}) then
                return 1
        else
                return 2
        end
end)()))

and here’s the equivalent code on 0.6.1:

return print1(1 + (function()
        local foo
        if empty_3f_1({tag="list", n=0}) then
                foo = 1
        else
                foo = 2
        end
        return foo
end)())

Another interesting optimisation we now apply is lowering “deferrable” variables. For instance, consider this definition:

(with (foo '())
  (when (empty? '())
    (debug foo)))

The definition of foo is only needed if then when block is executed, and so we’re able to “push” the definition down:

(when (empty? '())
  (with (foo '())
    (debug foo)))

Whilst this code doesn’t appear much in practice (or at least in this overly simplistic form), it’s nice to catch the few occasions where it does.

Wow, it’s been a while since the last update. During that time, we’ve been doing a lot of work on the standard library. So let’s dig right in an see what’s changed.

Standard library restructuring

Urn’s standard library has always been a bit of a mess, with features scattered everywhere. Whilst some chaos is inevitable due to having to bootstrap the library, there was still lots of improvement. Sadly, deciding on a good approach proved more complicated than expected.

For Urn 0.6 we finally went about this restructuring, splitting Urn into several modules such as core (for bootstrapping the stdlib), data (data manipulation libraries) and test (used for writing tests). It’s worth noting that some programs may need changing to work with the new layout, but rest assured we won’t be changing it again any time soon.

Extended math library

Lisp has always had a history of numerical computing, and Urn does not disappoint. 0.6 adds support for rationals, complex numbers, vectors and matrices. It’s now that much simpler to do your maths homework in Urn!

In order for these features to work fluidly together, we’ve added a collection of generic arithmetic methods, such as n+ and n* which operate on arbitrary data structures. These methods, along with rational literals, bring a lot to the table.

> (import math/vector ())
> (n* (vector 1 2 3) 2/3)
out = [2/3 4/3 2/1]

String formatting changes

hydraz has done a lot of work on developing more powerful ways to format strings. Whilst string/format is useful, it can still be tedious to use. Thankfully the data/format library accepts format strings which neatly integrate with pretty printing, as well as allowing keyword arguments and interpolation of symbols in the current scope. I’d really recommend checking out the documentation for more information.

> (format nil "0x{foo%x}" :foo 123)
out = "0x7b"

Internal changes

Whilst the obvious changes have occurred within the libraries, the compiler’s internals have also changed. Aside from improved codegen, line mappings are now substantially more accurate. Consequently code coverage and error messages are more informative - always a perk when debugging!

We’ve just released another Urn update - 0.5.1. This is a relatively small update, mostly composed of bug fixes and performance improvements. That’s not to say there aren’t some useful new features though!

REPL improvements

The Urn REPL is a great tool for testing and prototyping code. It’s something we regularly use, and so are always eager to improve it.

One task we often find ourselves doing is loading up the REPL, importing a module and testing a small feature in it. This process has now been streamlined: when using the --repl flag, any file provided on the command line will be automatically imported into the current scope.

We’ve also added a :view command, which allows you to preview a symbol’s definition. This saves you switching between source code and the REPL, trying to remember how you got that awesome feature to work.

Optimised optimiser

The Urn compiler has never been the fastest beast, with the optimiser being the worst offender. Whilst the optimiser is great at reducing the size (and speed) of generated code, it also takes considerable time to run. Even worse, much of the optimiser’s time was just spent in iterating over every node - not even running the various passes! Something had to be done.

Urn 0.5.1 introduces a new framework for optimisations, doing something so obvious I’m surprised we weren’t doing it from the start. Instead of running each pass sequentially, we have one unified visitor object which will call each pass for every node it hits. Not only does this substantially reduce the visitor overhead, it reduces the number of times we have to run a pass, resulting it in an even bigger performance increase! We’ve also made some improvements to definition and usage tracking, which ensures that the information is more up-to-date, and so the optimiser can make smarter decisions.

These various improvements have resulted in a 2-3x performance increase in the optimiser (depending on Lua version and implementation). In fact, it now takes less time to optimise the compiler than it does to load it. There is evidently room for performance improvements elsewhere!

Updates my dears, and not a moment too soon. It’s been 100 commits since our last release, and each commit is packed with lispy goodness. OK, maybe not every commit. In this time, we’ve reached 1000 commits which, whilst a rather arbitrary milestone, it still a milestone we can celebrate. There’s been more changes than I can cover here, so I’d really recommend checking out the release nodes for the whole picture. Anyway, here’s a small summary of what’s new:

Multimethods (aka defgeneric)

Whilst Urn prides itself on the extensible power that macros offer, the extensibility of the standard library has historically been lacking. There are several functions which do a long if-elseif chain of type comparisons, with a final metatable lookup if all else fails. This is obviously undesirable and so something needed to be done. Enter multimethods.

If you’ve used an object oriented language such as C++ or Java then you will have used methods before, and so multimethods will not be an entirely alien concept. However, instead of dispatching based on the type of the first argument, it uses all arguments to determine which function to call. This makes multimethods a much more powerful solution.

In order to demonstrate their power, we’ll create a generic merge method. This’ll attempt to merge two objects together and return the resulting object. First we define a new method with defgeneric, providing our argument names and a docstring:

(defgeneric merge (x y)
  "Merge X and Y together, returning a new object.")

We can now call merge, but we’ll just get an error. After all, we haven’t defined any implementations yet. This is done with defmethod, specifying the argument types and an implementation. Let’s do a couple of simple ones:

(defmethod (merge string string) (x y) (.. x y))
(defmethod (merge list list) (x y) (append x y))

Now let’s use these:

> (merge "foo" "bar")
out = "foobar"
> (merge '(1 2 3) '(4 5 6))
out = (1 2 3 4 5 6)

Of course, if you’ve got a fancy structure which can be merged, its trivial to add support for that too. We’ve made great use of this feature in the standard library, converting pretty and eq? to use it.

I’d really recommend reading hydraz’s blog for more information about this feature. He’s the one who wrote it, and goes into much more detail about the implementation and rational behind it. Now’s when I’d normally insert a cynical comment about no-one getting this far, but it’s late and there are plenty more features to cover…

Fancy structures

One of the gems of Lua is it’s table. This simple data structure forms the base of many of Urn’s standard types. However, using passing around tables and indexing them via strings does rather take the idea of abstraction and smash it on the floor. We’ve written the urn/struct library to provide a more intuitive way to create your own data structures.

This library provides the defstruct macro. This provides a small DSL which allows you to declare fields, constructors and more. This will, in turn, generate the appropriate constructor, getters, setters and pattern matching utilities. Sounds great? Well let’s jump in and declare the struct staple - a point.

(defstruct point
  (fields
    (immutable x)
    (immutable y)))

(defmethod (pretty point) (p)
  (string/format "(x=%d y=%d)" (point-x p) (point-y p)))

This structure definition will create a constructor (make-point) and a type predicate (point?). Each field definition will specify an argument to the constructor and a getter. If the field is marked as mutable, then a setter will also be generated. We also chuck in a pretty implementation for good measure.

Now we can start constructing points, indexing their fields and printing them:

> (point 1 2)
out = (x=1 y=2)
> (point-x (make-point 1 2))
out = 1
> (point-y (make-point 1 2))
out = 2

If you don’t like these function names, then customising them is trivial. One can also make particular fields hidden package-local, if you have implementation specific fields. See the documentation for more information.

Testing, coverage and stability

All these new features a awfully nice, if we don’t have any guarantees they actually work correctly. That’s why we’ve spent a lot of time getting our testing infrastructure up to date. I won’t go into the details as frankly, it isn’t that interesting but here’s a couple of highlights:

Example testing

The Urn documentation is full of example usage for various functions, complete with inputs and outputs. These examples are perfect for a trivial test case, to ensure the basic functionality is there. To make best use of these examples, we’ve written a compiler plugin which extracts codeblocks from docstrings and runs them, verifying the output is as expected.

Whilst I’m not proposing doing away with normal tests - after all, you don’t want to include every edge case in an example - they are a small step in making sure Urn is fully tested. Furthermore, it allows us to ensure all examples are up to date, and provides an incentive to write more documentation - always a plus.

Code coverage

Tests are great for seeing what works, but it’s much harder to find what doesn’t work. One way to help with this is to determine what the test suite doesn’t check. And so, of course, we’ve written a small code coverage utility for Urn. It generates files in the same format as LuaCov, and so provides some interpobility with other Lua-based tools. To get started, just run your code with the --profile=coverage switch.

Codegen and optimiser improvements

Normally I’d bang on about the most exciting optimisation changes this release. However, for better or for worse, there haven’t been many major changes. Instead, there’s been a lot of work on stabilising the various systems, resulting in a less buggy and more consistent experience. Whilst it’s far from perfect, we’re one step closer to not blowing up in your face the whole time.