Why TPL?

  by Duncan Grisby | 06 Jan 2009 | Permalink |

A question we are often asked is why we chose to invent our own language, TPL, rather than using an existing language for defining patterns. The answer is that there are a lot of benefits to having a specific language, some of which I’ll describe in this article.

TPL is a domain specific language, meaning that it is designed for writing patterns and nothing else. It has language syntax specifically for talking about patterns, triggers, removal blocks, and so on. If we were to use a general-purpose language, all those things would have to be specified in API calls and naming conventions instead, meaning that patterns would be much harder to read, and it would be much easier to get things wrong. Although you can of course write some pretty complex things in TPL, many patterns are very simple and we want people with no programming experience to be able to write them. Having to define classes and call APIs using a general-purpose language would make pattern writing a much more daunting experience than it is with TPL.

Some of the less obvious benefits of TPL come from the fact that the compiler that turns TPL into rules that execute in the Foundation engine is really rather clever. Patterns can be thought of as doing two kinds of work — retrieving data from the environment (a.k.a. discovery), and processing that data. Of course, discovery involves waiting for the device in the environment to respond, which might take a very long time (in fact, scanning schedules may mean it is hours or even days before the system is permitted to talk to the device). The TPL compiler splits patterns into separate rules that trigger when the results of discovery become available. This means that the system can efficiently continue to process other rules while waiting for a very large number of discovery results.

The way patterns are split also means that processing is carried out in small self-contained chunks. As it runs, the system keeps track of what rules are active and the outstanding discovery requests. If the system unexpectedly terminates (due to a power failure, aggressive system administrator or, God-forbid, a bug), it is able to resume processing with the system in exactly the same state is was in previously. If patterns were not split across discovery boundaries, the best the system could do would be to resume at the start of any executing patterns, potentially redoing a lot of work.

Another thing the TPL compiler does is enable the automatic generation of provenance information. All operations that read, manipulate and write data are annotated to track the origins of the data. So, in a simple set of TPL operations like this:

  name1 := node1.name;
  name2 := node2.name;
  node3.name := "The combination of %name1% and %name2%";

Node 3 is automatically given provenance relationships in the model to indicate that node 1 and node 2 are contributors to the name attribute. In a general-purpose language, provenance would have to be maintained manually, which would mean pattern writing involved much more effort, and would mean you could forget to do it. With TPL, you don’t have to remember to set provenance in these kinds of situations — the system just does it for you. The result is that it is possible to view and report on the source data for all information stored in the Foundation model, meaning it’s possible to understand and trust the information, without any work on the part of the pattern author.

These are just a few of the reasons we decided to invent our own language rather than using something general purpose. To be honest, we also did it because we like inventing things, but I think it was the right choice. What do you think?