As you may know, I always try to find (more) convincing and (more) accessible ways of explaining why using DSLs is a good idea and why they should be used more. Here's another attempt. Looking forward to your thoughts!
And no, the airplane up there has nothing to do with anything. It's just a recent picture I took (and edited) that I happen to like :-)
What do the following three software systems have in common?
· A CNC mill (a computer controlled milling machine) that cuts parts out of raw metal.
· A CAD system that can represent complex three-dimensional shapes and analyse their properties
· A radio telescope used to observe the sky.
All three systems -- and there are many more like these -- allow users to make the system do their own particular things. A CNC machine can produce any 2.5-dimensional part, the CAD system lets users design any shape they want, the radio telescope lets astronomers configure observations at (more or less) any point in the sky in a wide range of wavelengths. All three are highly configurable systems.
To make this possible, these systems all provide a language to the user with which they can express the configuration, express what they want the system to do. For the CNC machine, that language consists of ways for specifying paths, tool selection, mill head RPM and so on. For the CAD system, the language has abstractions for three-dimensional geometry represented WYSIWYG-style as 3D shapes. And for the radio telescope the language is mostly GUI-oriented and consists of all kinds of concepts from astronomy, geometry, Newton's mechanics, wavelengths and data processing.
These systems have one more thing in common. The users who configure them are not software engineers. They are design engineers, draftsmen and astrophysicists. So the languages that make the systems so configurable are targeted at the subject matter experts in these domains. They can be seen as "special-purpose programming languages" for non-programmers.
Here are three more systems that have something in common:
· A system for calculating taxes based on Germany's tax regulations
· A system for executing clinical drug trials
· A tachograph for tracking a truck-driver's driving and break times
These three systems -- and there are many more like these -- contain a whole wealth of application logic: all of Germany's tax law, the precise rules for running a particular drug trial, or all the rules regaring drive times and mandatory breaks for all the countries of the EU.
They have one more thing in common: all this application logic -- the subject matter of the domain -- doesn't originate from the brains of programmers, but from people who are subject matter experts in those domains. Just like the engineer, draftsman or astronomer in the first set of examples.
How does this subject matter get into the software? Is it "configured" by these subject matter experts, just like the in the case of the first three systems? Typically not. Instead, the subject matter experts write requirements, imprecise prose that is then (mis-) interpreted by software developers and translated into code. As we all know, this is a slow and error-prone process. Why do the subject matter experts in these domains accept this tedious, manual process? Why don't we as software engineers use the same approach as for the first three systems and make these "configurable systems" by developing a suitable language for the subject matter experts to directly configure the subject matter for these respective domains? Why is this approach commonplace for some kinds of systems and a rarity for others?
For the CNC machine, the CAD system and the radio telescope it is totally obvious that the manufacturer of these systems can't code up all the parts, drawings or observations their users might want to execute. Imagine a design engineer describing the shape they want milled by the CNC in prose, sending the document as an email to the manufacturer of the machine, and, after four weeks, receiving a patch of the machine's firmware that is able to mill his specific part. Sounds like a joke, right?
So again, why do we not use subject-matter-configuration approach for the second set of systems? One thing I hear is that "non-programmers can't program." But we're not trying to make them program. Programming and software engineering is about mostly the technical aspects of software systems: robustness, scalability, performance, maintainability and the like. We're merely trying to make them use a suitable tool to be precise and complete in (formally) configuring, specifying, modeling the application logic. And that this is possible is demonstrated, among others, by CNC machines, CAD systems and radio telescopes. Yes, it takes some learning and it is a bit of a culture shift for the subject matter experts. But it's absolutely feasible for the vast majority of subject matter experts. In fact, once they have gotten use to the approach, most of the subject matter experts I have worked with feel empowered.
Another reason why this approach might not be used more is that in the case of the business systems the people who configure the systems are inside the company (as opposed to the users of CNC machines, CAD systems or the worldwide community of astrophysicists). So it is *possible* to use the traditional prose-based manual approach. But that doesn't mean that it is a good idea -- again, it is slow and error-prone.
Yet another difference is that in the case of internal systems -- those where the subject matter experts are employees of the company that builds the system -- the domains are usually smaller, so the effort of building the configuration languages and tooling might be harder to justify. This is true. But building a configuration language for tax calculations is also significantly less effort than building a full-blown CAD system. It is true though that the approach only works for organisations that work in a stable -- albeit evolving -- domain for a reasonably long time. But this is the case for most medium to large companies.
In fact, the tax, trial and tachograph systems are examples of where we have done exactly this: we have built configuration languages that allow the non-programmer subject matter experts to express the application logic for their domains, and the underlying platform then executes this logic when it calculates taxes, runs a trial or records a driver's activities. The overall process is now much faster, less error-prone and easier to evolve.
One last comment. When we talk about configuration languages, what does this exactly mean? How do these languages look like? In the very simplest case, they could be a XML, Json or YAML file with a predefined structure. But for realistic systems in interesting domains, you need something more sophisticated, something that scales better with the size and complexity of the domains and provides a better UX for the subject-matter experts. The languages usually have a nice, textual syntax and/or rely on tables and diagrams. Some also use GUI forms, and a few rely on something completely custom, like the CAD system. They have strong tool support that helps with creating correct configurations, plus various analysers and checkers that further help in this regard. Almost all come with means to express tests for configurations, some come with simulators that let users run a configuration in a safe environment. In the future, they might even be supported by ChatGTP & friends, as I demonstrate in this video. We are talking about domain-specific languages.
So, to summarize, what exactly does it mean to build a configurable system?
· Our goal is to let non-programmer subject-matter experts configure what the system,
· in a way that is precise and directly executable (and not just a prose description),
· by using a language and tool that we software engineers have built for them,
· after scoping and analysing the domain and the relevant subject matter.
Unfortunately, there aren't too systems that are built this way more like this. The manual process based on developers translating prose to code is still commonplace. This needs to change.