The
Business-DSL DevOps Process
During my talk this week at the JAX conference one of the audience asked how the
devops process works for systems where domain experts contribute the
Fachlichkeit directly using DSLs. This prompted my drawing of the Big Picture
picture here which illustrates how the roles of domain experts, language
engineers, infrastructure guys and technical architects collaborate to deliver
DSL-based software in an efficient, fast and reliable way. Here is
the picture; I will discuss it in the rest of this article.
Modeling and Testing of
Fachlichkeit by Domain Experts
Not surprisingly, the
pipeline starts out with the domain experts. They use non-formalized processes to
build an intuitive understanding of “the world”, i.e., the the aspects of their
domain they are tasked with contributing to the software system. For example,
they might analyze tax law.
Based on this understanding,
they create models that faithfully represent the domain. They use one or more
suitable DSLs for this task (we will return to this below). Importantly, they
also write tests. Good business DSLs always have ways of expressing tests for
the Fachlichkeit at various level of granularity . The IDE supports the
execution of those tests, rendering the test results directly in the IDE. The
domain experts thus have an efficient and immediate way of expressing and
validating Fachlichkeit. If tests fail, this feedback arrives at the domain
experts, and they are primarily responsible for fixing the problem with the
Fachlichkeit.
In-IDE execution is often
achieved through in-IDE interpreters.
Automated Test Execution on
a CI server
Next, we use existing version
control technology to forward the updated models to a continuous integration
server. There, again, we run tests, just like in the IDE, but we run all of
them; a particular user might only have checked out a part of the overall
model, and thus also only run a part of the tests, so he might inadvertently
break tests “outside” his current scope. Running all tests on a CI server for
each commit addresses this risk. If tests fail, we notify the domain experts;
we still presume that there is a problem with the Fachlichkeit or its tests.
Both the VCS and the CI
server are painted green, because these are managed and setup by the
infrastructure people. While, in the spirit of devops, we don’t want to draw
the lines between those roles too strongly, it is quite clear that it will not
be the domain experts who set up and run this infrastructure.
Code Generation and Testing
of Code
Usually the interpreter that
is used inside the IDE is not the means of executing the Fachlichkeit in the
actual target system (if it is, this step can be skipped). Instead, in many
cases, we actually generate source code for downstream compilation, or we
generate an intermediate format which is then interpreted. In both cases, this
is, if you will, a secondary definition of the language semantics, and it has
to be aligned with the primary one, the interpreter. In other words: we have to
run all the tests in the generated version of the Fachlichkeit again. Of course
we do this on the CI server as well:
This is where the language
engineers come into the picture. It is their responsibility to write the
generator. And if tests that work in the interpreter fail in the generated
code, then it is a problem in the generator; thus, failed generated tests, or
failures in the generation itself, end up in the lap of the language engineer,
not the domain expert.
And since we’re talking about
the language engineer here: of course, they also develop the languages used by
the domain expert, and they are also responsible for the language workbench.
System Integration
The code generated so far is
only the direct representation of the Fachlichkeit. We can run the
Fachlichkeits-Tests against it, but it is not yet integrated with technical
concerns of the system such as databases, external system connectors or means
of handling -ilities (yes, I am vague here, because these really depend on each
particular system). Those aspects are handled by the technical system
architects. So in this step, we integrated the Fachlichkeits-core with the
frameworks and facilities of the overall system. The result of this is
essentially a deployable package that can technically be released into
production.
Note that we have made a
small change to the previous step: the generators might of course already know
something of those technical aspects as well (see the little yellow-ish arrow
on the brown generation arrow). So the generators are usually developed jointly
by the language engineers and the technical architects.
Staging
Once we have the overall
deployable package(s), those can be deployed to a staging infrastructure, where
the Fachlichkeit is ultimately executed in its “natural habitat”, i.e., in the
context of all the connected systems and infrastructure.
Now, if something goes wrong
here, it is much harder to tell whose contribution caused the problem. So
everybody is notified, and they need to figure out together what the problem is.
However, it should be quite rare that something goes wrong here because of all
the previous testing steps; in particular, a failure in the Fachlichkeit should
be exceedingly rare.
Going Live
The final step is going live;
essentially, the staging environment is “mirrored” into the real world. Nothing
should ever go wrong here; if so, it’s probably the CEO who’s notified :-)
This closes the circle for
the domain expert. As you can see from the highlight, they mostly care about
modelling, local testing, fixing problems that get reported from the first
stage of the build server and then, finally, smoke testing the staging environment
and releasing the new Fachlichkeit into production. Exactly as it should be :-)
Automation, Iteration and
Roles
Note that, in the spirit of
devops, the complete process is automated (except, of course, the release into
production, which requires explicit triggering). And despite the explicit
mentioning of roles and their responsibilities, the process should be
collaborative. Especially during the initial stages of establishing the
process, the assumption that failing “blue” tests are always a problem with the
Fachlichkeit or its test might not be true: there might be a bug in the (not
yet mature) interpreter or other aspects of the language definition.
The Language Development
Process
I didn’t cover the
development of the languages in this article. I will do that in a future post.
But suffice to say, that this also happens collaboratively between the at least
domain experts and language engineers, sometimes also with the technical
architects, and it is highly iterative, as the following picture hints at. As I
said: more details in a future post.