Why it is so hard to
change established processes in organizations
Or why so many DSL projects fail for non-technical reasons
Business Process
Improvement with DSLs
Most of my DSL-related
projects are ultimately about business process improvement: we’re trying to
improve the processes how subject matter finds its way into software. We
achieve that improvement by developing languages that are used by subject
matter experts to directly and formally express and test that subject matter,
and then automating the translation to source code.
As we do that, the existing
processes must go. This should be obvious. And companies only hire me and my
colleagues if they understand that there’s something wrong with their existing
process. So there is some degree of awareness that there’s a problem. And
problems there are! You wouldn’t believe how broken some (not all!) of these
processes are, even for companies that are leaders in their respective markets.
So replacing these broken, slow, and low-quality processes should be easy,
right?
Challenges
Of course I wouldn’t write
this if that were the case. Streamlining processes requires buy-in from all the
people and roles involved in the current process. And processes that are
especially broken often involve especially many roles — that’s often a major
reason why they are broken. You have to convince each involved party that the
new process makes their life easier while making sure they remain “important”
to the overall story — nobody wants to lose status. And most changes, when you
propose them, don’t seem to people that they make their life easier because
people tend to a) confuse the transient state of learning something new with a
potentially more productive steady state after the learning is done, b)
psychologically value loss (of the current way of working) higher than benefit
(of a new improved way) and c) oh, DSLs really look like programming, and
I hate programming.
Leadership (not pressure!)
from management can help overcome these problems by painting a picture of where
the organization should evolve to and how that improves the business and
everybody’s everyday work. But good leaders in management aren’t always
available. And often management is too far away from the particular processes
for them to convincingly talk to the people who must change their ways.
So what? Well, such projects
then tend to fail. Or they require so many compromises in the new process that
many of the benefits of the new approach disappear (which then leads to the new
process not delivering, and probably dying later). The organization muddles on
… until their competitor does a better job of improving their process.
The Accidental Process
So, you might ask, how did
the processes become so broken in the first place? Why did all the people buy
into the broken processes when that one was introduced? Say hello to path
dependence. Here’s the definition from Wikipedia (with my adaptation in brackets):
Path dependence is when the decisions
presented to people are dependent on previous decisions or experiences made in
the past. Path dependence exists when a feature of [the process] is not based
on current conditions, but rather has been formed by a sequence of past actions
each leading to a distinct outcome.
If you were to try to
introduce the existing process in its current (broken) state into the current
(often large) organization you’d fail because the brokenness of the process is
obvious to everybody. So what happened is that the process has become broken
through small, incremental changes based on the sequence of previous small,
incremental changes:
· time pressure didn’t allow for a reasonable
change, so you “improvise”
· people had certain skills and didn’t know
about a better solution
· “we already use Excel for X, so let’s also
use it for A, B, C and D” (even though it’s not a good fit)
· “let’s go with this industry standard” (even
though what we try to do is quite different from what the standard is built
upon)
Through chains of incremental
changes — each of them somehow kinda justifiable — organizations paint
themselves into a corner that makes them slow and ineffective. If you ask
people why things are as they are, they often can’t explain. And in fact, there
isn’t a good reason. It just evolved this way, the current state is basically
an accident. A corollary of this is that the statement, “well, we do it this
way, so it must make sense” just isn’t true. When I was a teenager, frustrated
about some of the ways my flying club conducted its business, my buddies and I
used the term “Tradition statt Vernunft” to describe this situation; tradition
instead of reason.
How can it be changed?
When you try to change the
process to something more reasonable, it’s often not feasible to rely on
incremental changes. It would take too long, and the outcome is hard to
predict. To achieve meaningful business process change, some people have to
change how they work significantly. Or people might have to be educated or
retrained. Some people might become more “important” while others will lose
some of their status. Or maybe you have to change the structure of your
workforce (letting some people go and hiring others with different skills).
Obviously there is resistance that requires management attention. Nothing can
be achieved with just a better tool!
These problems make this kind
of change really hard. Which is also why it happens relatively rarely (and often
frustrates Yours Truly). I’ll write some more about some of this in the future.
Zooming out
By the way: path dependence
and the resulting “unexplainable” ways of how some things are done is of course
not just a thing in business. It’s a feature of our society and political
system as well. And the same reasons explain why obviously nonsensical things
persist for so long in a society, even though the environment might have
changed a lot. This frustrates the citizen Markus just as much as it frustrates
the consultant Markus.