This blog post is about a design opinion I hold because I’ve made the wrong choice before. This may be obvious to a lot of people, but I’ve now seen this so many times in the wild that I thought I should write this down.

You’re an engineer working on a project and you saw that your team has been implementing the same thing again and again with little variation. You think, this should be a framework. We’re essentially doing the same thing, why should we duplicate this much code.

Nobody wants to copy-paste the same code ten times. So you extract the common parts, identify the things that vary, and build abstractions around them.

Instead of writing code for each variant, you define each variant in JSON or YAML. The framework reads the configuration and changes its behavior accordingly.

“This is amazing”, you think to yourself.

I’ve done this (twice). Once in my first job, then at my own startup. However, I’ve learned there is a better way.

I now believe it’s always a better option to build your framework in code.

Do not use config files. You don’t need them (in most cases).

Allow me to make my case.

At my first job, we were building an ETL tool that needed to support multiple integrations for extracting data, transforming it, and loading it somewhere else. These integrations needed to operate in both batch and streaming modes. It was a pretty clear-cut case for getting rid of duplication by creating a framework.

What we did was make every integration in the framework a JSON configuration. Each integration defined some user-facing metadata:

{
   "name": "Some Integration",
   "description": "...",
   "icon": "..."
}

But it also described how the integration actually worked.

Which authentication scheme does it use?
How do we access the data? Is it an object store or a REST API?
What format does the data come in?
How do we process that data?
How do we paginate?
How do we authenticate?

You get the point.

The framework would read this configuration and do the right thing.

And it worked great.

Initially.

What started with a simple configuration like this:

{
  "auth": {
    "type": "api_key",
    "token_location": "header"
  }
}

Eventually, turned into this:

{
  "auth": {
    "type": "oauth",
    "token_location": "header",
    "refresh": true,
    "custom_refresh": true
  },
  "pagination": {
    "type": "cursor",
    "cursor_location": "response.headers"
  },
  "rate_limit": {
    "..."
  }
}

The exact fields aren’t important. The pattern is.

“It’s just one integration.”
“Let’s just add a key.”
“We don’t want to make a breaking change.”
“The framework already supports 95% of this. We just need one more flag.”

And so you add the flag.
Until you need another one.

First, your configuration contains values:

authentication: oauth
format: json

Then you need a few more options:

authentication:
  type: oauth
  token_location: header

Then you need combinations of options.
Then overrides.
Then special cases.
Then conditional behavior.
Then you need to express relationships between fields.

It’s possible to do it right.

You can,
Add templates.
Add expressions.
Add conditionals.
Add functions.
Add scripting.
Add overrides.
Add custom plugins.

This can work.

But…

At what point are you just building a programming language?

If your configuration has conditionals, functions, composition, custom behavior, and special cases, you probably don’t have configuration anymore.

You have a programming language with worse tooling.

Sometimes that’s actually the right decision.
But it should be a conscious decision.

Not something you accidentally arrive at one special_case field at a time.

The problem with config-driven frameworks isn’t that the first version is bad.
It’s usually the opposite.

The first version is fantastic.

It’s simple. It’s elegant. There’s very little code. Adding a new variant feels trivial.
The problem is what happens six months later.

Every new requirement asks:
“Can our configuration model represent this?”
If the answer is no, someone adds another key.
Then another.
Eventually the configuration becomes a collection of historical decisions rather than a clean representation of the domain.

The framework becomes responsible for interpreting all of those decisions.
And now a tiny change to one integration can require understanding the entire configuration system.

Instead, it could’ve been just a simple interface in code that gave you the standardisation of a framework with flexibility to handle all kinds of new customizations and edge cases without you having to implement a whole new DSL or a programming language.