Why We Built "Wayler": A Script Orchestrator

Spend five minutes on the corners of social media where people talk about technology and you come away with the impression that a hype has taken hold — one saying that the only way to build software (as of the date of this article) is now, beyond any doubt, through AI agents, for any task. I’ll say that again: any task. This may sound anti-AI, but don’t get us wrong — at Waymotion we use AI models and tools every day to help produce source code, design systems, and work through problems.

But recently, faced with the task of migrating complex geospatial data from ArcGIS REST services and internal APIs into an OGC API (see this case), we decided to go against the current: we built our own script orchestrator which, in this case, ended up being used to orchestrate an ETL system — simple and deterministic, like in the old days.

We called it Wayler (since we used docker at the core of both the architecture and the operation, picking the name was the easy part). Here is why we built it: why we didn’t go for the obvious industry choices (the open-source ones included) and why betting on Docker and on Python gave us exactly the control and the software sustainability we judged sufficient for what we had committed to.

The “problem” with the alternatives

When everything you need to do is recurring and repetitive at an operational level — collecting data from an API, transforming it to a specific standard (like INSPIRE) and loading it reliably into a database — the ecosystem currently available in the open-source community presents a few long-term challenges that are crucial to weigh, especially in a small team:

• (Spark, Airflow, Mage.ai): incredible tools for Big Data lakes and distributed systems. But configuring a cluster or defining DAGs just to paginate through a data source (an endpoint, say) is like using a Ferrari for a daily 5 km commute.
• (n8n, Zapier): excellent for webhooks, but they easily turn into “visual spaghetti” the moment you have to deal with transformations (geospatial ones, for example) or with coordinate conversions (EPSG:3763 to EPSG:4326).

As a software engineer, I believe source code still demands — and will keep demanding — human judgement, regardless of the tool or of who generates it. So we didn’t want to abstract the process away entirely, because we wanted to keep absolute control over the logic of the processing scripts, in this case written in Python. If an IF based on a 1:50k vs 1:200k scale is needed, we want to see that condition explicitly in the code.

When scripts are written for repetitive tasks tied to deterministic processes, in our view they offer far greater value for the long-term maintenance and sustainability of the software. We don’t have to ask ourselves why an LLM hallucinated a data mapping (something that does happen in the output these models generate). A deterministic output simply works (or doesn’t), exactly the same way, every time.

On top of that, at the infrastructure level, we didn’t want to depend on the server’s operating system, or on third parties, to manage the scheduling of script runs. We needed internal control, running exclusively on Wayler’s own “infrastructure”, fully isolated.

Wayler and the “Harpoons”

To address this question, we implemented Wayler — a lightweight, custom-built orchestrator that depends on Docker.

In our architecture, Wayler acts as a command centre. It doesn’t process logic or data; it relies on a local database to manage the pipelines, retry runs and trigger alerts. When the time comes, it simply launches a Harpoon — a Python script with one specific purpose, wrapped in a Docker container.

The decision to use Docker

Total isolation. By forcing every process to live inside a Docker container, we guarantee that each Python script lives in its own environment. There are no dependencies where, say, a GeoPandas update for one script has an impact on another. It also makes rolling out new versions incredibly simple — we update and release a new Harpoon Docker image, update the record in the database*, and the next scheduled run uses the new code without affecting the rest of the system.

It also let us enforce strict behaviour in the SDK we decided to build to tie the pieces together (internally we call it “Iron”). Every Harpoon inherits from a BaseHarpoon Python class. This simplified our script development process, because it defines a contract: for a script to be picked up by Wayler it must implement a process() method, and the SDK automatically handles consistent logging, database sessions and error catching.

But there is a limitation. An orchestrator like this comes with an important caveat: it depends 100% on Docker. Wayler has to have access to the Docker socket to create containers and monitor them (run state, prune, and so on). For us, though, this dependency was deliberate, and we believe it was the right decision.

In any case, implementing a Harpoon with the SDK ends up being as simple as the following example:

from wayler_sdk import BaseHarpoon
from integrations.erp import ERPClient

class ERPUsersLeaveHarpoon(BaseHarpoon):
    def process(self):
        self.logger.info("Starting Voyage: Fetching users data...")

        # 1. Fetch data from source
        client = ERPClient(url=self.config.SOURCE_URL)
        raw_data = client.fetch_all_users()

        # 2. Procedural, deterministic transformation
        self.logger.info(f"Retrieved {len(raw_data)} records. Transforming...")
        clean_data = self.transform_to_something(raw_data)

        # 3. The SDK handles the atomic swap to the DB
        self.logger.info("Loading into Analytics Staging...")
        self.atomic_load(
            target_table="stg_leave_users",
            data=clean_data
        )
        self.logger.info("Voyage successful!")

if __name__ == "__main__":
    # The run() method handles dry-runs, retries, and DB sessions
    ERPUsersLeaveHarpoon().run()

A practical case

In Wayler’s first real use case, the main data consumer was a public pygeoapi service reading from a PostGIS database. And that is where we hit our first problem: a Harpoon takes 20 minutes to download and process thousands of geological features, and in this case the data was going into a public API that has to keep working with no interruption of service.

But because we have full control over the procedural Python, our SDK enforces an atomic execution pattern, which is particularly useful when migrating data:

Next steps

Wayler has proven stable and efficient in real, complex ETL systems, so we are going to use it for other system synchronisation processes beyond geospatial data. Even so, any engineering team knows its system isn’t perfect yet (and never will be :D). We have at least two improvements to make in the short term:

• A graphical interface: *right now, managing Wayler means running a few docker commands and writing SQL INSERTs into its database. We need to build a dashboard to visualise pipelines, trigger manual Harpoon runs and follow the monitoring logs.
• Secrets: integrate or build a secrets manager, so that Harpoons fetch their credentials dynamically at runtime (mage.ai, for example, does this very well).

Conclusion

In an era where the tech industry seems to move at something close to the speed of light, it’s worth remembering that data synchronisation is, in most cases, like a meal served cold and predictable. We love AI for what it does best (which is a lot), but for the business of running scripts? I don’t think we need it.

As a developer trying to keep up with how LLMs are evolving, I admit there was a small part of me that felt the “fun” of using AI for orchestration would be lost by choosing the traditional, deterministic route instead of spawning autonomous agents to run scripts. But the truth is that the Harpoons still have to be developed, the architecture still has to be designed, and so on. And that is exactly where LLMs are strong and genuinely useful — we get the reliability of deterministic code (even when it was written with AI), without losing the joy of “modern development”.

Which leaves me thinking: sometimes, the most innovative thing you can do is write code that does exactly what you tell it to do, even if it’s a bug.