We built Wayler, our own small, deterministic ETL orchestrator, instead of reaching for the industry giants. The case for procedural Python, the strict boundaries of Docker, and builds that do the same thing every single time.
Contents
If you spend five minutes on tech social media today, you’d think the only way to build software is by deploying a swarm of autonomous AI agents. Don’t get us wrong — in Waymotion we are not anti-AI. We use AI tools on a daily basis to write code, architect systems, and solve complex problems.
But recently, when tasked with migrating complex geospatial data from legacy ArcGIS REST services and internal APIs into a standardized OGC API (see this), we made a contrarian choice: We built our own simple, deterministic ETL (like in the old-days) orchestrator.
We call it Wayler (since we used docker at core, the name was the easy part) . Here is why we built it, why we didn’t opt-in to the industry giants, and how leaning into Docker and procedural Python gave us exactly the control and software sustainability we needed.
The “Problem” with Modern Data Tools
When all you need to do is collect data from an API, transform it to a specific standard (like INSPIRE), and safely load it into a relational database, the current ecosystem presents a frustrating dichotomy, that its crucial to judge for small teams:
- The Heavyweights (Spark, Airflow, Mage.ai): These are incredible tools for massive, distributed data lakes. But configuring a cluster or writing complex DAG abstractions just to paginate through an ArcGIS REST endpoint is like using a sledgehammer to crack a walnut.
- The Visual Builders (n8n, Zapier): Great for simple webhooks, but they easily become visual spaghetti the moment you need to handle complex geospatial data transformations or spatial coordinate conversions (EPSG:3763 to EPSG:4326).
We realized we didn’t want to abstract the code away, since we wanted the control of writing procedural Python. If an IF statement based on a 1:50k vs 1:200k map scale is required, we want to see it explicitly in a .py script.
When deterministic processes and builds are possible, in our vision, they are vastly superior for long-term software sustainability. You don’t have to wonder why an LLM hallucinated a schema mapping, and you don’t have to untangle a massive visual node graph. It just works (or not), exactly the same way, every single time.
Furthermore, we didn’t want to rely on the host operating system or third-party SaaS for scheduling. We needed an in-house scheduler that ran purely on our infrastructure, fully isolated and secure.
Wayler & Its Harpoons
To solve this, we designed Wayler—a lightweight, custom orchestrator, docker based.
In our architecture, the Wayler acts as the command center. It doesn’t process data itself; it relies on a local database to manage schedules, retries, and alerts. When a scheduled time arrives, it launches a Harpoon—a single-purpose, containerized Python script.
The Docker Decision: A Feature, Not a Bug
Building an orchestrator this way comes with a notable caveat: It is 100% dependent on Docker. The Wayler must have access to the Docker socket to spin up containers, monitor their exit codes, and tear them down. But for us, this dependency was a deliberate, and we believe it was a highly advantageous decision.
By forcing every ETL job into a Docker container, we ensured that each Python script lives in its own world. There is no “dependency hell” where an update to GeoPandas for one script breaks another. It also makes rolling out new versions incredibly easy—we just build and tag a new Harpoon image, update the database record, and the next scheduled run uses the new code without affecting the rest of the fleet.
Additionally, this allowed us to enforce strict SDK Behavior (we call it internally “Iron”). Every Harpoon inherits from a BaseHarpoon Python class. Developers just implement a process() method, and the SDK automatically handles consistent logging, database sessions, error catching, and a mandatory —dry-run flag.
Here is a quick look at how simple a Harpoon implementation is using the SDK:
from wayler_sdk import BaseHarpoon
from integrations.arcgis import ArcGISClient
class MineralOccurrencesHarpoon(BaseHarpoon):
def process(self):
self.logger.info("Starting Voyage: Fetching ArcGIS data...")
# 1. Fetch data from source
client = ArcGISClient(url=self.config.SOURCE_URL)
raw_data = client.fetch_all_features()
# 2. Procedural, deterministic transformation
self.logger.info(f"Retrieved {len(raw_data)} records. Transforming...")
clean_data = self.transform_to_inspire(raw_data)
# 3. The SDK handles the atomic swap to the DB
self.logger.info("Loading into PostGIS Staging...")
self.atomic_load(
target_table="stg_inspire_ge_minocc",
data=clean_data
)
self.logger.info("Voyage successful!")
if __name__ == "__main__":
# The run() method handles dry-runs, retries, and DB sessions
MineralOccurrencesHarpoon().run()Zero-Downtime Data Delivery (The Atomic Swap)
Our primary consumer for this data is a public-facing pygeoapi service reading from a PostGIS database. If a Harpoon takes 20 minutes to download and process thousands of geological features, the public API cannot serve empty or partial data during that window.
Because we have full control over the procedural Python, our SDK enforces an Atomic Swap pattern:
Where We Go From (a possible roadmap)
The Wayler has proven so stable that we are already planning to roll it out for other internal synchronization processes beyond just geospatial data. However, any honest engineering team knows their system isn’t perfect yet (and will never be :D). We have two major items on our technical debt roadmap:
- A UI Frontend: Right now, managing the Wayler means running some docker commands and writing clinical SQL INSERT statements into the orchestrator’s database. We need to build a clean, pretty UI dashboard to visualize pipelines, trigger manual Harpoon runs, and monitor logs.
- Secret Management: Our next major architectural upgrade is to integrate a dedicated secrets manager, so Harpoons fetch their credentials dynamically at runtime (mage.ai for instance does this very well).
Conclusion
In an era where the tech industry is racing to hand over every workflow to an LLM, it’s worth remembering that data synchronization is often best served cold and deterministic. We love AI for what it does best. But for migrating data? I don’t think we need it. We embraced the sustainability of procedural Python, the strict boundaries of Docker, and the reliability of a custom in-house scheduler.
As a developer trying to be updated on LLM’s superpowers, I assume there is a small part of me that initially worried the “fun” of AI would be lost by choosing a traditional deterministic route over an autonomous agent swarm. But the truth is, the Harpoons still have to be developed. The architecture still needs to be designed, the APIs still need to be reverse-engineered, and the AI is sitting right there in our IDEs helping us write that procedural Python faster and better than ever before. We get the reliability of deterministic code, without losing the joy of “modern development”.
Which makes me thinking: sometimes, the most innovative thing you can do is write code that does exactly what you tell it to do, even if it’s a bug.
