Introduction

Most data platforms eventually reach the same problem. Jobs that once ran from a cron entry now depend on each other, run at different frequencies, fail for reasons nobody remembers, and need to be retried, backfilled, and monitored. A shell script and a crontab are no longer enough. What teams need is an orchestrator: a system that knows what should run, in which order, when, and what to do when something breaks.

Apache Airflow is one of the most widely adopted orchestrators for this job. It lets you describe workflows as code, schedule them, visualize their state, and recover from failures in a consistent way. It is written in Python, and its workflow definitions are Python files, which can make it look like a tool that lives outside the world of Java and Spring Boot. In practice, Java engineers use Airflow constantly: they trigger its workflows from services, consume its APIs, run the jobs it orchestrates, and operate the platforms it depends on.

This article explains what Airflow is, how its core concepts fit together, how to write and test a workflow, and how a Java backend can integrate with it through the REST API. The goal is not to turn you into a Python developer, but to give you a clear mental model so you can design better systems around Airflow.

The Problem Airflow Solves

Consider a typical nightly pipeline for a digital banking platform. It extracts transactions from several operational databases, validates them, aggregates them into daily balances, generates regulatory reports, and publishes the results to a data warehouse. Each step depends on the previous one. Some steps are slow, some are flaky, and some must not run if an upstream check fails.

Without an orchestrator, this usually turns into a chain of scripts with hard-coded timing. If the extraction takes longer than expected, the aggregation starts too early and produces incomplete numbers. If a step fails at 3 a.m., someone has to read logs, figure out where the process stopped, and rerun the right part without duplicating work. Backfilling a month of data becomes a manual, error-prone exercise.

An orchestrator addresses these issues by separating three concerns:

  • What needs to be done, expressed as tasks and their dependencies.
  • When it should run, expressed as schedules, triggers, or external events.
  • How failures are handled, expressed as retries, timeouts, alerts, and recovery rules.

Airflow models all three explicitly, and gives you a web interface to see what happened.

Core Concepts

Airflow is built on a small set of concepts. Understanding them makes the rest of the platform easier to reason about.

DAGs

A DAG (Directed Acyclic Graph) is a workflow. It is a collection of tasks with dependencies, and it has no cycles: a task cannot depend, directly or indirectly, on itself. Each DAG has a schedule, a start date, and a set of default arguments.

Tasks and Operators

A task is a unit of work inside a DAG. Tasks are created from operators, which are templates for common actions. Airflow ships with operators for running Python functions, executing shell commands, transferring files, and calling external systems. Providers add many more, covering cloud storage, databases, and messaging systems.

Dependencies

Dependencies define the execution order. In code, you express them with the >> operator or with methods such as set_downstream. Airflow uses these relationships to decide which tasks can run in parallel and which must wait.

Task Instances and DAG Runs

Each execution of a DAG is a DAG run, identified by a logical date. Each task within a run is a task instance, with its own state: queued, running, success, failed, skipped, or up for retry. This model is what allows Airflow to backfill historical data: it simply creates DAG runs for past logical dates.

Scheduler, Executor, and Metadata Database

The scheduler decides which task instances are ready to run and hands them to the executor. The executor determines where tasks run: locally, in a pool of Celery workers, or as Kubernetes pods. All state is stored in a metadata database, usually PostgreSQL in production.

Knowing this split is important operationally. The scheduler and the database are the heart of the platform, and they need the same care you would give any critical backend component.

A First Workflow

Airflow workflows are Python files placed in the dags folder. Here is a minimal example that runs three dependent tasks every day:

from datetime import datetime, timedelta from airflow import DAG from airflow.operators.python import PythonOperator def extract_transactions(**context): print(f"Extracting transactions for {context['ds']}") def validate_transactions(**context): print("Running validation checks") def publish_balances(**context): print("Publishing daily balances") default_args = { "owner": "data-platform", "retries": 2, "retry_delay": timedelta(minutes=5), } with DAG( dag_id="daily_balances", default_args=default_args, start_date=datetime(2026, 1, 1), schedule="0 2 * * *", catchup=False, tags=["banking", "reporting"], ) as dag: extract = PythonOperator( task_id="extract_transactions", python_callable=extract_transactions, ) validate = PythonOperator( task_id="validate_transactions", python_callable=validate_transactions, ) publish = PythonOperator( task_id="publish_balances", python_callable=publish_balances, ) extract >> validate >> publish

A few details deserve attention:

  • schedule="0 2 * * *" runs the DAG every day at 02:00.
  • catchup=False prevents Airflow from creating runs for every missed interval since the start date. In production, this default is often what you want for new pipelines, but backfilling must then be an explicit decision.
  • retries and retry_delay handle transient failures, such as a database connection that times out once.
  • The >> chain defines a strict sequence. Publishing never starts unless validation succeeded.

Save this file, and Airflow's scheduler will parse it, register the DAG, and start scheduling it.

Designing Good Tasks

The example above is intentionally simple. Real workflows need more discipline. These practices make a large difference over time.

Keep tasks idempotent. If a task runs twice for the same logical date, the result should be the same as running it once. Write to partitions keyed by date and overwrite them, instead of appending rows blindly. Idempotency is what makes retries and backfills safe.

Keep tasks small but meaningful. A task should represent a recoverable step. If one task performs extraction, transformation, and publication, a failure in publication forces you to repeat everything. Splitting the work lets Airflow retry only the part that failed.

Avoid heavy computation inside the scheduler. DAG files are parsed frequently. Top-level code that queries databases or makes network calls slows the whole platform down. Put work inside tasks, not at module level.

Pass data by reference, not by value. Airflow stores small pieces of data in XCom, its mechanism for sharing values between tasks. Large datasets belong in object storage or a database, and tasks should pass only the location or identifier.

Make failures visible. Configure alerts through callbacks or notifications, and use SLAs where delays matter. A pipeline that fails silently is worse than one that fails loudly.

Testing Workflows

Because DAGs are code, they can be tested like code. At minimum, verify that every DAG imports without errors and that the dependency graph is what you expect. A simple test with DagBag catches most mistakes before deployment:

from airflow.models import DagBag def test_dags_load_without_errors(): dag_bag = DagBag(dag_folder="dags", include_examples=False) assert not dag_bag.import_errors, f"Import errors: {dag_bag.import_errors}" def test_daily_balances_structure(): dag_bag = DagBag(dag_folder="dags", include_examples=False) dag = dag_bag.get_dag("daily_balances") assert dag is not None assert dag.task_dict["validate_transactions"] in dag.task_dict["extract_transactions"].downstream_list

For tasks that call external systems, isolate the business logic in plain Python functions and test them independently with mocks. Keep the Airflow-specific code thin. This is the same separation of concerns you would apply to a Spring service: controllers stay small, and the logic lives in testable components.

Integrating Airflow with Java Services

Here is where many Java teams start to interact with Airflow. A backend service might need to trigger a pipeline after a business event, such as the end of a customer onboarding batch, or check whether the latest report has finished. Airflow exposes a stable REST API for these operations.

To trigger a DAG run, you send a POST request to the DAG runs endpoint. Authentication depends on your deployment, but a common approach is a bearer token issued for a service account. A minimal Java client using the standard HttpClient looks like this:

import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.time.Instant; public class AirflowDagTrigger { private final HttpClient httpClient = HttpClient.newHttpClient(); private final String baseUrl; private final String token; public AirflowDagTrigger(String baseUrl, String token) { this.baseUrl = baseUrl; this.token = token; } public String triggerDagRun(String dagId, String conf) throws Exception { String body = """ { "conf": %s, "logical_date": "%s" } """.formatted(conf, Instant.now().toString()); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(baseUrl + "/api/v1/dags/" + dagId + "/dagRuns")) .header("Authorization", "Bearer " + token) .header("Content-Type", "application/json") .POST(HttpRequest.BodyPublishers.ofString(body)) .build(); HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 300) { throw new IllegalStateException( "Airflow returned " + response.statusCode() + ": " + response.body()); } return response.body(); } }

A few practices make this integration robust:

  • Use a dedicated service account with the minimum permissions needed to trigger specific DAGs.
  • Pass identifiers in conf, not full payloads. The DAG can fetch the details from your system.
  • Handle conflicts. Triggering a run with a logical date that already exists returns an error. Decide whether your service should reuse, skip, or fail in that case.
  • Poll or subscribe carefully. To check run status, query the DAG run endpoint with exponential backoff rather than polling in a tight loop.
  • Treat Airflow as an external dependency. Add timeouts, circuit breakers, and clear error messages, just as you would for any downstream system.

If your Spring Boot application is the caller, wrap this logic in a dedicated client component, expose it through a service interface, and cover it with integration tests against a local Airflow instance.

Operating Airflow in Production

Running Airflow well is a platform concern, and several areas need attention.

Database and scheduler health. The metadata database receives constant writes from the scheduler and workers. Monitor its connections, growth, and query latency. Plan a cleanup policy for old task instances and logs, because an unpruned database slows the entire platform.

Executor choice. Start simple. A local or Celery executor fits many teams. Kubernetes executors provide isolation and elastic capacity but add operational complexity. Choose based on workload, not on what is fashionable.

Pools and concurrency. Use pools to limit how many tasks can hit a fragile system at once, such as a legacy database. Without pools, a burst of parallel tasks can overload a dependency and cause the very failures you are trying to recover from.

Secrets and connections. Store credentials in Airflow connections or in an external secrets backend. Never hard-code passwords in DAG files, which are often visible to many engineers.

Version upgrades. Test upgrades in a staging environment with a copy of your DAGs. Check provider compatibility, deprecated operators, and changes in the scheduler behavior before applying them to production.

Observability. Export metrics to your monitoring stack, track DAG parsing time, scheduler heartbeat, and task failure rates, and define alerts for the conditions that matter to the business, such as a daily report that has not completed by its deadline.

Best Practices Checklist

  • Model workflows explicitly as DAGs with clear, documented dependencies.
  • Make every task idempotent and safe to retry.
  • Keep DAG files lightweight and free of heavy top-level code.
  • Pass references, not large payloads, between tasks.
  • Set retries, timeouts, and alerts deliberately for each pipeline.
  • Test DAG imports and structure in continuous integration.
  • Store secrets outside DAG code.
  • Limit concurrency against fragile dependencies with pools.
  • Give external callers, including Java services, a dedicated and minimally privileged account.
  • Monitor the scheduler, the database, and business-level deadlines, not only task status.

Conclusion

Apache Airflow gives data and platform teams a disciplined way to describe, schedule, and recover complex workflows. Its Python-based DAGs can look unfamiliar to Java engineers, but the underlying ideas are familiar: explicit dependencies, idempotent operations, clear configuration, testing, and operational monitoring. Those are the same principles that make backend systems reliable.

If your organization runs batch processing, reporting, or data synchronization, Airflow is worth evaluating. Start with one pipeline, make it idempotent and observable, and integrate it with your existing services through a well-designed API client. Once the first workflow runs reliably, the rest of the platform tends to follow.

Key Takeaways

  • Airflow orchestrates workflows as DAGs, separating what runs, when it runs, and how failures are handled.
  • Idempotent, small tasks make retries and backfills safe.
  • Tests for DAG imports and structure catch most errors before deployment.
  • Java services can trigger and monitor workflows through the stable REST API.
  • Production success depends on the metadata database, pools, secrets management, and observability.