Airflow
Frequently used code for airflow related code snippets
Last updated
Was this helpful?
Frequently used code for airflow related code snippets
Last updated
Was this helpful?
Was this helpful?
By using DockerOperators/KubernetesOperators only we can avoid the technical debt that would existing by using different operators, and the dependency on Airflow development of operators(more on this).
Activate the execution of a DAG
airflow unpause dag_id
Setting it up in Amazon linux AMI
#!/bin/bash
SLUGIFY_USES_TEXT_UNIDECODE=yes pip install apache-airflow[s3,postgres]
# sudo su
yum install postgresql postgresql-contrib postgresql-server
/etc/init.d/postgresql92 initdb
/etc/init.d/postgresql92 start
Triggering dags from python client
The following nodes are available with the given host names:
master1: Will have the role(s): Web Server, Scheduler
master2: Will have the role(s): Web Server
worker1: Will have the role(s): Worker
worker2: Will have the role(s): Worker
A Queuing Service is Running. (RabbitMQ, AWS SQS, etc)
You can install RabbitMQ by following these instructions:
Installing RabbitMQ
If youโre using RabbitMQ, it is recommended that it is also setup to be a cluster for High Availability. Setup a Load Balancer to proxy requests to the RabbitMQ instances.
Scale workers vertically by providing higher values to celeryd_concurrency