{"id":37035,"date":"2023-03-18T16:43:31","date_gmt":"2023-03-18T16:43:31","guid":{"rendered":"https:\/\/naiveskill.com\/?p=37035"},"modified":"2023-03-18T16:44:48","modified_gmt":"2023-03-18T16:44:48","slug":"airflow-operators","status":"publish","type":"post","link":"https:\/\/naiveskill.com\/airflow-operators\/","title":{"rendered":"Getting to Know Airflow Operators: A Comprehensive Guide in 2023"},"content":{"rendered":"\n<p>In this blog, we will learn about <strong>Airflow operators<\/strong>, what they are, and how they are used in Airflow workflows. We will explore various examples of Airflow operators to understand their functionality better. Let&#8217;s dive in!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What are operators in airflow?<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"256\" src=\"https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow-1024x256.webp\" alt=\"What are operators in airflow\" class=\"wp-image-37102\" srcset=\"https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow-1024x256.webp 1024w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow-300x75.webp 300w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow-768x192.webp 768w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow-150x38.webp 150w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/What-are-operators-in-airflow.webp 1200w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>In Apache <a href=\"https:\/\/airflow.apache.org\/\" data-type=\"URL\" data-id=\"https:\/\/airflow.apache.org\/\" target=\"_blank\" rel=\"noopener\">Airflow<\/a>, operators are the building blocks of workflows representing a single task in a DAG. Operators define the actions to be taken when a task is executed<\/p>\n\n\n\n<p>Operators can be used to define tasks that perform a wide range of actions, such as:<\/p>\n\n\n\n<ul>\n<li>Extracting data from a source system<\/li>\n\n\n\n<li>Transforming data<\/li>\n\n\n\n<li>Loading data<\/li>\n\n\n\n<li>Running a Python function<\/li>\n\n\n\n<li>Running a Bash command<\/li>\n\n\n\n<li>Triggering an external process or application<\/li>\n<\/ul>\n\n\n\n<p>Airflow provides a variety of built-in operators that cover many common use cases, such as BashOperator, PythonOperator, etc. Additionally, users can create their own custom operators to handle the specific requirement.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Airflow operators list<\/h2>\n\n\n\n<p>Apache Airflow provides a wide variety of operators that can be used to define tasks in a DAG. Below is the list of most commonly used operators:<\/p>\n\n\n\n<ol>\n<li><strong>BashOperator<\/strong>: Executes a bash command or script.<\/li>\n\n\n\n<li><strong>PythonOperator<\/strong>: Executes a Python function.<\/li>\n\n\n\n<li><strong>DummyOperator<\/strong>: Does nothing. Useful as a placeholder or for testing.<\/li>\n\n\n\n<li><strong>EmailOperator<\/strong>: Sends an email.<\/li>\n\n\n\n<li><strong>PostgresOperator<\/strong>: Executes a PostgreSQL query.<\/li>\n\n\n\n<li><strong>S3FileTransferOperator<\/strong>: Transfers files to\/from Amazon S3.<\/li>\n\n\n\n<li><strong>MySqlOperator<\/strong>: Executes a MySQL query.<\/li>\n\n\n\n<li><strong>SparkSubmitOperator<\/strong>: Submits a Spark job to a cluster.<\/li>\n\n\n\n<li><strong>SSHOperator<\/strong>: Executes a command on a remote SSH server.<\/li>\n\n\n\n<li><strong>SubDagOperator:<\/strong> Executes a sub-DAG.<\/li>\n\n\n\n<li><strong>SlackAPIOperator<\/strong>: Sends a message to a Slack channel.<\/li>\n\n\n\n<li><strong>KubernetesPodOperator<\/strong>: Executes a task in a Kubernetes pod.<\/li>\n\n\n\n<li><strong>HiveOperator<\/strong>: Executes a Hive query.<\/li>\n\n\n\n<li><strong>HdfsSensor<\/strong>: Waits for a file to be added to HDFS.<\/li>\n\n\n\n<li><strong>FileSensor<\/strong>: Waits for a file to be added to a specified directory.<\/li>\n<\/ol>\n\n\n\n<p>These are just a few of the many operators available in Airflow. Users can also create their own custom operators by subclassing the BaseOperator class.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Airflow operator example<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"256\" src=\"https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example-1024x256.webp\" alt=\"\" class=\"wp-image-37104\" srcset=\"https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example-1024x256.webp 1024w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example-300x75.webp 300w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example-768x192.webp 768w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example-150x38.webp 150w, https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operator-example.webp 1200w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Below is an example of a simple BashOperator in an airflow DAG to execute a bash command:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from airflow import DAG\nfrom datetime import datetime, timedelta\nfrom airflow.operators.bash import BashOperator\nfrom datetime import datetime\n\n\ntest_dag = DAG(\n    'test_operator',\n    start_date=datetime(2023, 3, 15)\n)\n\nbash_task = BashOperator(\n    task_id='test_operator',\n    bash_command='echo \"Test bash\"',\n    dag=test_dag\n)\n\nbash_task<\/pre>\n\n\n\n<p>The code defines an Airflow DAG named &#8216;test_operator&#8217; with a start date of March 15th, 2023. The DAG has a single task of type BashOperator named &#8216;test_operator&#8217;, which simply prints the string &#8216;Test bash&#8217; to the console using the &#8216;echo&#8217; command in a Bash shell.<\/p>\n\n\n\n<p>When the DAG is executed, the BashOperator task will be triggered and it will run the specified bash command. The BashOperator is a simple operator that can be used to execute arbitrary Bash scripts or commands as a task in an Airflow DAG.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Important airflow operators<\/h2>\n\n\n\n<p>In this session, I will briefly explain a few of the widely used airflow operators.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Airflow python operator<\/h3>\n\n\n\n<p>The PythonOperator executes a Python callable task within a DAG. A PythonOperator allows you to define a custom Python function that performs some specific and then run that function as a task in your DAG.<\/p>\n\n\n\n<p>Let&#8217;s take the below example<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from airflow import DAG\nfrom airflow.operators.python_operator import PythonOperator\nfrom datetime import datetime\n\n\ndef test_python__function():\n    print(\"hello python\")\n\n\nmy_dag = DAG(\n    'sample_python_dag',\n    start_date=datetime(2023, 3, 1),\n    schedule_interval='@daily'\n)\npython_task = PythonOperator(\n    task_id='python_task',\n    python_callable=test_python__function,\n    dag=my_dag\n)<\/pre>\n\n\n\n<p>Refer my tutorial on <a href=\"https:\/\/naiveskill.com\/airflow-pythonoperator\/\" data-type=\"URL\" data-id=\"https:\/\/naiveskill.com\/airflow-pythonoperator\/\">airflow pythonOperator<\/a> to get more information.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Airflow Bash operator<\/h3>\n\n\n\n<p>The&nbsp;<strong>Airflow BashOperator<\/strong>&nbsp;is a basic operator in Apache Airflow that allows you to execute a Bash command or shell script within an Airflow DAG.<\/p>\n\n\n\n<p>Below is the sample BashOperator airflow DAG:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from airflow import DAG\nfrom airflow.operators.bash import BashOperator\nfrom datetime import datetime\n\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': False,\n    'start_date': datetime(2023, 2, 15),\n    'retries': 0,\n}\ntest_dag = DAG(\n    'test_bash_dag',\n    default_args=default_args,\n    schedule_interval=timedelta(days=1)\n)\n# Define the BashOperator task\nbash_task = BashOperator(\n    task_id='bash_task_execute_script',\n    bash_command='echo \"Hello world\"',\n    dag=test_dag\n)\n# Set task dependencies\nbash_task<\/pre>\n\n\n\n<p>Follow my tutorial on <a href=\"https:\/\/naiveskill.com\/airflow-bashoperator\/\" data-type=\"URL\" data-id=\"https:\/\/naiveskill.com\/airflow-bashoperator\/\">airflow BashOperator<\/a> to get more details<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Airflow spark operator<\/h3>\n\n\n\n<p>The Spark Operator allows you to submit Apache Spark jobs as part of your Airflow DAGs to a Spark cluster.<\/p>\n\n\n\n<p>To use the Spark Operator, you need to configure it with the Spark connection information, such as the Spark master URL, the location of the Spark binaries, and any Spark configuration properties.<\/p>\n\n\n\n<p>When the SparkSubmitOperator is executed as part of your DAG, it creates a Spark application and submits it to the Spark cluster using the configured connection information<\/p>\n\n\n\n<p>Let&#8217;s take the below example:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator\n\ndefault_args = {\n    'owner': 'Naiveskill',\n    'depends_on_past': False,\n    'start_date': datetime(2023, 3, 7),\n}\n\nmy_dag = DAG(\n    'spark_example',\n    default_args=default_args, \n    schedule_interval='@daily'\n)\n\nspark_test_task = SparkSubmitOperator(\n    task_id='spark_test_job',\n    conn_id='spark_default',\n    application='\/&lt;path_to_your_spark_job.py>',\n    name='test_spark_job',\n    executor_memory='2g',\n    num_executors=4,\n    driver_memory='1g',\n    conf={\n        'spark.app.name': 'my_spark_job',\n        'spark.executorEnv.PYTHONPATH': '\/usr\/lib\/python3.7\/site-packages'\n    },\n    dag=my_dag,\n)\n\nspark_test_task<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Airflow Kubernetes operator<\/h3>\n\n\n\n<p>The Kubernetes Operator allows you to run Kubernetes objects, such as pods and deployments, as part of your Airflow DAG. To use the Kubernetes Operator, you need to configure it with the connection information for your Kubernetes cluster, such as the cluster API server URL.<\/p>\n\n\n\n<p>Then, you can define a KubernetesPodOperator to specify the Kubernetes object to run and any necessary configuration options.<\/p>\n\n\n\n<p>The KubernetesPodOperator creates and runs a Kubernetes Pod. It allows you to specify the Docker image to use, the command to run in the container, the arguments to pass to the command, and any necessary environment variables and volume mounts.<\/p>\n\n\n\n<p>When the Kubernetes operator is executed as part of your DAG, it creates and submits the specified Kubernetes object to the Kubernetes cluster using the configured connection information.<\/p>\n\n\n\n<p>Let&#8217;s take the below example<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator\nfrom kubernetes.client import models as k8s\n\ndefault_args = {\n    'owner': 'naiveskill',\n    'depends_on_past': False,\n    'start_date': datetime(2023, 3, 7),\n}\n\nmy_dag = DAG('kubernetes_example', default_args=default_args, schedule_interval='@daily')\n\nk8s_task = KubernetesPodOperator(\n    task_id='kubernetes_sample_pod',\n    name='kubernetes_sample_pod',\n    image='&lt;docker_image_name:latest>',\n    cmds=['python', '\/path\/to\/my\/test_script.py'],\n    arguments=['--arg1', 'value1', '--arg2', 'value2'],\n    env_vars={\n        'ENV_VAR_1': 'value1',\n        'ENV_VAR_2': 'value2',\n    },\n    volumes=[\n        k8s.V1Volume(\n            name='my_volume',\n            persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(\n                claim_name='PVC_name',\n            ),\n        ),\n    ],\n    dag=my_dag,\n)\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>I hope you have liked this tutorial on airflow operators. Please do let me know if you are facing any issues while following along.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">More to explore<\/h2>\n\n\n\n<p><a href=\"https:\/\/naiveskill.com\/airflow-scheduler\/\">Airflow Schedular<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/naiveskill.com\/airflow-database\/\">Airflow database<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/naiveskill.com\/airflow-max_active_runs\/\">Understand airflow max_active_runs<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/naiveskill.com\/airflow-shortcircuitoperator\/\" data-type=\"URL\" data-id=\"https:\/\/naiveskill.com\/airflow-shortcircuitoperator\/\">Airflow ShortCircuitOperator<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog, we will learn about Airflow operators, what they are, and how they are used in Airflow workflows. We will explore various examples of Airflow operators to understand &hellip;<\/p>\n<p class=\"read-more\"> <a class=\"\" href=\"https:\/\/naiveskill.com\/airflow-operators\/\"> <span class=\"screen-reader-text\">Getting to Know Airflow Operators: A Comprehensive Guide in 2023<\/span> Read More \u00bb<\/a><\/p>\n","protected":false},"author":1,"featured_media":37103,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"","footnotes":""},"categories":[41,68],"tags":[42,72],"uagb_featured_image_src":{"full":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators.webp",1200,628,false],"thumbnail":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-150x150.webp",150,150,true],"medium":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-300x157.webp",300,157,true],"medium_large":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-768x402.webp",768,402,true],"large":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-1024x536.webp",1024,536,true],"1536x1536":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators.webp",1200,628,false],"2048x2048":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators.webp",1200,628,false],"web-stories-poster-portrait":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-640x628.webp",640,628,true],"web-stories-publisher-logo":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-96x96.webp",96,96,true],"web-stories-thumbnail":["https:\/\/naiveskill.com\/wp-content\/uploads\/2023\/03\/Airflow-operators-150x79.webp",150,79,true]},"uagb_author_info":{"display_name":"naivetechblog@gmail.com","author_link":"https:\/\/naiveskill.com\/author\/naivetechbloggmail-com\/"},"uagb_comment_info":9,"uagb_excerpt":"In this blog, we will learn about Airflow operators, what they are, and how they are used in Airflow workflows. We will explore various examples of Airflow operators to understand &hellip; Getting to Know Airflow Operators: A Comprehensive Guide in 2023 Read More \u00bb","_links":{"self":[{"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/posts\/37035"}],"collection":[{"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/comments?post=37035"}],"version-history":[{"count":15,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/posts\/37035\/revisions"}],"predecessor-version":[{"id":37105,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/posts\/37035\/revisions\/37105"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/media\/37103"}],"wp:attachment":[{"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/media?parent=37035"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/categories?post=37035"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/naiveskill.com\/wp-json\/wp\/v2\/tags?post=37035"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}