Airflow with MySQL raising error on libraries

Airflow with MySQL as back end will raise error on libmysqlclient.so.1020 and let us see Why and How to resolve this errors with Apache Airflow

Apache Airflow is an opensource tool for orchestrating complex workflows and data processing pipelines. Workflow in Airflow is designed as a directed acylic graph(DAGs) meaning the tasks in a workflow can be split and executed independently and later be logically merged in a task




Airflow has a webserver, scheduler and worker as part of the Asudo find / -name libmysqlclient.so.1020pache Airflow installation and configuration

One of the basic Airflow commands to compile all dags of Airflow server is

                                                   airflow list_dags

Purpose of the above command is to compile the list of dags that are located in dags folder as per the airflow.cfg file


Issue: On issuing, airflow list_dags, we might encounter "ImportError: libmysqlclient.so.1020: cannot open shared object file: No such file or directory"

Cause: Reason for this error is that the library file libmysqlclient.so.1020 couldnt be located by the airflow service which is running with MySQL as the back end

Resolution:

1. Locate the location of the library libmysqlclient.so.1020 in the server where Airflow is installed
2. Set the library location in LD_LIBRARY_PATH variable

       From shell, execute the below commands

• sudo find / -name libmysqlclient.so.1020

                                        /usr/local/bin/libmysqlclient.so.1020

                                       /usr/lib64/mysql57/libmysqlclient.so.1020

• export LD_LIBRARY_PATH=/usr/lib64/mysql57/

                              Now add the path returned by find against LD_LIBRARY_PATH as above

Voila, it resolves the error !!!!!