Airflow with MySQL as back end will raise error on libmysqlclient.so.1020 and let us see Why and How to resolve this errors with Apache Airflow
Apache Airflow is an opensource tool for orchestrating complex workflows and data processing pipelines. Workflow in Airflow is designed as a directed acylic graph(DAGs) meaning the tasks in a workflow can be split and executed independently and later be logically merged in a task
Airflow has a webserver, scheduler and worker as part of the Asudo find / -name libmysqlclient.so.1020pache Airflow installation and configuration
One of the basic Airflow commands to compile all dags of Airflow server is
airflow list_dagsPurpose of the above command is to compile the list of dags that are located in dags folder as per the airflow.cfg file
Issue: On issuing, airflow list_dags, we might encounter "ImportError: libmysqlclient.so.1020: cannot open shared object file: No such file or directory"
Cause: Reason for this error is that the library file libmysqlclient.so.1020 couldnt be located by the airflow service which is running with MySQL as the back end
- Locate the location of the library libmysqlclient.so.1020 in the server where Airflow is installed
- Set the library location in LD_LIBRARY_PATH variable
From shell, execute the below commands
- sudo find / -name libmysqlclient.so.1020
- export LD_LIBRARY_PATH=/usr/lib64/mysql57/
Now add the path returned by find against LD_LIBRARY_PATH as above
Voila, it resolves the error !!!!!