airflow快速安装——MySQL

1) 进行pip的更新，否则很多安装会报错

pip install --upgrade pip
pip install --upgrade setuptools

2) 安装开发库

yum install python3-devel
yum install libevent-devel
yum install mysql-devel

3) 创建airflow的MySQL数据库schema及用户

create database apache_airflow;
create user 'airflow'@'%' identified by '123123';
GRANT all privileges on apache_airflow.* TO 'airflow'@'%'  IDENTIFIED BY '123123';
FLUSH PRIVILEGES;
set explicit_defaults_for_timestamp = 1; --这一行至关重要

4) 安装airflow

# airflow 需要 home 目录，默认是~/airflow，
# 但是如果你需要，放在其它位置也是可以的
# (可选)
export AIRFLOW_HOME = ~/airflow

# 使用 pip 从 pypi 安装
pip3 install apache-airflow

5) 编辑airflow.cfg配置文件，一般在~/airflow/airflow.cfg

#sql_alchemy_conn = sqlite:////data/airflow/airflow.db
sql_alchemy_conn = mysql://airflow:123456@127.0.0.1:3306/apache_airflow

6) 初始化MySQL数据库

> airflow initdb
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 405, in initialize
    configure_orm()
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 294, in configure_orm
    engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
    return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'

说明缺少MySQL组件

> yum install -y mysql-devel
> pip3 install pymysql
> pip3 install mysql

> airflow initdb
DB: mysql://airflow:***@10.1.11.12:3306/shark_airflow
[2020-08-27 09:25:52,760] {db.py:378} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.models.crypto] cryptography not found - values will not be stored encrypted.
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
Done.

7) 启动web服务器和调度器

> airflow webserver -p 8080
> airflow scheduler

8) 防止密码明文存储

pip3 install cryptography

参考：https://blog.csdn.net/gqj54082161/article/details/103756354

最后编辑于：2020.09.01 17:08:41