Jupyter notebook 使用jupyter笔记本电脑内的气流挂钩

我使用postgres后端运行气流-一切正常。此外，我在运行Airflow的同一台主机上运行Jupyter服务器。现在我想我可以从笔记本电脑上取下气流挂钩了

import pandas as pd
import numpy as np
import matplotlib as plt
from airflow.hooks.mysql_hook import MySqlHook

mysql = MySqlHook(mysql_conn_id = 'mysql-x')
sql = "select 1+1"

mysql.get_pandas_df(sql)

但我得到一个例外信息：

OperationalError: (sqlite3.OperationalError) no such table: connection
[SQL: SELECT connection.password AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted AS connection_is_extra_encrypted 
FROM connection 
WHERE connection.conn_id = ?]
[parameters: ('mysql-x',)]
(Background on this error at: http://sqlalche.me/e/e3q8)

但让我怀疑的是，它不仅找不到连接id（我可以在UI中清楚地看到）。它还说：sqlite3.OperationalError——看起来它甚至没有连接到同一个postgres数据库。我已经检查了

os.environ[“AIRFLOW\u HOME”]

，这似乎是正确的

编辑1：

在我启动jupyter笔记本服务器之后，在设置了所有环境变量之后，我得到了一个不同的错误：

/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py in __get__(self, instance, owner)
    351 
    352         def __get__(self, instance, owner):
--> 353             retval = self.descriptor.__get__(instance, owner)
    354             # detect if this is a plain Python @property, which just returns
    355             # itself for class level access.  If so, then return us.

/usr/local/lib/python3.7/site-packages/airflow/models/connection.py in get_password(self)
    188                     "Can't decrypt encrypted password for login={}, \
    189                     FERNET_KEY configuration is missing".format(self.login))
--> 190             return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
    191         else:
    192             return self._password

/usr/local/lib/python3.7/site-packages/cryptography/fernet.py in decrypt(self, msg, ttl)
    169             except InvalidToken:
    170                 pass
--> 171         raise InvalidToken

InvalidToken: 

您可以使用此dockerfile复制它：

FROM apache/airflow

USER root

# install mysql client
RUN apt-get update && apt-get install -y mariadb-client-10.3 unzip

# install mssql client and tools
RUN apt-get install -y curl gnupg libicu-dev libicu63
RUN curl https://packages.microsoft.com/keys/microsoft.asc -o key
RUN apt-key add < key
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/msprod.list

ENV ACCEPT_EULA=Y
RUN apt-get update && apt-get install -y mssql-tools msodbcsql17 unixodbc unixodbc-dev unzip libunwind8
RUN curl -Lq 'https://go.microsoft.com/fwlink/?linkid=2108814' -o sqlpackage-linux-x64-latest.zip
RUN mkdir /opt/sqlpackage/ && unzip sqlpackage-linux-x64-latest.zip -d /opt/sqlpackage/
RUN chmod a+x /opt/sqlpackage/sqlpackage && ln -sfn /opt/sqlpackage/sqlpackage /usr/bin/sqlpackage
RUN ln -sfn /opt/mssql-tools/bin/sqlcmd /usr/bin/sqlcmd

# install notebooks
RUN pip install jupyterlab pandas numpy scikit-learn matplotlib pymssql

#RUN cat /entrypoint.sh
# start additional notebok server
# this is a dirty hack but for the sake of this prototype good enough
RUN sed -i -e's/\# Run the command/airflow scheduler \& \njupyter notebook --ip=0.0.0.0 --port=9000 --NotebookApp.token="" --NotebookApp.password="" \& \n/' /entrypoint 

EXPOSE 9000

# switch back to airflow user 
USER airflow

RUN airflow initdb
RUN alias ll='ls -al'

来自apache/API的

用户根
#安装mysql客户端
运行apt-get-update&&apt-get-install-y-mariadb-client-10.3解压
#安装mssql客户端和工具
运行apt get install-y curl gnupg libicu dev libicu63
长卷https://packages.microsoft.com/keys/microsoft.asc -o键
运行apt key add/etc/apt/sources.list.d/msprod.list
ENV ACCEPT_EULA=Y
运行apt-get-update&&apt-get-install-y mssql工具msodbcsql17 unixodbc unixodbc dev-unzip libunwind8
运行curl-Lq'https://go.microsoft.com/fwlink/?linkid=2108814'-o sqlpackage-linux-x64-latest.zip
运行mkdir/opt/sqlpackage/&解压缩sqlpackage-linux-x64-latest.zip-d/opt/sqlpackage/
运行chmoda+x/opt/sqlpackage/sqlpackage&&ln-sfn/opt/sqlpackage/sqlpackage/usr/bin/sqlpackage
运行ln-sfn/opt/mssqltools/bin/sqlcmd/usr/bin/sqlcmd
#安装笔记本电脑
运行pip安装jupyterlab pandas numpy scikit学习matplotlib pymssql
#运行cat/entrypoint.sh
#启动其他notebok服务器
#这是一个肮脏的黑客，但为了这个原型足够好
运行sed-i-e的/\ \运行命令/气流调度器\&\njupyter notebook--ip=0.0.0.0--port=9000--NotebookApp.token=“”--NotebookApp.password=“”\&\n/”/entrypoint
暴露9000
#切换回“气流用户”
用户气流
运行initdb
运行别名ll='ls-al'

您还需要至少

AIRFLOW\uuuu CORE\uuuu SQL\u ALCHEMY\u CONN

环境，否则AIRFLOW将默认为本地sqlite3数据库。Thx！这是因为我启动笔记本服务器太早（在设置所有环境之前）。但是现在我得到了“InvalidToken”，但是我可以在

os.environ[“AIRFLOW\uuu CORE\uuu FERNET\u KEY”

中看到一个标记。您的AIRFLOW.cfg文件是否与

AIRFLOW\u HOME

环境变量位于同一目录中？是，系统中唯一的airflow.cfg位于airflow\u HOME/Location您还需要至少

airflow\u CORE\u\u SQL\u ALCHEMY\u CONN

环境，否则airflow将默认为本地sqlite3数据库.Thx！这是因为我启动笔记本服务器太早（在设置所有环境之前）。但是现在我得到了“InvalidToken”，但我可以在

os.environ[“AIRFLOW\u CORE\u FERNET\u KEY”]

中看到一个标记。您的AIRFLOW.cfg文件是否与

AIRFLOW\u HOME

环境变量位于同一目录中？是的，系统中唯一的AIRFLOW.cfg位于AIRFLOW\u HOME/位置