Django 让virtualenv中的PyPy使用操作系统中的共享库,而不是它自己的副本(libsqlite3)

Django 让virtualenv中的PyPy使用操作系统中的共享库,而不是它自己的副本(libsqlite3),django,continuous-integration,shared-libraries,virtualenv,pypy,Django,Continuous Integration,Shared Libraries,Virtualenv,Pypy,在我正在开发的基于Python/Django的应用程序中,我们使用Travis和GH操作进行CI。我们支持PyPy,因此也为PyPy运行CI测试。几个月以来,我们再也无法成功地运行那些PyPy测试了,因为我们一直遇到这样的错误:OSError:cannotloadlibgdal.so.20:/usr/lib/libgdal.so.20:undefined symbol:sqlite3\u column\u table\u name,如果我们运行Django的manage.py test命令(在文

在我正在开发的基于Python/Django的应用程序中,我们使用Travis和GH操作进行CI。我们支持PyPy,因此也为PyPy运行CI测试。几个月以来,我们再也无法成功地运行那些PyPy测试了,因为我们一直遇到这样的错误:
OSError:cannotloadlibgdal.so.20:/usr/lib/libgdal.so.20:undefined symbol:sqlite3\u column\u table\u name
,如果我们运行Django的
manage.py test
命令(在文章末尾进行回溯),就会出现这种情况。Django中的一些GIS相关功能需要GDAL库,而GDAL库又需要SQLite3库。而且它似乎要求在编译sqlite3时启用列元数据

只有在编译安装的
sqlite3
库而不使用
SQLITE\u ENABLE\u COLUMN\u元数据时,才可以在本地复制此数据。搜索CI服务器上所有已安装的
sqlite3
库后,PyPy显然已安装了自己的
libsqlite3.so.0
副本,在运行时,PyPy显然比OS安装版本更喜欢该副本,尽管
ldd
会引用OS库:

$ ldd -d /usr/lib/libgdal.so | grep sqlite3
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x00007f09d6124000)
我怀疑PyPy在运行时使用不同库(它自己的副本)的原因是它的动态加载程序(在回溯中引用)和它的外观。我怀疑它自己的
libsqlite3.so.0的副本是在CI提供程序安装PyPy时创建的

我发现解决此问题的唯一方法是在运行PyPy测试之前显式预加载OS级库:

export LD_PRELOAD=/usr/local/lib/libsqlite3.so.0
然而,这感觉像一个黑客,我想知道是否有更好的方法来做到这一点?我可以让PyPy(或virtualenv)只使用操作系统库而不是自己的副本,或者至少让它更新副本吗

未定义符号错误的回溯

$ coverage run manage.py test catmaid.tests
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 121, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 325, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/options.py", line 208, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 207, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/travis/build/[secure]/CATMAID/django/lib/custom_postgresql_psycopg2/base.py", line 33, in <module>
    from django.contrib.gis.db.backends.postgis.base import DatabaseWrapper as PostGISDatabaseWrapper
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
    from .features import DatabaseFeatures
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
    from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
    from django.contrib.gis.db.models import aggregates
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
    import django.contrib.gis.db.models.functions  # NOQA
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/functions.py", line 3, in <module>
    from django.contrib.gis.db.models.fields import BaseSpatialField, GeometryField
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
    from django.contrib.gis import forms, gdal
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
    from .fields import (  # NOQA
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
    from django.contrib.gis.gdal import GDALException
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
    from django.contrib.gis.gdal.datasource import DataSource
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
    from django.contrib.gis.gdal.driver import Driver
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
    from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
    from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/libgdal.py", line 46, in <module>
    lgdal = CDLL(lib_path)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/ctypes/__init__.py", line 350, in __init__
    pypy_dll = _ffi.CDLL(name, mode)
OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name
$coverage run manage.py test catmaid.tests
回溯(最近一次呼叫最后一次):
文件“manage.py”,第11行,在
从命令行(sys.argv)执行命令
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/core/management/_init__.py”,第401行,从命令行执行
utility.execute()
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/core/management/_init__.py”,第377行,在execute中
django.setup()
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/_init__.py”,第24行,在安装程序中
应用程序。填充(设置。已安装的应用程序)
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/registry.py”,第114行,填充
app_config.import_models()
导入模型中的文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/apps/config.py”,第211行
self.models\u module=导入模块(models\u module\u name)
导入模块中的文件“/opt/python/pypypy3.6-7.3.1/lib python/3/importlib/_init__uuu.py”,第126行
return _bootstrap._gcd_import(名称[级别:],包,级别)
文件“”,第1003行,在\u gcd\u导入中
文件“”,第980行,在“查找”和“加载”中
文件“”,第964行,在“查找”和“加载”中解锁
文件“”,第674行,在“加载”中
exec模块中的第691行文件“/冻结导入库.\u引导\u外部”
文件“”,第228行,在调用中删除了帧
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/auth/models.py”,第2行,在
从django.contrib.auth.base_user导入AbstractBaseUser、BaseUserManager
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/auth/base_user.py”,第47行,在
类AbstractBaseUser(models.Model):
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/models/base.py”,第121行,新__
新建类。将类添加到类(“元”,选项(元,应用标签))
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/models/base.py”,第325行,添加到类中
值。贡献给类(cls,名称)
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/models/options.py”,第208行,贡献给类
self.db\u table=truncate\u name(self.db\u table,connection.ops.max\u name\u length())
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/_init__.py”,第28行,在__
返回getattr(连接[默认\u DB\u别名],项)
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/utils.py”,第207行,在__
后端=加载\u后端(db['ENGINE'])
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/db/utils.py”,第111行,在load_后端
返回导入\u模块(“%s.base”%backend\u名称)
导入模块中的文件“/opt/python/pypypy3.6-7.3.1/lib python/3/importlib/_init__uuu.py”,第126行
return _bootstrap._gcd_import(名称[级别:],包,级别)
文件“”,第1003行,在\u gcd\u导入中
文件“”,第980行,在“查找”和“加载”中
文件“”,第964行,在“查找”和“加载”中解锁
文件“”,第674行,在“加载”中
exec模块中的第691行文件“/冻结导入库.\u引导\u外部”
文件“”,第228行,在调用中删除了帧
文件“/home/travis/build/[secure]/CATMAID/django/lib/custom\u postgresql\u psycopg2/base.py”,第33行,在
从django.contrib.gis.db.backends.postgis.base导入DatabaseWrapper作为PostGISDatabaseWrapper
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/gis/db/backends/postgis/base.py”,第6行,在
从.features导入数据库功能
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/gis/db/backends/postgis/features.py”,第1行,在
从django.contrib.gis.db.backends.base.features导入BaseSpatialFeatures
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/gis/db/backends/base/features.py”,第3行,在
从django.contrib.gis.db.models导入聚合
文件“/home/travis/virtualenv/pypy3.6-7.3.1/site packages/django/contrib/gis/db/models/_init__.py”,第3行
导入django.contrib.gis.db.models.functions#NOQA
文件“/home/travis/virtualenv/py3.6-