Python 在amazon lambda中使用moviepy、scipy和numpy

我想使用AWS Lambda功能生成视频

我已经按照指示找到了

现在，我有以下过程来构建我的

Lambda

函数：

第一步 启动一个

Amazon Linux EC2

实例并以root用户身份在其上运行：

#/usr/bin/env bash
#在Amazon Linux上安装SciPy堆栈，并为AWS Lambda做好准备
yum-y更新
yum-y groupinstall“开发工具”
yum-y安装blas--enablerepo=epel
yum-y安装lapack--enablerepo=epel
yum-y安装atlas-sse3-devel--enablerepo=epel
yum-y安装Cython--enablerepo=epel
yum-y安装python27
yum-y安装python27-numpy.x86_64
yum-y安装python27-numpy-f2py.x86_64
yum-y安装python27-scipy.x86_64
/usr/local/bin/pip安装--升级pip
mkdir-p/home/ec2用户/堆栈
/usr/local/bin/pip安装moviepy-t/home/ec2 user/stack
cp-R/usr/lib64/python2.7/dist-packages/numpy/home/ec2 user/stack/numpy
cp-R/usr/lib64/python2.7/dist-packages/scipy/home/ec2 user/stack/scipy
tar-czvf stack.tgz/home/ec2 user/stack/*

步骤2 我将产生的焦油球scp到我的笔记本电脑上。然后运行此脚本以构建zip存档

#/usr/bin/env bash
mkdir tmp
rm lambda.zip
tar-xzf stack.tgz-C tmp
zip-9 lambda.zip进程\u movie.py
zip-r9 lambda.zip*.ttf
cd tmp/home/ec2用户/堆栈/
zip-r9../../../../../../lambda.zip*

process\u movie.py

脚本目前只是一个测试，以查看堆栈是否正常：

def制作电影（事件、上下文）：
导入操作系统
打印（os.listdir（'.'））
打印（os.listdir（'numpy'））
尝试：
进口西皮
除恐怖外：
打印（'无法导入scipy'）
尝试：
进口numpy
除恐怖外：
打印（'无法导入numpy'）
尝试：
进口电影
除恐怖外：
打印（'无法导入电影副本'）

步骤3 然后我将生成的存档上传到S3，作为我的

lambda

函数的源。 当我测试该函数时，我得到以下

callstack

：

START RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca Version: $LATEST
['tqdm', 'imageio-1.4.egg-info', 'decorator.pyc', 'process_movie.py', 'decorator-4.0.6.dist-info', 'imageio', 'moviepy', 'tqdm-3.4.0.dist-info', 'scipy', 'numpy', 'OpenSans-Regular.ttf', 'decorator.py', 'moviepy-0.2.2.11.egg-info']
['add_newdocs.pyo', 'numarray', '__init__.py', '__config__.pyc', '_import_tools.py', 'setup.pyo', '_import_tools.pyc', 'doc', 'setupscons.py', '__init__.pyc', 'setup.py', 'version.py', 'add_newdocs.py', 'random', 'dual.pyo', 'version.pyo', 'ctypeslib.pyc', 'version.pyc', 'testing', 'dual.pyc', 'polynomial', '__config__.pyo', 'f2py', 'core', 'linalg', 'distutils', 'matlib.pyo', 'tests', 'matlib.pyc', 'setupscons.pyc', 'setup.pyc', 'ctypeslib.py', 'numpy', '__config__.py', 'matrixlib', 'dual.py', 'lib', 'ma', '_import_tools.pyo', 'ctypeslib.pyo', 'add_newdocs.pyc', 'fft', 'matlib.py', 'setupscons.pyo', '__init__.pyo', 'oldnumeric', 'compat']
can not import scipy
'module' object has no attribute 'core': AttributeError
Traceback (most recent call last):
  File "/var/task/process_movie.py", line 91, in make_movie
    import numpy
  File "/var/task/numpy/__init__.py", line 122, in <module>
    from numpy.__config__ import show as show_config
  File "/var/task/numpy/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/var/task/numpy/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/var/task/numpy/lib/__init__.py", line 13, in <module>
    from polynomial import *
  File "/var/task/numpy/lib/polynomial.py", line 11, in <module>
    import numpy.core.numeric as NX
AttributeError: 'module' object has no attribute 'core'

END RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca
REPORT RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca  Duration: 112.49 ms Billed Duration: 200 ms     Memory Size: 1536 MB    Max Memory Used: 14 MB

我现在有以下错误：

START RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113 Version: $LATEST
running make movie
Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there.: ImportError
Traceback (most recent call last):
  File "/var/task/process_movie.py", line 3, in make_movie
    import numpy
  File "/var/task/numpy/__init__.py", line 127, in <module>
    raise ImportError(msg)
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there.

END RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113
REPORT RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113  Duration: 105.95 ms Billed Duration: 200 ms     Memory Size: 1536 MB    Max Memory Used: 14 MB

启动请求ID:6abd7ef6-b9de-11e5-8aee-918ac0a06113版本：$LATEST
跑步拍电影
导入numpy时出错：不应尝试从导入numpy
其源目录；请退出numpy源代码树，然后重新启动
您的python intepreter从此开始
回溯（最近一次呼叫最后一次）：
文件“/var/task/process\u movie.py”，make\u movie中的第3行
进口numpy
文件“/var/task/numpy/_init__uuu.py”，第127行，在
提高效率（msg）
ImportError:导入numpy时出错：不应尝试从导入numpy
其源目录；请退出numpy源代码树，然后重新启动
从那里开始你的python intepreter。
结束请求ID:6abd7ef6-b9de-11e5-8aee-918ac0a06113
报告请求ID:6abd7ef6-b9de-11e5-8aee-918ac0a06113持续时间：105.95毫秒计费持续时间：200毫秒内存大小：1536 MB最大使用内存：14 MB

我还在跟踪您的第一个链接，并通过Lambda函数以这种方式（在Windows上）导入了numpy和pandas：

使用64位Amazon Linux AMI 2015.09.1启动了一个（免费层）t2.micro，并使用Putty来实现SSH

尝试了与您使用的和Amazon文章推荐的相同的命令：

sudo yum -y update
sudo yum -y upgrade
sudo yum -y groupinstall "Development Tools"
sudo yum -y install blas --enablerepo=epel
sudo yum -y install lapack --enablerepo=epel
sudo yum -y install Cython --enablerepo=epel
sudo yum install python27-devel python27-pip gcc

创建了虚拟环境：

virtualenv ~/env
source ~/env/bin/activate

sudo ~/env/bin/pip2.7 install numpy
sudo ~/env/bin/pip2.7 install pandas

安装了软件包：

virtualenv ~/env
source ~/env/bin/activate

sudo ~/env/bin/pip2.7 install numpy
sudo ~/env/bin/pip2.7 install pandas

然后，使用WinSCP，我从

/home/ec2 user/env/lib/python2.7/dist packages

登录并下载了

/home/ec2 user/env/lib64/python2.7/site packages

中的所有内容（除了\u-markerlib、pip*、pkg\u资源、setuptools*和easyinstall*）

我将所有这些文件夹和文件与包含Lambda函数的.py文件一起放入一个zip。

因为这个.zip大于10MB，所以我创建了一个S3存储桶来存储文件。我从那里复制了文件的链接，并在Lambda函数的“Upload a.ZIP from amazons3”处粘贴

EC2实例可以关闭，不再需要它了

---

有了这个，我就可以进口努比和熊猫了。我不熟悉moviepy，但是scipy可能已经很棘手了，因为Lambda对解压部署包大小的限制是262144000字节。恐怕numpy和scipy已经一起解决了这个问题。

在本帖所有帖子的帮助下，这里有一个记录解决方案：

要使其正常工作，您需要：

用至少2GO RAM启动一个

EC2

实例（以便能够编译

NumPy

&

SciPy

）

安装所需的依赖项

sudo yum -y update
sudo yum -y upgrade
sudo yum -y groupinstall "Development Tools"
sudo yum -y install blas --enablerepo=epel
sudo yum -y install lapack --enablerepo=epel
sudo yum -y install Cython --enablerepo=epel
sudo yum install python27-devel python27-pip gcc
virtualenv ~/env
source ~/env/bin/activate
pip install scipy
pip install numpy
pip install moviepy

sudo yum -y install python-devel
sudo yum -y install gcc-c++
sudo yum -y install gcc-gfortran
sudo yum -y install libgfortran

将

stack

文件夹中目录的所有内容（除_markerlib、pip*、pkg\u resources、setuptools*和easyinstall*外）复制到您的区域设置计算机：

• home/ec2用户/env/lib/python2.7/dist包

• home/ec2 user/env/lib64/python2.7/dist包

从您的

EC2

实例获取所有必需的共享库：

• libatlas.so.3

• libf77blas.so.3

• liblapack.so.3

• libptf77blas.so.3

• libcblas.so.3

• libgfortran.so.3

• libptcblas.so.3

• libquadmath.so.0

将它们放入

stack

文件夹的

lib

子文件夹中

imageio

是

moviepy

的依赖项，您需要下载其依赖项的一些二进制版本：

libfreeimage

和

ffmpeg

；可以找到它们。把

mkdir -p /var/task/lib

cp /usr/lib64/atlas-sse3/liblapack.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libatlas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/libgfortran.so.3 /var/task/lib/.
cp /usr/lib64/libquadmath.so.0 /var/task/lib/.

[atlas]
libraries=lapack,f77blas,cblas,atlas
search_static_first=true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

python setup.py build

python setup.py install

ldd $PYTHON_HOME/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so

linux-vdso.so.1 =>  (0x00007ffe0dd2d000)
liblapack.so.3 => /var/task/lib/liblapack.so.3 (0x00007ffad6be5000)
libptf77blas.so.3 => /var/task/lib/libptf77blas.so.3 (0x00007ffad69c7000)
libptcblas.so.3 => /var/task/lib/libptcblas.so.3 (0x00007ffad67a7000)
libatlas.so.3 => /var/task/lib/libatlas.so.3 (0x00007ffad6174000)
libf77blas.so.3 => /var/task/lib/libf77blas.so.3 (0x00007ffad5f56000)
libcblas.so.3 => /var/task/lib/libcblas.so.3 (0x00007ffad5d36000)
libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007ffad596d000)
libgfortran.so.3 => /var/task/lib/libgfortran.so.3 (0x00007ffad5654000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffad5352000)
libquadmath.so.0 => /var/task/lib/libquadmath.so.0 (0x00007ffad5117000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffad4f00000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffad4b3e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffad4922000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffad471d000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007ffad451a000)
/lib64/ld-linux-x86-64.so.2 (0x000055cfc3ab8000)

cp /usr/lib64/atlas-sse3/*.a /var/task/lib/

python setup.py config

atlas_threads_info:
Setting PTATLAS=ATLAS   libraries ptf77blas,ptcblas,atlas not found in /root/Envs/skl/lib
    libraries lapack_atlas not found in /root/Envs/skl/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64   
    libraries lapack_atlas not found in /usr/local/lib64
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib         
    libraries lapack_atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib64/atlas-sse3
<class 'numpy.distutils.system_info.atlas_threads_info'>
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
    libraries lapack not found in ['/var/task/lib']
Runtime library lapack was not found. Ignoring
    libraries f77blas not found in ['/var/task/lib']
Runtime library f77blas was not found. Ignoring
    libraries cblas not found in ['/var/task/lib']
Runtime library cblas was not found. Ignoring
    libraries atlas not found in ['/var/task/lib']
Runtime library atlas was not found. Ignoring
    FOUND:
        extra_link_args = ['-lgfortran -lquadmath']
        define_macros = [('NO_ATLAS_INFO', -1)]
        language = f77
        libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas', 'lapack', 'f77blas', 'cblas', 'atlas']
        library_dirs = ['/usr/lib64/atlas-sse3']
        include_dirs = ['/usr/include']

[atlas]
libraries = lapack,f77blas,cblas,atlas
search_static_first = true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

docker run -v "$PWD":/var/task -it lambci/lambda:build bash

export share=/var/task
easy_install pip
pip install -t $share numpy

export share=/var/task
export PS1="[\u@\h:\w]\$ " # required by virtualenv
easy_install pip
pip install virtualenv
# ... make the venv, install numpy, and copy it to $share

docker pull amazonlinux && docker run -it amazonlinux
# Follow @attila-tanyi steps
# Note - sudo is not necessary here

yum install python36 python36-virtualenv python36-pip

virtualenv -p python3.6 /tmp/my_python_lib

source /tmp/my_python_lib/bin/activate

which python --to check which version s installed

pip3 install  numpy

  grep -r dist-packages *. 

/tmp/my_python_lib/lib64/python3.6,
/tmp/my_python_lib/lib/python3.6

 pip install --upgrade --target=/tmp/my_python_lib/lib/python3.6/dist-packages pandas

    cd YourApplicationName
    /venv/bin/pip install scipy -t .
    /venv/bin/pip install numpy -t .
    /venv/bin/pip install moviepy -t .

import os
import sys 

# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
workdir = os.getenv('LAMBDA_TASK_ROOT')
version = f'{sys.version_info[0]}.{sys.version_info[1]}'
additionals = [f'{workdir}/venv/lib64/python{version}/site-packages',
               f'{workdir}/venv/lib64/python{version}/lib-dynload',
               f'{workdir}/venv/lib64/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/site-packages']
sys.path = additionals + sys.path

import pandas_datareader as pdr