Google app engine 非批量上传至应用程序数据存储

我有一个非常大的SQLite数据库。无论如何，对于SQLite db来说是大的：1.13 GB。我试图批量上传它，除了我无法将数据库转储到CSV。我试了好几次，基本上都放弃了

更实用的方法似乎是调整我在中找到的代码，一次上传一条记录到数据存储中。让它通宵运行。诸如此类的事情

#-------------------------------------------------------------------------------
# Name:        App Data Store
# Purpose:      Move chess database to the app engine datastore
#               from c:\\PGNSDB
#               Comcast is the worst company in the world
# Created:     22/06/2013
# Copyright:   (c) Administrator 2013
#-------------------------------------------------------------------------------

from google.appengine.ext import db
from google.appengine.api import users
import sqlite3
import google
import logging

class game(db.Model):
        Event = db.StringProperty(required=False)
        Site = db.StringProperty(required=False)
        EventDate = db.StringProperty(required=False, indexed=True)
        Round = db.StringProperty(required=False)
        White = db.StringProperty(required=True, indexed=True)
        Black = db.StringProperty(required=True, indexed=True)
        Result = db.StringProperty(required=True,
                            choices=set(["1-0","0-1","1/2-1/2"]), indexed=True)
        ECO = db.StringProperty(required=False)
        WhiteELO = db.StringProperty(required=False)
        BlackELO = db.StringProperty(required=False)
        PlyCount = db.StringProperty(required=False)
        PGN = db.StringProperty(required=True)
        email = db.StringProperty()

def main():
    logging.info('Beginning upload')
    conn = sqlite3.connect('C:\\PGNSDB')
    c = conn.cursor()
    games = c.execute("select Event, Site, Date, Round, White, Black, Result, ECO, WhiteELO, BlackELO, PGN from games")

    logging.info('Local database is now open on C drive.')

    for agame in games:
        logging.info('Uploading a PGN.')
        thisgame = game(Event = agame[0],
                        Site = agame[1],
                        EventDate = agame[2],
                        Round = agame[3],
                        White = agame[4],
                        Black = agame[5],
                        Result = agame[6],
                        ECO = agame[7],
                        WhiteELO = agame[8],
                        BlackELO = agame[9],
                        PGN = agame[10],
                        #email = users.get_current_user().email())
                        email = "xxx@gmail.com")
        logging.info('About to put.')
        thisgame.put()

if __name__ == '__main__':
    main()

因此，我在Google应用程序引擎启动器中使用以下App.yaml运行了它：

application: pgnhelper
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /
  script: home.app

- url: /index\.html
  script: home.app

- url: /stylesheets
  static_dir: stylesheets

- url: /(.*\.(gif|png|jpg))
  static_files: static/\1
  upload: static/(.*\.(gif|png|jpg))

- url: /admin/.*
  script: admin.app
  login: admin

- url: /.*
  script: not_found.app

builtins:
- remote_api: on

。。。并具有以下记录的输出：

2013-07-01 21:40:59 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', u'C:\\Code\\uploadpgns']"
INFO     2013-07-01 21:41:06,479 devappserver2.py:528] Skipping SDK update check.
WARNING  2013-07-01 21:41:06,530 api_server.py:314] Could not initialize images API; you are likely missing the Python "PIL" module.
WARNING  2013-07-01 21:41:06,546 simple_search_stub.py:955] Could not read search indexes from c:\users\jj\appdata\local\temp\appengine.pgnhelper\search_indexes
INFO     2013-07-01 21:41:06,612 api_server.py:138] Starting API server at: http : //XXX:58254
INFO     2013-07-01 21:41:06,621 dispatcher.py:164] Starting server "default" running at: http : //XXX:8080
INFO     2013-07-01 21:41:06,627 admin_server.py:117] Starting admin server at: http : //XXX:8000
INFO     2013-07-01 21:46:46,724 api_server.py:509] Applying all pending transactions and saving the datastore
INFO     2013-07-01 21:46:46,724 api_server.py:512] Saving search indexes
2013-07-01 21:46:46 (Process exited with code 0)

2013-07-01 21:53:53 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', u'C:\\Code\\uploadpgns']"
INFO     2013-07-01 21:53:54,956 devappserver2.py:528] Skipping SDK update check.
WARNING  2013-07-01 21:53:54,963 api_server.py:314] Could not initialize images API; you are likely missing the Python "PIL" module.
INFO     2013-07-01 21:53:54,974 api_server.py:138] Starting API server at: http : //XXX:58311
INFO     2013-07-01 21:53:54,980 dispatcher.py:164] Starting server "default" running at: http : //XXX:8080
INFO     2013-07-01 21:53:54,984 admin_server.py:117] Starting admin server at: http : //XXX:8000
ERROR    2013-07-02 01:54:46,207 wsgi.py:219] 

Traceback (most recent call last):

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 196, in Handle

    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 255, in _LoadHandler

    handler = __import__(path[0])

ImportError: No module named home

INFO     2013-07-01 21:54:46,223 server.py:593] default: "GET / HTTP/1.1" 500 -
ERROR    2013-07-02 01:54:46,325 wsgi.py:219] 

Traceback (most recent call last):

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 196, in Handle

    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 255, in _LoadHandler

    handler = __import__(path[0])

ImportError: No module named not_found

INFO     2013-07-01 21:54:46,332 server.py:593] default: "GET /favicon.ico HTTP/1.1" 500 -

。。。其中XXX表示指向本地主机的链接。。。我想我遗漏了这个WSGI的一些基本内容。而名为“未找到”的模块？找不到它

我发现的示例没有提到网关接口。如何合并一个

---

谢谢

您根本不需要使用dev appserver（或本地app.yaml等）

将代码放入模块（没有处理程序等），并将其导入远程api外壳

然后，您可以导入任何您想要的内容，因为您没有在沙箱中运行，并且您直接与appengine数据存储进行对话

---

你也可以通过批量处理游戏对象的put来加快速度。假设在一个列表中每隔100存储一次，然后执行db.put（the_list）

仍然有点像python noob，我可以问另一个问题吗？我的文件名为upload.py，我导入了它（“导入上载”）并键入了“main（）”，但它无法识别它。最后一行一行地粘贴代码，让它在shell中执行。。。正确的方法是什么？谢谢。运行远程api外壳时，请确保当前工作目录是upload.py的位置。