Python 使用PyMongo从Twitter流式API存储JSON字典

目前，我有很多tweet，我打算将它们存储在我实验室的服务器上。然而，我在决定如何做这件事时遇到了一些问题

例如，Tweet具有以下格式：

{
    "contributors": null,
    "coordinates": null,
    "created_at": "Tue Jul 10 17:09:12 +0000 2012",
    "entities": {
        "hashtags": [{
            "indices": [62, 78],
            "text": "thestrongnation"
        }],
        "urls": [],
        "user_mentions": [{
            "id": 376483630,
            "id_str": "376483630",
            "indices": [0, 8],
            "name": "SherryHonig",
            "screen_name": "sahonig"
        }]
    },
    "favorited": false,
    "geo": null,
    "id": 222739261219282945,
    "id_str": "222739261219282945",
    "in_reply_to_screen_name": "sahonig",
    "in_reply_to_status_id": 222695060528037889,
    "in_reply_to_status_id_str": "222695060528037889",
    "in_reply_to_user_id": 376483630,
    "in_reply_to_user_id_str": "376483630",
    "place": {
        "attributes": {},
        "bounding_box": {
            "coordinates": [
                [
                    [-106.645646, 25.837164000000001],
                    [-93.508038999999997, 25.837164000000001],
                    [-93.508038999999997, 36.500703999999999],
                    [-106.645646, 36.500703999999999]
                ]
            ],
            "type": "Polygon"
        },
        "country": "United States",
        "country_code": "US",
        "full_name": "Texas, US",
        "id": "e0060cda70f5f341",
        "name": "Texas",
        "place_type": "admin",
        "url": "http://api.twitter.com/1/geo/id/e0060cda70f5f341.json"
    },
    "retweet_count": 0,
    "retweeted": false,
    "source": "web",
    "text": "@sahonig BOOM !!!! I feel a 1 coming on!!! Awesome! #thestrongnation",
    "truncated": false,
    "user": {
        "contributors_enabled": false,
        "created_at": "Wed Feb 15 14:40:48 +0000 2012",
        "default_profile": false,
        "default_profile_image": false,
        "description": "Living life on 30A & doing it my way. My mind is Stronger than physical challenge. Runner, Crosfit, Fitness Challenges. Proud member of #thestrongnation. ",
        "favourites_count": 17,
        "follow_request_sent": null,
        "followers_count": 215,
        "following": null,
        "friends_count": 184,
        "geo_enabled": true,
        "id": 493181025,
        "id_str": "493181025",
        "is_translator": false,
        "lang": "en",
        "listed_count": 4,
        "location": "Seagrove Beach, FL",
        "name": "30A My Way \u2600",
        "notifications": null,
        "profile_background_color": "c0deed",
        "profile_background_image_url": "http://a0.twimg.com/profile_background_images/590670431/aj7p0c6j2oevdj240jz2.jpeg",
        "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/590670431/aj7p0c6j2oevdj240jz2.jpeg",
        "profile_background_tile": true,
        "profile_image_url": "http://a0.twimg.com/profile_images/2381704869/b7bizspexjgmyspqesg0_normal.jpeg",
        "profile_image_url_https": "https://si0.twimg.com/profile_images/2381704869/b7bizspexjgmyspqesg0_normal.jpeg",
        "profile_link_color": "0084B4",
        "profile_sidebar_border_color": "C0DEED",
        "profile_sidebar_fill_color": "DDEEF6",
        "profile_text_color": "333333",
        "profile_use_background_image": true,
        "protected": false,
        "screen_name": "30A_MyWay",
        "show_all_inline_media": false,
        "statuses_count": 1731,
        "time_zone": "Central Time (US & Canada)",
        "url": null,
        "utc_offset": -21600,
        "verified": false
    }
}

当然，这是一个Python字典，它恰好遵循JSON格式。 MongoDB方便地接受JSON格式的这些，但问题是，我不希望提供所有信息。流式API给了我20个字段，而实际上我现在只想处理用户ID、文本和位置。我最初打算通过它进行解析，只提取我想要的文本，但我找不到一个可靠的解析器，考虑到开发这种解析器的条件，我觉得编写一个解析器只是浪费时间

然而，我正在考虑的另一个解决方案是，既然这些都被读取到MongoDB中，也许我可以在字典中只存储我想要的东西，而去掉其余的。唯一的问题是Twitter接收到的文件格式将所有字典放在同一行——我觉得不管怎样，我都必须进行某种提取

---

其他人有什么建议吗？

如果有必要，您可以使用

json.loads

（它将返回一个

dict

s的

list

，如上格式）来获取结果并将其放入Python结构（如果还没有），以便对其进行操作。（但人们通常会使用一些Python Twitter库来透明地实现这一点）

只需为所需数据创建一个新的dict，并将其插入MongoDB，例如：

假设

ret

=如上所述的tweet响应

mydata = {
    'name': ret['user']['screen_name'],
    'text': ret['text']
}

print mydata['name'], 'wrote', mydata['text'] # or something

# insert mydata into appropriate MongoDB DB/collection here

---

这里的pymongo python示例代码应该会有很大帮助