Python 如何从json中删除重复项，然后插入mysql？

我想删除JSON数据中的重复项，然后将数据放入MySQL 有人能告诉我如何删除副本吗（在插入MySQL之前）

---

我不会删除python中的重复项，而是使用数据库来实现这一点。首先，定义您所说的复制的含义-

缩略图

列是否相同？如果是，请在向其中插入数据之前。现在使用python的异常处理，将其包装到事务中，您将被排序：

import json
import pymysql
con=pymysql.connect(host="localhost",user="root",password="",db="bulk")
cursor=con.cursor()
file_name=["prime (1).json","prime (2).json", "prime (3).json","prime (4).json","prime (5).json","prime (6).json","prime (7).json","prime (8).json","prime (9).json","prime.json"]
for file_name in file_name:
    with open(file_name) as f:
        file_data=json.load(f)
        for item in file_data:
            href=item.get("href")
            thumbnail=item.get("thumbnail")
            text=item.get("text")
            imdbRating=item.get("imdbRating")
            year=item.get("year")
            imdbUrl=item.get("imdbUrl")
            wikiUrl=item.get("wikiUrl")
            try:
                cursor.execute('insert into bulk_data ( href,thumbnail,text,imdbRating,year,imdbUrl,wikiUrl) values("%s","%s","%s","%s","%s","%s","%s")',(href,thumbnail,text,imdbRating,year,imdbUrl,wikiUrl))
                con.commit()
            except:
                con.rollback()            

con.close()

---

你是说我应该在MySQL中定义唯一的列？我建议

import json
import pymysql
con=pymysql.connect(host="localhost",user="root",password="",db="bulk")
cursor=con.cursor()
file_name=["prime (1).json","prime (2).json", "prime (3).json","prime (4).json","prime (5).json","prime (6).json","prime (7).json","prime (8).json","prime (9).json","prime.json"]
for file_name in file_name:
    with open(file_name) as f:
        file_data=json.load(f)
        for item in file_data:
            href=item.get("href")
            thumbnail=item.get("thumbnail")
            text=item.get("text")
            imdbRating=item.get("imdbRating")
            year=item.get("year")
            imdbUrl=item.get("imdbUrl")
            wikiUrl=item.get("wikiUrl")
            try:
                cursor.execute('insert into bulk_data ( href,thumbnail,text,imdbRating,year,imdbUrl,wikiUrl) values("%s","%s","%s","%s","%s","%s","%s")',(href,thumbnail,text,imdbRating,year,imdbUrl,wikiUrl))
                con.commit()
            except:
                con.rollback()            

con.close()