Python 查找所有公共的、不重叠的子字符串
给定两个字符串,我想识别从最长到最短的所有公共子字符串 我想删除任何“sub-”子字符串。例如,“1234”的任何子字符串都不会包含在“12345”和“51234”之间的匹配中Python 查找所有公共的、不重叠的子字符串,python,string,substring,matching,suffix-tree,Python,String,Substring,Matching,Suffix Tree,给定两个字符串,我想识别从最长到最短的所有公共子字符串 我想删除任何“sub-”子字符串。例如,“1234”的任何子字符串都不会包含在“12345”和“51234”之间的匹配中 string1 = '51234' string2 = '12345' result = ['1234', '5'] 我在考虑查找,然后递归地查找左/右最长的子字符串。但是,我不想在找到后删除公共子字符串。例如,下面的结果在中间共享一个6: string1 = '12345623456' string2 = '6
string1 = '51234'
string2 = '12345'
result = ['1234', '5']
我在考虑查找,然后递归地查找左/右最长的子字符串。但是,我不想在找到后删除公共子字符串。例如,下面的结果在中间共享一个6:
string1 = '12345623456'
string2 = '623456'
result = ['623456', '23456']
最后,我需要检查一个字符串与数千个字符串的固定列表。我不确定在散列出这些字符串中的所有子字符串时是否有一个明智的步骤
以前的答案:
在本文中,我们发现了一个动态规划解决方案,它需要O(nm)时间,其中n和m是字符串的长度。我对一种更有效的方法感兴趣,它将使用后缀树
背景:
我正在根据旋律片段创作歌曲旋律。有时,一个组合会产生一个旋律,与现有的一行中的太多音符相匹配
我可以使用字符串相似性度量,比如编辑距离,但我相信与旋律差别很小的曲调是独特而有趣的。不幸的是,这些曲调与连续复制一段旋律的多个音符的歌曲具有相似的相似程度。让我们从树开始
from collections import defaultdict
def identity(x):
return x
class TreeReprMixin(object):
def __repr__(self):
base = dict(self)
return repr(base)
class PrefixTree(TreeReprMixin, defaultdict):
'''
A hash-based Prefix or Suffix Tree for testing for
sequence inclusion. This implementation works for any
slice-able sequence of hashable objects, not just strings.
'''
def __init__(self):
defaultdict.__init__(self, PrefixTree)
self.labels = set()
def add(self, sequence, label=None):
layer = self
if label is None:
label = sequence
if label:
layer.labels.add(label)
for i in range(len(sequence)):
layer = layer[sequence[i]]
if label:
layer.labels.add(label)
return self
def add_ngram(self, sequence, label=None):
if label is None:
label = sequence
for i in range(1, len(sequence) + 1):
self.add(sequence[:i], label)
def __contains__(self, sequence):
layer = self
j = 0
for i in sequence:
j += 1
if not dict.__contains__(layer, i):
break
layer = layer[i]
return len(sequence) == j
def depth_in(self, sequence):
layer = self
count = 0
for i in sequence:
if not dict.__contains__(layer, i):
print "Breaking"
break
else:
layer = layer[i]
count += 1
return count
def subsequences_of(self, sequence):
layer = self
for i in sequence:
layer = layer[i]
return layer.labels
def __iter__(self):
return iter(self.labels)
class SuffixTree(PrefixTree):
'''
A hash-based Prefix or Suffix Tree for testing for
sequence inclusion. This implementation works for any
slice-able sequence of hashable objects, not just strings.
'''
def __init__(self):
defaultdict.__init__(self, SuffixTree)
self.labels = set()
def add_ngram(self, sequence, label=None):
if label is None:
label = sequence
for i in range(len(sequence)):
self.add(sequence[i:], label=label)
要填充树,可以使用.add\ngram
方法
下一部分有点棘手,因为您正在寻找字符串的并发遍历,同时跟踪树坐标。为了实现这一切,我们需要一些对树和查询字符串进行操作的函数
def overlapping_substrings(string, tree, solved=None):
if solved is None:
solved = PrefixTree()
i = 1
last = 0
matching = True
solutions = []
while i < len(string) + 1:
if string[last:i] in tree:
if not matching:
matching = True
else:
i += 1
continue
else:
if matching:
matching = False
solutions.append(string[last:i - 1])
last = i - 1
i -= 1
i += 1
if matching:
solutions.append(string[last:i])
for solution in solutions:
if solution in solved:
continue
else:
solved.add_ngram(solution)
yield solution
def slide_start(string):
for i in range(len(string)):
yield string[i:]
def seek_subtree(tree, sequence):
# Find the node of the search tree which
# is found by this sequence of items
node = tree
for i in sequence:
if i in node:
node = node[i]
else:
raise KeyError(i)
return node
def find_all_common_spans(string, tree):
# We can keep track of solutions to avoid duplicates
# and incomplete prefixes using a Prefix Tree
seen = PrefixTree()
for substring in slide_start(string):
# Drive generator forward
list(overlapping_substrings(substring, tree, seen))
# Some substrings are suffixes of other substrings which you do not
# want
compress = SuffixTree()
for solution in sorted(seen.labels, key=len, reverse=True):
# A substrings may be a suffix of another substrings, but that substrings
# is actually a repeating pattern. If a solution is
# a repeating pattern, `not solution in seek_subtree(tree, solution)` will tell us.
# Otherwise, discard the solution
if solution in compress and not solution in seek_subtree(tree, solution):
continue
else:
compress.add_ngram(solution)
return compress.labels
def search(query, corpus):
tree = SuffixTree()
if isinstance(corpus, SuffixTree):
tree = corpus
else:
for elem in corpus:
tree.add_ngram(elem)
return list(find_all_common_spans(query, tree))
如果有什么不清楚的地方,请让我知道,我会尽力澄清。它似乎是在一个解决方案的后面附加:搜索('abc','b')>>['bc']也试图找出为什么它会首先找到最短的解决方案:搜索('test','tests')>['te','st']由于索引管理仍然有点草率,所以在重叠的子字符串中可能会出现一个off-by-one错误。还请注意,在我的使用示例中,“语料库”是字符串列表,而不是单个字符串。谢谢。实际上是在contains过程中,在执行检查之前,j=j+1递增,因此“bc”在n_gram“b”中。还发现我不想添加任何已经添加的n_g,因为这样可以避免重复检查/奇怪的结果。
search("12345", ["51234"])
search("623456", ["12345623456"])