Python PLY：分析面向行的语法

我需要解析一种相对简单的、面向行的语言，这不是我发明的语言本身，它是图形的定义语言

我的测试输入非常简单：

@startuml
Alice -> Bob: Authentication Request
Bob --> Alice: Authentication Response
Alice -> Bob: Another authentication Request
Alice <-- Bob: another authentication Response
@enduml

正如您所看到的，第二个标识“Bob”后面的标记是一个结束行：“Authentication Request”，它包含冒号作为第一个字符，因此解析器完全失去了控制

---

对此，建议的修复方法是什么？

这个词法分析器甚至可以工作一点点，这是Ply应用词法规则的特殊顺序的结果。[注1]

词法分析最简单的方法是将输入分析成一系列词素，在这些词素中可以识别出一个词素，而不必考虑以前的词素。这是任何tokeniser框架的默认模型。在该模型中，定义为行尾的任何内容的词汇模式总是适用的，这意味着您的输入将被分析为新行和其他行。那可能不是你想要的

看起来词素实际上是一个冒号，后跟行的其余部分，因此没有必要将冒号和行的其余部分分成两个标记。如果是这样的话，那么这个模式真的很容易写：r'：*。如果冒号在其他地方用于其他目的，这将不起作用。希望他们不是

如果您将冒号和行的其余部分分成两个标记，以便冒号不属于匹配标记的值，那么您可以通过在：.*标记函数内修改t.value来实现相同的效果

笔记： 帘布层按以下顺序检查图案：

首先，按照文件中定义函数的顺序从令牌函数中提取模式 第二，标记变量的模式，按长度的相反顺序，即从最长到最短。 由于模式.*比模式长：，因此将首先尝试该模式，因此将永远无法识别冒号。我相信->之前匹配的是纯粹的运气。*。对于长度相同的图案，不应依赖于按长度排列的图案顺序

总的来说，最好使用以下策略之一：

仅使用令牌函数并按正确顺序手动排序

仅对明确的模式使用标记变量

---

我被纠正了。我现在重新阅读了文件，当然与您的解释一致。我觉得lexer会寻找解析器实际期望的标记。这对我现在的语法很好，但事实并非如此。“我想我会有很多麻烦，因为这东西看起来不太好。”@ziobyte:它的表现不太好，但快速浏览一下手册并没有告诉我冒号的其他用法，所以制作一个从冒号到行尾匹配的标记似乎是可以接受的。

tokens = (
    'BEGIN', 'END', 'START', 'STATE', 'RARROW2', 'RARROW1', 'LARROW2', 'LARROW1',
    'IDENT', 'COLON', 'NUMBER', 'BSCRIPT', 'ESCRIPT', 'ENDLINE', 'FULLINE', 'newline'
)

literals = '{:}'

t_BEGIN = r"@startuml"
t_END = r"@enduml"
t_START = r"\[\*\]"
t_RARROW2 = r"-->"
t_RARROW1 = r"->"
t_LARROW2 = r"<--"
t_LARROW1 = r"<-"
t_BSCRIPT = r"/'--"
t_ESCRIPT = r"--'/"
t_ENDLINE = r'.+'
t_FULLINE = r'^.*\n'

def t_IDENT(t):
    r"""[a-zA-Z_][a-zA-Z0-9_]*"""
    return t

t_ignore = " \t"


def t_newline(t):
    r"""\n+"""
    t.lexer.lineno += t.value.count("\n")
    return t

def t_error(t):
    print("Illegal character '%s'" % t.value[0])
    t.lexer.skip(1)

def p_diagram(p):
    """diagram : begin diags end"""

def p_begin(p):
    """begin : BEGIN newline"""

def p_end(p):
    """end : END newline"""

def p_diags1(p):
    """diags : diag"""

def p_diags2(p):
    """diags : diags diag"""

def p_diag_t(p):
    """diag : tranc"""

def p_tranc1(p):
    """tranc : trans newline"""

def p_tranc2(p):
    """tranc : trans ':' ENDLINE newline"""

def p_transr(p):
    """trans : node rarrow node"""

def p_transl(p):
    """trans : node larrow node"""

def p_node(p):
    """node : IDENT
            | START"""

def p_rarrow(p):
    """rarrow : RARROW1
              | RARROW2"""
    p[0] = p[1]
    print("rarrow : (%s)" % p[1])


def p_larrow(p):
    """larrow : LARROW1
              | LARROW2"""

   yacc.py: 360:PLY: PARSE DEBUG START
   yacc.py: 408:
   yacc.py: 409:State  : 0
   yacc.py: 433:Stack  : . LexToken(BEGIN,'@startuml',1,0)
   yacc.py: 443:Action : Shift and goto state 2
   yacc.py: 408:
   yacc.py: 409:State  : 2
   yacc.py: 433:Stack  : BEGIN . LexToken(newline,'\n',1,9)
   yacc.py: 443:Action : Shift and goto state 11
   yacc.py: 408:
   yacc.py: 409:State  : 11
   yacc.py: 433:Stack  : BEGIN newline . LexToken(IDENT,'Alice',2,10)
   yacc.py: 469:Action : Reduce rule [begin -> BEGIN newline] with ['@startuml','\n'] and goto state 1
   yacc.py: 504:Result : <NoneType @ 0x5584868800e0> (None)
   yacc.py: 408:
   yacc.py: 409:State  : 1
   yacc.py: 433:Stack  : begin . LexToken(IDENT,'Alice',2,10)
   yacc.py: 443:Action : Shift and goto state 8
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin IDENT . LexToken(RARROW1,'->',2,16)
   yacc.py: 469:Action : Reduce rule [node -> IDENT] with ['Alice'] and goto state 10
   yacc.py: 504:Result : <Node @ 0x7fa389dae9e8> ([[Alice]])
   yacc.py: 408:
   yacc.py: 409:State  : 10
   yacc.py: 433:Stack  : begin node . LexToken(RARROW1,'->',2,16)
   yacc.py: 443:Action : Shift and goto state 20
   yacc.py: 408:
   yacc.py: 409:State  : 20
   yacc.py: 433:Stack  : begin node RARROW1 . LexToken(IDENT,'Bob',2,19)
   yacc.py: 469:Action : Reduce rule [rarrow -> RARROW1] with ['->'] and goto state 22
   yacc.py: 504:Result : <str @ 0x7fa389daea78> ('->')
   yacc.py: 408:
   yacc.py: 409:State  : 22
   yacc.py: 433:Stack  : begin node rarrow . LexToken(IDENT,'Bob',2,19)
   yacc.py: 443:Action : Shift and goto state 8
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin node rarrow IDENT . LexToken(ENDLINE,': Authentication Request',2,22)
   yacc.py: 578:Error  : begin node rarrow IDENT . LexToken(ENDLINE,': Authentication Request',2,22)
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin node rarrow IDENT . LexToken(newline,'\n',2,46)
   yacc.py: 469:Action : Reduce rule [node -> IDENT] with ['Bob'] and goto state 26
   yacc.py: 504:Result : <Node @ 0x7fa389daeb00> ([[Bob]])
   yacc.py: 408:
   yacc.py: 409:State  : 26
   yacc.py: 433:Stack  : begin node rarrow node . LexToken(newline,'\n',2,46)
   yacc.py: 469:Action : Reduce rule [trans -> node rarrow node] with [[[Alice]],'->',[[Bob]]] and goto state 9
   yacc.py: 504:Result : <Trans @ 0x7fa389daea58> ([[Alice]] --> [[Bob]])
   yacc.py: 408:
   yacc.py: 409:State  : 9
   yacc.py: 433:Stack  : begin trans . LexToken(newline,'\n',2,46)
   yacc.py: 443:Action : Shift and goto state 16
   yacc.py: 408:
   yacc.py: 409:State  : 16
   yacc.py: 433:Stack  : begin trans newline . LexToken(IDENT,'Bob',3,47)
   yacc.py: 469:Action : Reduce rule [tranc -> trans newline] with [<Trans @ 0x7fa389daea58>,'\n'] and goto state 4
   yacc.py: 504:Result : <Trans @ 0x7fa389daea58> ([[Alice]] --> [[Bob]])
   yacc.py: 408:
   yacc.py: 409:State  : 4
   yacc.py: 433:Stack  : begin tranc . LexToken(IDENT,'Bob',3,47)
   yacc.py: 469:Action : Reduce rule [diag -> tranc] with [<Trans @ 0x7fa389daea58>] and goto state 5
   yacc.py: 504:Result : <Trans @ 0x7fa389daea58> ([[Alice]] --> [[Bob]])
   yacc.py: 408:
   yacc.py: 409:State  : 5
   yacc.py: 433:Stack  : begin diag . LexToken(IDENT,'Bob',3,47)
   yacc.py: 469:Action : Reduce rule [diags -> diag] with [<Trans @ 0x7fa389daea58>] and goto state 6
   yacc.py: 504:Result : <list @ 0x7fa389db3ac8> ([[[Alice]] --> [[Bob]]])
   yacc.py: 408:
   yacc.py: 409:State  : 6
   yacc.py: 433:Stack  : begin diags . LexToken(IDENT,'Bob',3,47)
   yacc.py: 443:Action : Shift and goto state 8
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin diags IDENT . LexToken(RARROW2,'-->',3,51)
   yacc.py: 469:Action : Reduce rule [node -> IDENT] with ['Bob'] and goto state 10
   yacc.py: 504:Result : <Node @ 0x7fa389daeb00> ([[Bob]])
   yacc.py: 408:
   yacc.py: 409:State  : 10
   yacc.py: 433:Stack  : begin diags node . LexToken(RARROW2,'-->',3,51)
   yacc.py: 443:Action : Shift and goto state 21
   yacc.py: 408:
   yacc.py: 409:State  : 21
   yacc.py: 433:Stack  : begin diags node RARROW2 . LexToken(IDENT,'Alice',3,55)
   yacc.py: 469:Action : Reduce rule [rarrow -> RARROW2] with ['-->'] and goto state 22
   yacc.py: 504:Result : <str @ 0x7fa389daeb90> ('-->')
   yacc.py: 408:
   yacc.py: 409:State  : 22
   yacc.py: 433:Stack  : begin diags node rarrow . LexToken(IDENT,'Alice',3,55)
   yacc.py: 443:Action : Shift and goto state 8
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin diags node rarrow IDENT . LexToken(ENDLINE,': Authentication Response',3,60)
   yacc.py: 578:Error  : begin diags node rarrow IDENT . LexToken(ENDLINE,': Authentication Response',3,60)
   yacc.py: 408:
   yacc.py: 409:State  : 8
   yacc.py: 433:Stack  : begin diags node rarrow IDENT . LexToken(newline,'\n',3,85)
   yacc.py: 469:Action : Reduce rule [node -> IDENT] with ['Alice'] and goto state 26
   yacc.py: 504:Result : <Node @ 0x7fa389dae9e8> ([[Alice]])
   yacc.py: 408: