Files
openide/python/testData/psi/CompleteFStringFragmentTerminatedAtStatementOnlyKeyword.txt
Mikhail Golubev 10808546fc PY-63393 Handle keywords terminating lexing of f-strings fragments in the lowermost JFlex lexer
Previously, we acknowledged them in PythonIndentingProcessor.adjustBraceLevel, inserting
synthetic STATEMENT_BREAK in front of them to stop recovery in the parser, similarly
to how we handle other kinds of incomplete brackets, but the state inside PyLexerFStringHelper
was not reset, so it kept trying to find matching closing brackets, quotes and interpreting
colons as PyTokenTypes.FSTRING_FRAGMENT_FORMAT_START instead of just PyTokenTypes.COLON.
The state in PyLexerFStringHelper and PythonIndentingProcessor became out of sync, which
led to assertion violations.

It's not an optimal solution, since now these tokens are listed both in
PythonTokenSetContributor.getUnbalancedBracesRecoveryTokens and in Python.flex lexer
specification, and we need to keep them in sync. Also, PythonTokenSetContributor
can provide additional tokens from other languages, such as Cython. But it's simple
and seems "good enough" to patch the problem in the release.

GitOrigin-RevId: 4e156314cc02aba0634d5d9e3008177f49105051
2023-10-05 22:02:27 +00:00

37 lines
1.2 KiB
Plaintext

PyFile:CompleteFStringFragmentTerminatedAtStatementOnlyKeyword.py
PyAssignmentStatement
PyTargetExpression: s
PsiElement(Py:IDENTIFIER)('s')
PsiWhiteSpace(' ')
PsiElement(Py:EQ)('=')
PsiWhiteSpace(' ')
PyStringLiteralExpression: {
PyFormattedStringElement
PsiElement(Py:FSTRING_START)('f'')
PyFStringFragment
PsiElement(Py:FSTRING_FRAGMENT_START)('{')
PsiErrorElement:Expression expected
PsiWhiteSpace('\n')
PsiErrorElement:Type conversion, ':' or '}' expected
<empty list>
PsiErrorElement:' expected
<empty list>
PyRaiseStatement
PsiElement(Py:RAISE_KEYWORD)('raise')
PsiErrorElement:End of statement expected
<empty list>
PsiElement(Py:COLON)(':')
PsiErrorElement:Statement expected, found Py:COLON
<empty list>
PyExpressionStatement
PyReferenceExpression: foo
PsiElement(Py:IDENTIFIER)('foo')
PsiErrorElement:End of statement expected
<empty list>
PsiElement(Py:RBRACE)('}')
PsiErrorElement:Statement expected, found Py:RBRACE
<empty list>
PyExpressionStatement
PyStringLiteralExpression:
PsiElement(Py:SINGLE_QUOTED_STRING)(''')