mirror of
https://gitflic.ru/project/openide/openide.git
synced 2025-12-14 18:05:27 +07:00
37d25ee815
PEP 498 required f-strings to be recognizable by existing tooling, such as syntax highlighters,
by prohibiting re-using quotes of the same kind and having line breaks inside expression fragments.
We used to detect these problems already at the lexer level, correctly replacing violating quotes
with FSTRING_END token, and appending STATEMENT_BREAK tokens to illegal line breaks inside expressions,
depending on the lexer's state. Now, thanks to a general f-string grammar in PEP 701, most of this
bookkeeping could be moved from the lexer to the CompatibilityVisitor (to still be reported
for previous versions of the language and by the compatibility inspection).
Previously forbidden backslashes and line comments are now also detected by the CompatibilityVisitor
instead of the version-agnostic FStringAnnotator.
One side effect of the new grammar is that parser recovery in pre-3.12 version of Python became
slightly worse. For instance, something like `f'{foo'` used to be recognized as an f-string
with an incomplete fragment lacking its closing brace. Now, it's parsed as an incomplete
f-string, lacking its own closing quote, containing an incomplete string literal inside
an incomplete fragment. What's more, parsing of this fragment's expression doesn't terminate
until the end of a file, because STATEMENT_BREAK is never produced by PythonIndentingProcessor
while it's inside an f-string fragment, and every quote is considered a new string literal.
Examples of parsing tests affected by this are:
PythonParsingTest.testFStringFragmentIncompleteTypeConversionBeforeClosingQuote
PythonParsingTest.testFStringIncompleteFragmentWithTypeConversion
PythonParsingTest.testFStringIncompleteFragment
I also had to simplify some scenarios from PythonHighlightingTest, removing snippets
with incomplete fragments or moving such examples to the very end of a file.
It's not clear how to handle these situations not overcomplicating the lexer.
(cherry picked from commit 03ba6d7fba1b45a84aa92221e6a452645a765205)
IJ-MR-115763
GitOrigin-RevId: cd36470d9cae353fe3caeb2d3b628d8743b46cbb
8 lines
436 B
Python
8 lines
436 B
Python
f'{42!r}'
|
|
f'{42!s}'
|
|
f'{42!a}'
|
|
f'{42<error descr="An illegal conversion character 'z': should be one of 's', 'r', 'a'">!z</error>}'
|
|
f'{42<error descr="An illegal conversion character 'foo': should be one of 's', 'r', 'a'">!foo</error>}'
|
|
f'{42<error descr="A conversion character is expected: should be one of 's', 'r', 'a'">!</error>}'
|
|
f'{42<error descr="A conversion character is expected: should be one of 's', 'r', 'a'">!</error>:2}'
|