PEP 498 required f-strings to be recognizable by existing tooling, such as syntax highlighters,
by prohibiting re-using quotes of the same kind and having line breaks inside expression fragments.
We used to detect these problems already at the lexer level, correctly replacing violating quotes
with FSTRING_END token, and appending STATEMENT_BREAK tokens to illegal line breaks inside expressions,
depending on the lexer's state. Now, thanks to a general f-string grammar in PEP 701, most of this
bookkeeping could be moved from the lexer to the CompatibilityVisitor (to still be reported
for previous versions of the language and by the compatibility inspection).
Previously forbidden backslashes and line comments are now also detected by the CompatibilityVisitor
instead of the version-agnostic FStringAnnotator.
One side effect of the new grammar is that parser recovery in pre-3.12 version of Python became
slightly worse. For instance, something like `f'{foo'` used to be recognized as an f-string
with an incomplete fragment lacking its closing brace. Now, it's parsed as an incomplete
f-string, lacking its own closing quote, containing an incomplete string literal inside
an incomplete fragment. What's more, parsing of this fragment's expression doesn't terminate
until the end of a file, because STATEMENT_BREAK is never produced by PythonIndentingProcessor
while it's inside an f-string fragment, and every quote is considered a new string literal.
Examples of parsing tests affected by this are:
PythonParsingTest.testFStringFragmentIncompleteTypeConversionBeforeClosingQuote
PythonParsingTest.testFStringIncompleteFragmentWithTypeConversion
PythonParsingTest.testFStringIncompleteFragment
I also had to simplify some scenarios from PythonHighlightingTest, removing snippets
with incomplete fragments or moving such examples to the very end of a file.
It's not clear how to handle these situations not overcomplicating the lexer.
(cherry picked from commit 03ba6d7fba1b45a84aa92221e6a452645a765205)
IJ-MR-115763
GitOrigin-RevId: cd36470d9cae353fe3caeb2d3b628d8743b46cbb
It's beneficial for two reasons: it allows us to avoid occasional
PsiWhitespaces at the end of incomplete fragments and also we
can now better report illegal line comments this way, because even though
they still terminate f-strings they are now under the corresponding AST
nodes for expression fragments and can be processed by the annotator
in the same fashion.
Such errors are difficult to spot because most of the time the
corresponding markers are displayed only in the right gutter, not in
the editor (though, sometimes a single character is highlighted for
some reason).
For an empty expression fragment the whole its content range is
highlighted (from "{" to either "!", ":" or "}").
For the first unclosed expression fragment the range from "{" to the
end of the containing string node is highlighted.
In case of a missing conversion character the character "!" itself is
highlighted.
