The problem previously appeared in situations when an identifier was changed to a keyword, i.e. `a: Tru` -> `a: True` and `True` keyword was not highlighted properly as it was substituted by an identifier AST leaf as is bypassing the highlighting lexer
Add a couple of tests on it
GitOrigin-RevId: ec25d2115ec650bf781a9fe8140e8708e849ff60
- PEP-696 adds a new syntax for declaring the default types of Type Parameters in new-new style generic classes, functions and type alias statements. Support these grammar changes.
- Store info about default types in stubs for Type Parameters
- Increment the stub version counter in PyFileElementType
GitOrigin-RevId: b6b22e3eaa86ce06132885781e5775a89bf4b840
Instead, parse them as usual and later report in the dedicated AssignTargetAnnotator
and TypeAnnotationTargetAnnotator. This way, a PsiError element appearing in the PSI
tree of a type declaration statement doesn't cause PyAstTypeDeclarationStatement.getTarget
nullability contract violation.
GitOrigin-RevId: a3e90088cfac7938c398d4d293a72dbd127a2cd0
Previously, we acknowledged them in PythonIndentingProcessor.adjustBraceLevel, inserting
synthetic STATEMENT_BREAK in front of them to stop recovery in the parser, similarly
to how we handle other kinds of incomplete brackets, but the state inside PyLexerFStringHelper
was not reset, so it kept trying to find matching closing brackets, quotes and interpreting
colons as PyTokenTypes.FSTRING_FRAGMENT_FORMAT_START instead of just PyTokenTypes.COLON.
The state in PyLexerFStringHelper and PythonIndentingProcessor became out of sync, which
led to assertion violations.
It's not an optimal solution, since now these tokens are listed both in
PythonTokenSetContributor.getUnbalancedBracesRecoveryTokens and in Python.flex lexer
specification, and we need to keep them in sync. Also, PythonTokenSetContributor
can provide additional tokens from other languages, such as Cython. But it's simple
and seems "good enough" to patch the problem in the release.
GitOrigin-RevId: 4e156314cc02aba0634d5d9e3008177f49105051
PEP 498 required f-strings to be recognizable by existing tooling, such as syntax highlighters,
by prohibiting re-using quotes of the same kind and having line breaks inside expression fragments.
We used to detect these problems already at the lexer level, correctly replacing violating quotes
with FSTRING_END token, and appending STATEMENT_BREAK tokens to illegal line breaks inside expressions,
depending on the lexer's state. Now, thanks to a general f-string grammar in PEP 701, most of this
bookkeeping could be moved from the lexer to the CompatibilityVisitor (to still be reported
for previous versions of the language and by the compatibility inspection).
Previously forbidden backslashes and line comments are now also detected by the CompatibilityVisitor
instead of the version-agnostic FStringAnnotator.
One side effect of the new grammar is that parser recovery in pre-3.12 version of Python became
slightly worse. For instance, something like `f'{foo'` used to be recognized as an f-string
with an incomplete fragment lacking its closing brace. Now, it's parsed as an incomplete
f-string, lacking its own closing quote, containing an incomplete string literal inside
an incomplete fragment. What's more, parsing of this fragment's expression doesn't terminate
until the end of a file, because STATEMENT_BREAK is never produced by PythonIndentingProcessor
while it's inside an f-string fragment, and every quote is considered a new string literal.
Examples of parsing tests affected by this are:
PythonParsingTest.testFStringFragmentIncompleteTypeConversionBeforeClosingQuote
PythonParsingTest.testFStringIncompleteFragmentWithTypeConversion
PythonParsingTest.testFStringIncompleteFragment
I also had to simplify some scenarios from PythonHighlightingTest, removing snippets
with incomplete fragments or moving such examples to the very end of a file.
It's not clear how to handle these situations not overcomplicating the lexer.
(cherry picked from commit 03ba6d7fba1b45a84aa92221e6a452645a765205)
IJ-MR-115763
GitOrigin-RevId: cd36470d9cae353fe3caeb2d3b628d8743b46cbb
In case of syntactic ambiguity with previous versions of the grammar, such as
"with (expr)" or "with (expr1, expr2)", PyWithStatement is still parsed as
having its own parentheses, not a parenthesized expression or a tuple as
a single context expression. The latter case, even though syntactically legal,
is still reported by the compatibility inspection in Python <3.9.
These changes also include proper formatter and editing support (e.g. not
inserting backslashes on line breaks inside parentheses), as well as
Complete Current Statement, which now takes possible parentheses into account
while inserting a missing colon.
The changes in the formatter are somewhat ad-hoc, intended to minimize the effect
on other constructs. "With" statement is somewhat special in the sense that it's
the first compound statement (having a statement list) with its own list-like
part in parentheses.
Existing tests on with statement processing were expanded and uniformly named.
Co-authored-by: Semyon Proshev <semyon.proshev@jetbrains.com>
GitOrigin-RevId: 15c33e97f177e81b5ed23891063555df016feb05
Not stopping at a statement break token and continuing recovery until a colon,
we considered the subsequent well-formed case clause to be a part of an error
message about a missing pattern, thus, moving the caret to its colon.
GitOrigin-RevId: f4ee0e12876960e989de3dee89925b65e3cf2339
Namely, for their case clauses and inner comments.
Right now, each of them is indented on its own, as a separate formatting block.
It's still not entirely clear whether we should have a dedicated indented
container element for case clauses, similar to PyStatementList for statements.
It might simplify the formatter and some editing actions, but cause confusion
between the two container elements.
GitOrigin-RevId: 69184d2f8f78e2e113e8f40a310bb13ac0b5e71a
All the corresponding PSI elements now have empty interfaces. The API will be
"beefed up" as we start adding actual processing of them in the code insight
(e.g. for the upcoming CFG and inspections).
The trickiest part of the parsing was the recovery. Patterns allow only
a limited subset of expression syntax, but I tried to sensibly consume and report
everything else (not building PSI for it). So that if user starts typing more
general expressions in the midst of a pattern, we still give meaningful error
messages. It seems a likely cause of errors when the feature first rolls out in
Python 3.10.
GitOrigin-RevId: fae40034964e4a25d91dab06a43d3fc07225d9c7
We assumed that if the first token of a Python file is a string literal, then it's a docstring. It's not the case for the Python console, where each input is a separate "file".
Now we pass the `PythonLexerKind` to the `PythonLexer` so that we can parse string literals differently if we are in the console "file".
I've also added Cython as a lexical kind for the Python lexer, since there is at least one place in the lexical rules that is specific to Cython. Also we have separate lexer subclasses for Cython, so having it expressed as a kind for the JFlex rules seems logical, even if we don't use it right now.
GitOrigin-RevId: f5e34fa2dc3b3da84cacf6cee69a4ba0ee674ad5
It not only makes AST of f-string nodes simpler and more obvious to work with,
but, in general, is also better supported by various platform functionality that
assumes that raw text parts of string literals are not broken into multiple
elements.
GitOrigin-RevId: 931d1ea4c09d145e763aed839dcb4acbb3e43ec7
Every expression on top-level inside f-string that could be considered as an assignment expression is actually a format expression.
GitOrigin-RevId: a375543c80d549d5c08166f33b401206ab31f8b8
Highlight expressions that are invalid by grammar or unacceptable in runtime
Highlight expressions in pythons < 3.8
GitOrigin-RevId: 89acec9db5b3a931da31c33778185e147240ec01
so as not to cause confusion with ASTNode. Also additionally rename
PyLiteralStringElement to PyPlainStringElement, again, not to confuse
users with subtle "string literal" vs. "literal string" differences
in meaning.
Daniil Kalinin
Mikhail Golubev
Petr
andrey.matveev
Irina Fediaeva
Andrey Vlasovskikh
Lada Gagina
Rustam Vishnyakov
Egor Zhdan
Anna Kozlova
Julia Beliaeva
Eldar Abusalimov