so as not to cause confusion with ASTNode. Also additionally rename
PyLiteralStringElement to PyPlainStringElement, again, not to confuse
users with subtle "string literal" vs. "literal string" differences
in meaning.
There is actually no need to match them together with braces and then return
the second character back to the stream. The longest match rule will ensure
that all allowed escape sequences will be recognised as one unit, so it seems
safe to match backslash individually for all remaining cases.
Made the corresponding states of the lexer exclusive so that the pattern
for line comments doesn't match literal fragments starting with a hash sign
as the longest (together with closing braces and quotes). It also means that
we need to replicate the BAD_CHARACTER rule in these states as a fallback
pattern for unmatched input.
It's beneficial for two reasons: it allows us to avoid occasional
PsiWhitespaces at the end of incomplete fragments and also we
can now better report illegal line comments this way, because even though
they still terminate f-strings they are now under the corresponding AST
nodes for expression fragments and can be processed by the annotator
in the same fashion.
Also fixed the problem that whitespaces inside literal parts of f-strings were
matched as PsiWhitespaces instead of FSTRING_TEXT tokens and updated all relevant
test data.
When the terminating quote is either at the beginning of a string literal or
inside it, either in the given f-string itself or inside a nested one and
either in normal fragment or in a fragment located inside a format part of
another fragment. Normal string literals and nested f-strings are also checked
separately.
The cause of the problem was the difference in parsing for different
language levels.
This commit improves error recovery and adds support for stubs for
syntax elements unavailable in older Python versions.
Cases that are covered are necessary for stubs generation
for Python 3.6 standard library. They are discovered by automatic
validation executed by stubs generator in StubsGenerator.kt.
Those are the cases that were fixed:
* A stub for single star argument in Py2 was absent
* Type annotations in Py2 were absent
* Print with end argument in Py2 broke parser
* Exec function as argument in Py2 broke parser
* Async keyword after decorator broke parser
All the cases are listed as test cases in PyUnifiedStubsTest
P.S. It is possible though that more cases will be revealed in future.
Typeshed doesn't contain the definitions for them in __builtins__.pyi
so we have to provide types and detect the builtin status for these
words as a special case.
Check for "for" or "async for" while parsing generators, list, dict and set literals
Highlight "async" as keyword in generators, list, dict and set literals
Annotation is preserved at the level of assignment nodes similar to
where CPython keeps them in its AST (in special "augassign" nodes).
For type annotations in form "x: int" without variable initialization
special statement PyTypeDefinitionStatement was introduced.
We had to switch to stack-based parsing contexts in order to be able
to use context-sensitive token filtering for 'async'.
Also this commit fixes the highlighting annotator issue with blinking
names of the functions and blinking 'async'.
Mikhail Golubev
Semyon Proshev
Dmitry Trofimov
Andrey VlasovskikhandAndrey Vlasovskikh