Commit Graph

2993 Commits

Author SHA1 Message Date
Ilya.Kazakevich
820a665815 PY-29154: Return test runner exit code
When test runner returns code > 0 (because of test failure), our runner must do the same.

See issue description for the reason

GitOrigin-RevId: 231c9ab337e80f70872bf6e6f61a397599f18e26
2019-12-23 20:53:08 +00:00
Andrey Lisin
a22effa122 PY-38347 Fix frame evaluator opcodes to lines mapping procedure
The previous version of the procedure could miss breakpoints when extra `EXTENDED_ARG` instructions are added before the injected code. This happened because in some corner cases after updating the opcodes to lines mapping an `EXTENDED_ARG` instruction could land as the last instruction in a line, and the instruction which argument it extends - on another. This lead to the situation when it was impossible to detect that we have hit the next line. These changes also add the ability to trace opcodes when the `PYDEVD_TRACE_OPCODES` environment variable is set to `True` (available only for Cython) to ease the debugging of such cases in the future.

(cherry picked from commit 77399d3b7b2b9bc6aac1b85d59ae305500cb6ecb)

GitOrigin-RevId: 105bf9d3d2cc51a940b03b1cb11c07e97074048b
2019-12-19 14:08:32 +00:00
Mikhail Golubev
abcc326dc7 Allow to change multiprocessing start method via environment for testing purposes
GitOrigin-RevId: 2e0cb4c240f02a600b37e7511bdb6903a6708d1d
2019-12-18 11:08:59 +00:00
Mikhail Golubev
e8d5bd4376 Properly re-configure logging in a subprocess during skeleton generation
On platforms where "fork" is not available or not used by multiprocessing
root logger configuration is not automatically inherited and, thus, we loose
messages sent by worker processes. So as not to depend on particular
implementation of the module, e.g. by checking sys.platform or
multiprocessing.get_start_method(), we now explicitly reset and setup
logging anew in every created process.

GitOrigin-RevId: 572b5a1a77346f46421a6dd1e0ed0b1ed9c4ca6f
2019-12-18 11:08:59 +00:00
Mikhail Golubev
ce6b36702a PY-34047 Handle keyword-only arguments in constructors of stub return values
On the whole, such stub calls look shaggy and could probably be replaced by
plain parameterless invocations altogether. It's also doubtful why we use
them only when analyzing docstrings of non-function callables, but questioning
whether they are needed in the first place seems out of scope of the current
issue. At least we don't produce warnings while generating them anymore.

GitOrigin-RevId: 25650f121e88141df081874f346d0cff2d31e3bb
2019-12-17 12:36:49 +00:00
Mikhail Golubev
15eeb501e9 PY-34047 Generator properly restores signatures of functions with keyword-only arguments
GitOrigin-RevId: d66bcb11e1fd3d9d27fb9a9dab19b6575cfffaa3
2019-12-17 12:36:49 +00:00
Elizaveta Shashkova
5e6dcf687f IDEA-CR-56420: PY-39414 Fix TypeError exception with Task class under Concurrency Visualizer
Replace Task class with another mock class, not function

GitOrigin-RevId: 4f522b3308d3289b3d64f9d19e51908c90985368
2019-12-16 10:03:13 +00:00
Ilya.Kazakevich
c51f222882 PY-39146: Do not use deprecated API.
See ``load()`` body

GitOrigin-RevId: 822dd2033d2f306c82400c8b1bb29fd769207d7e
2019-12-10 21:08:52 +00:00
Elizaveta Shashkova
5c67bc0373 IDEA-CR-56419: PY-34555 Clean module flag for child process
If a process was started with `-m` key (and converted to `--module` flag), this parameter will be passed to all its child processes. So we need to clear this flag in SetupHolder before patching arguments

GitOrigin-RevId: 00ae0b23fc6107bee75272a03a5bd15ca474fbc7
2019-12-10 15:38:42 +00:00
Mikhail Golubev
ca3e96dbee PY-38479 Rely on "venv" defaults regarding symlinks when creating virtualenvs
Because we used to call venv.create() function directly in packaging_tool.py when
creating environments we ended up using function's own default unconditionally
creating copies of executables on all platforms, including those where venv itself
creates symlinks (Linux, MacOS). Turned out, it leads to issues with locating
dynamic libraries when one attempts to create a virtualenv with MacOS 10.15 system
Python 3 interpreter. Now we call venv module directly as we already do with "pip"
relying on its default behavior and patching the only command-line flag that we
actually need -- "--without-pip".

GitOrigin-RevId: d65aaa325817933de7121225038a5cf84fa2c77a
2019-12-05 12:38:14 +00:00
Serhii Charykov
3fbb05afd2 PY-38963 Fix string formatting exception (close #1240)
(cherry picked from commit e0a82bb25b7b6d8bb03e89733647ecbe2496d7cc)

GitOrigin-RevId: 4ee0751e58aadc0bec77fc2eb36fd7a442e0754e
2019-12-04 16:06:45 +00:00
Andrey Lisin
eb956ca29d PY-38378 PY-38424 Fix updating variables in debugger console
We now access the `frame.f_locals` attribute only once. Multiple access leads to the loss of the changes we have made. We also alter the second argument of the `PyFrame_LocalsToFast()` to 1 to ensure that all the deleted variables will be wiped out from the frame locals.

(cherry picked from commit 28abff39b2c37f30651db220e8700032a629ce73)

GitOrigin-RevId: 8514da7ec187c1192ea9cdf851f89f008242fc30
2019-11-25 09:31:23 +00:00
Semyon Proshev
cff3f1f86d Enable pyi-stubs for os (PY-23258, PY-21395, PY-21394, PY-21397, PY-17420, PY-28984, PY-27584)
GitOrigin-RevId: 34ecf07bc68276e62315d01f3c3347f79026ec65
2019-11-20 11:37:02 +00:00
Andrey Lisin
ba365df917 PY-38941 Support debugging of processes created with posix_spawn
(cherry picked from commit e55c2168595e22e80a8f95c2e3812cf7f687c31c)

GitOrigin-RevId: 4db4aaaa9ce7198cfb5177bac262f9f72b92cdee
2019-11-19 17:05:35 +00:00
Mikhail Golubev
edc0ae4fcf PY-34019 Don't call instance methods with multiprocessing to avoid possible pickling issues
GitOrigin-RevId: b1c332cbdd9cfcf2939aa40caafa66030a0529be
2019-11-18 21:11:28 +00:00
Mikhail Golubev
0d1cb9c85d PY-34019 Use multiprocessing.Pipe instead of a queue to synchronize back a return value
It allows us to overcome a Python 3.7.2 specific regression in multiprocessing
(https://bugs.python.org/issue35797).

GitOrigin-RevId: 122543ed586e3b34317e13811b5a87d8631cec61
2019-11-18 21:11:28 +00:00
Mikhail Golubev
a197154f80 PY-34019 Comment why we need to use Queue.get_no_wait() for subprocess result
GitOrigin-RevId: 7f0219c7af51c925c417ad88cbe73028f66f8a86
2019-11-18 21:11:28 +00:00
Mikhail Golubev
64c9499598 PY-34019 Fix pickling error by making local helper function top-level
GitOrigin-RevId: ffc734792c48c865cb8203748c89d3bbc6b85afc
2019-11-18 21:11:28 +00:00
Mikhail Golubev
d7800ddb55 PY-34019 Fix "_Py_HashRandomization_Init: failed to get random numbers" in tests on Windows
GitOrigin-RevId: deefe1bc45fb54e35e23d834983203e2c35f8784
2019-11-18 21:11:28 +00:00
Mikhail Golubev
8f3b13f501 PY-34019 Remove path to the executable from generator's progress messages
It's already been done so for messages on the Java side. The reason is that with
the new UI for progress indicators in the status bar the link can become too
lengthy and, on the whole, it just doesn't have much value: the trailing part
of the path is usually invisible anyway, and it's obvious that it's happening
for the current project interpreter.

GitOrigin-RevId: 1d1e9da9c1f7b92f072f5030c4bd2968c6ba464c
2019-11-18 21:11:28 +00:00
Mikhail Golubev
d7f50bdefb PY-34019 Add a comment about why TYPE_CHECKING=False constant is needed
GitOrigin-RevId: 1dbfbef92ef5299ea879f2e45a361e8cf7058ac6
2019-11-18 21:11:28 +00:00
Mikhail Golubev
1a081d6e2b PY-34019 Use "logging" module API directly instead of wrappers
GitOrigin-RevId: 605317ae0e7365d318de749a0d45f01e39b02741
2019-11-18 21:11:28 +00:00
Mikhail Golubev
a46d649753 PY-34019 Read the current generator version directly from resources
We used to launch the generator itself for that but it seems completely
unnecessary since these versions can only diverge in case of remote interpreters
and we forcefully update helpers on a remote on every IDE release, so they
should be always in sync.

GitOrigin-RevId: 6d22dd0c0f3c3938a1a739246c98eda2f05c1ce0
2019-11-18 21:11:28 +00:00
Mikhail Golubev
36391174a7 PY-34019 Remove debug_mode flag altogether since it was always on anyway
GitOrigin-RevId: d8a98887734f61dc8b081a74f9c5fdd73a549b49
2019-11-18 21:11:28 +00:00
Mikhail Golubev
4ecad317a9 PY-34019 Remove pointless parameter of check_generator_output() helper test method
GitOrigin-RevId: 9c9f377d54b37f36522c1ac762f65cf4b6a9566c
2019-11-18 21:11:28 +00:00
Mikhail Golubev
fb216c1cb9 PY-34019 Restore command-line flag to run generation only for builtin modules
GitOrigin-RevId: a79ce8fadddcdc2e6db967924ae3e6a80bfa99b2
2019-11-18 21:11:28 +00:00
Mikhail Golubev
a949b9706e PY-34019 Move tests on former "builtins-only" generation to MultiModuleGenerationTest
Also allow to override the defaults for extra_sys_path and extra_args with
explicit [].

GitOrigin-RevId: 61db81e2c69eab69b8e92d0f31cbb2b9f05ac972
2019-11-18 21:11:28 +00:00
Mikhail Golubev
8e1da1c25b PY-34019 Fix tests broken after "-b" option had been removed
GitOrigin-RevId: b59b3756d78ddba32863da8991372a63ad5a6449
2019-11-18 21:11:28 +00:00
Mikhail Golubev
b7f4b58f6a PY-34019 Finally switch to standard "argparse" module for CLI parsing
GitOrigin-RevId: 6e66d83750c57c1d8cbc47a5e201c730e2608053
2019-11-18 21:11:28 +00:00
Mikhail Golubev
5e45d95b50 PY-34019 Extract faulthandler initialization
GitOrigin-RevId: dc4ce7397ef700e35515ec547abccf91fd7b6a0f
2019-11-18 21:11:28 +00:00
Mikhail Golubev
1e054c1520 PY-34019 Be more explicit about reading/writing .state.json strategy
instead of relying on confusing indicators such as passed in
dictionary content.

GitOrigin-RevId: 95ac2efa13c14ecd7c7297aadce3a3a332bbb9d1
2019-11-18 21:11:28 +00:00
Mikhail Golubev
d1721a5b15 PY-34019 Properly remove records for no longer existing binaries from .state.json
GitOrigin-RevId: e81bd407a05b7c45580b3b008eaeb14d9256017a
2019-11-18 21:11:28 +00:00
Mikhail Golubev
ef803deaee PY-34019 Make sure that trailing slashes in paths don't affect skeletons layout (again)
GitOrigin-RevId: 218f3aa3c10094b17ad2f70d001c56b760196fdd
2019-11-18 21:11:28 +00:00
Mikhail Golubev
1048df5bd5 PY-34019 Better name for options related to writing/reading of .state.json
GitOrigin-RevId: 0a101045cb9f0140f77492b00ce02983591885bb
2019-11-18 21:11:28 +00:00
Mikhail Golubev
ef7d12b168 PY-34019 Hide some excess logging under TRACE level
GitOrigin-RevId: cd52d6a1039b3e89b11aa80bcaf56098a31ffcd1
2019-11-18 21:11:28 +00:00
Mikhail Golubev
745379a6d3 PY-34019 Start using the standard Python facilities for logging
GitOrigin-RevId: 0b8a93cbb8a83ecb8cd75826ee231f3aa73cafcd
2019-11-18 21:11:28 +00:00
Mikhail Golubev
2cdadd883f PY-34019 Turn more top-level functions into methods of SkeletonGenerator
GitOrigin-RevId: 1306bb0e937939fdd9df4addd6c090cb51cc0b7e
2019-11-18 21:11:28 +00:00
Mikhail Golubev
201dcb7a54 PY-34019 Type hint more utility functions
GitOrigin-RevId: 5f2312926854e188bfc4cb9ad2b906fff26c032e
2019-11-18 21:11:28 +00:00
Mikhail Golubev
a892b4b74d PY-34019 Type hint skeleton statuses and generation results with NewType
GitOrigin-RevId: 993a759f112b3a2b88f7a5ef07cb83fbd2dd3004
2019-11-18 21:11:28 +00:00
Mikhail Golubev
64c6aa060a PY-34019 Continue refactoring to a class-based generator
GitOrigin-RevId: cb3974a845f65a64e286f70baf1bc029668e70d4
2019-11-18 21:11:28 +00:00
Mikhail Golubev
112538a9ed PY-34019 Start implementing class-based generator API for its further evolution
GitOrigin-RevId: ffdc062c37fc8bcc9cfa663375cecb990062958c
2019-11-18 21:11:28 +00:00
Mikhail Golubev
10ff4a43e8 PY-34019 Read skeleton state passed via stdin in a single line
We can't consume the entire stdin in case of Docker as it would require
closing the stream which apparently is not possible.

GitOrigin-RevId: 773bc28f673bf6f23ae8d25b000429abfb1480fb
2019-11-18 21:11:28 +00:00
Mikhail Golubev
58896944a9 PY-34019 Properly take into account versions of cached skeletons
Previously versions found in sdk skeletons directory and in provided .state.json
might have affected whether we take skeletons from the cache or not.

GitOrigin-RevId: d089f0531cdcdeecfbf4318fa0743ccff524d091
2019-11-18 21:11:28 +00:00
Mikhail Golubev
1dfa562a38 PY-34019 Simplify building .state.json for failing modules
GitOrigin-RevId: a691547b12901e040eb91666beffcc83652474ff
2019-11-18 21:11:28 +00:00
Mikhail Golubev
ca805f05ba PY-34019 Test that we don't update version of already failed skeleton in .state.json
GitOrigin-RevId: b2dbffeb64356686a762a5e1d019d4d1a82f0ebc
2019-11-18 21:11:28 +00:00
Mikhail Golubev
5732493568 PY-34019 Properly generate .state.json for up-to-date and cached skeletons
GitOrigin-RevId: 80145388966757ee194137f107cd875608f768b2
2019-11-18 21:11:28 +00:00
Mikhail Golubev
f4a0c28cd8 PY-37530 Don't pack skeletons.zip inside itself
It leads to indefinitely large ZIP archive and hangs up initialization
of a remote interpreter.

Ideally, we should create it in a temporary directory in the first place.

GitOrigin-RevId: 9d4dea9560c14fe253d30f82c2103907fcc5636c
2019-11-18 21:11:28 +00:00
Mikhail Golubev
cbe3752665 PY-34019 Remove obsolete "GENERATOR3_STANDALONE_MODE" env flag as it's now the default
GitOrigin-RevId: d7f4abce4b7df70daf4b12772d417ab73ac9ee20
2019-11-18 21:11:28 +00:00
Mikhail Golubev
5b4526c1e1 PY-34019 Save binary modification times for new skeletons in .state.json
GitOrigin-RevId: 275d5552df2ae8e7d5ecb6bd41d18ce1a010e33c
2019-11-18 21:11:28 +00:00
Mikhail Golubev
943ceea4f6 PY-34019 Make sure that we populate .state.json for builtin modules
GitOrigin-RevId: 03e7df1e7db1a4711f6dd572ac8ea6c94231d480
2019-11-18 21:11:28 +00:00