Commit Graph

1255 Commits

Author SHA1 Message Date
Andrei.Kuznetsov
c4eb1eb3c6 IJPL-174027: use the same project files condition in FileBasedIndexImpl.forceUpdate and FileBasedIndexImpl.indexUnsavedDocuments
(cherry picked from commit a3df35169570a3605dd10de57d8479a54ce6f530)

IJ-CR-152316

GitOrigin-RevId: 486f82ff62bae94293b5e3db4ca1a211e476d504
2025-01-14 18:41:59 +00:00
Andrei.Kuznetsov
7bec186e06 IJPL-174027: more accurate logging
(cherry picked from commit 095dda3c4f916d8738880dfcf5d2ba0c51ad558b)

IJ-CR-152316

GitOrigin-RevId: 06f03a4a514385f18c33fbb96f885f20263ca9b0
2025-01-14 18:41:59 +00:00
Andrei.Kuznetsov
1247257fe0 Don't log psiLen in VfsEventsMerger, because there is no fast and reliable way to understand if file may have PSI or not
GitOrigin-RevId: f0de712a22a10d2ec2bff34a7d9798f320556bf6
2024-10-11 15:21:07 +00:00
Andrei.Kuznetsov
8a089889af Logging in indexes should not throw errors and break business logic
GitOrigin-RevId: 6a2f8cd5e0aa56442d3eb20a229836133d516704
2024-10-11 15:21:07 +00:00
Daniil Ovchinnikov
823e985f18 API dumps: re-generate dumps
GitOrigin-RevId: b5cec844cbfe3313723404f7a85b590c25350b41
2024-10-04 21:20:21 +00:00
Ruslan Cheremin
f76738fd86 [indexes][cleanup] remove excessive final modifiers, cleanup names, etc
- no functional changes

GitOrigin-RevId: 0239cb5221444b9ba112e84cea322b77e9f45c34
2024-09-30 15:59:12 +00:00
Vladimir Krivosheev
a9e7930abe bazel - build lang-impl
GitOrigin-RevId: c4c4dce20bd6e4046644f95f6e7b3f437c51668d
2024-09-27 13:47:17 +00:00
Vladimir Krivosheev
a1210118c0 bazel - fix JPMS, exports and internal
GitOrigin-RevId: 3dc80a9ee01e1900c0cebc24b2d08f04f93ddeff
2024-09-25 19:05:37 +00:00
Ruslan Cheremin
97615afac1 [indexing][refactoring] rename methods to more readable
GitOrigin-RevId: 7f58c47bc01ccd4daf5bf295406d32cd2db8317f
2024-09-25 17:07:44 +00:00
Ruslan Cheremin
de3bcb66a0 [indexing][cleanup] minor code shaving
GitOrigin-RevId: d951014c19b06975da7b91f2e8bd23e97462fed1
2024-09-25 17:07:44 +00:00
Ruslan Cheremin
7cec4c609c [indexing][cleanup] minor code shaving
GitOrigin-RevId: c470817eb6e72256b658dd09d408c2b20159e80f
2024-09-25 17:07:44 +00:00
Alexey Kudravtsev
12b36cba91 cleanup
GitOrigin-RevId: 855c03c503b7cf09bec19b9f3bab659246c03ca6
2024-09-25 13:26:59 +00:00
Vladimir Krivosheev
4eb9b78b78 use bazel to build platform-impl
GitOrigin-RevId: b5bbc75323a08c7674777e512dcfd988ceb4461f
2024-09-22 17:01:28 +00:00
Vladimir Krivosheev
834bd75254 run jps-to-bazel by Bazel, compile platform using Bazel
GitOrigin-RevId: 3774d5adc404b02036f70b7cc35c9e60fdb5d84c
2024-09-21 00:54:17 +00:00
Max Medvedev
fa779374dd IJPL-161383 mark registerSearcherForTesting as internal
GitOrigin-RevId: 67b145b6687ad8744f728e3f21dfcb065f1569d6
2024-09-12 10:38:36 +00:00
Ruslan Cheremin
307242bfa4 [indexes][refactoring] cleanup PsiSearchHelperImpl code, add comments
GitOrigin-RevId: 8e81c00a82bf51c6ea3901c0578c5a9075c96e74
2024-09-11 18:39:31 +00:00
Ruslan Cheremin
cb9e7b8b86 [indexes][refactoring] cleanup IdIndexEntry code, add comments
GitOrigin-RevId: b3daa76fa5aee5f29c09dce7ad7730147abe281d
2024-09-11 18:39:31 +00:00
Max Medvedev
20c95bc127 IJPL-161383 fix Find Usages for symbols
Find usages for symbols did not work because SearchTargetVariantsDataRuleKt.targetVariants did not get any results from dc.getData(FindUsagesAction.SEARCH_TARGETS).
This happened because SEARCH_TARGETS (SearchTargetsDataRule) is a data context rule,
and it uses another data context rule, CommonDataKeys.SYMBOLS (TargetSymbolsDataRule).

These rules use data keys that are never present in one given DataProvider. So, their
call chain never succeeds.

The fix is to mark both rules as `type="context"`.

Also, a test is written with a generic test language and a test search target.
The key for proper data context testing is calling `fallbackToProductionDataManager(...)`.
Otherwise, TestDataProvider is used, which indeed has all necessary keys in a single place.

GitOrigin-RevId: 9e4f3a59b9afb58424307faaaf3665a77308ed56
2024-08-31 15:13:15 +00:00
Alexey Kudravtsev
641d857f5a do not return duplicate values from getValues() when there are many files associated with each value (part of KTIJ-27513 K2 IDE: Slow reference search for specific symbols in unused symbol inspection)
GitOrigin-RevId: 90db6f345b035d3e63b207e80665612a40ed127c
2024-08-29 17:34:57 +00:00
Ruslan Cheremin
f2121f04c2 [indexes][monitoring] IJPL-160180: report technical indexing stats to FUS
+ mainly: a fraction of time indexers were idle due to backpressure from index writers lagging behind -- important metrics for deciding which part of indexing pipeline is really a bottleneck in users hardware/software setups, as opposite to our benchmarks hardware/software setup

GitOrigin-RevId: 9349a4349af8120f4b9c67ab8f927af6a6d2d8d6
2024-08-29 17:33:18 +00:00
Ruslan Cheremin
13dfa850c0 [indexes][refactoring] combine add/remove/update processors into single UpdatedEntryProcessor
+ 3 different processors were used during diff(old,new) calculation, to report added/updated/removed entries -> actual processing of all 3 categories is quite similar, so I replace 3 processors with 1 `UpdatedEntryProcessor` with an additional `UpdateKind` enum arg.

GitOrigin-RevId: 53b1121ee1cb0899c87de17a9148b0b51029dd45
2024-08-24 12:52:25 +00:00
Ruslan Cheremin
86437db586 [indexing][refactoring] IJPL-160177: merge AbstractUpdateData with UpdateData in MapReduceIndex
+ `UpdateData` is now implemented via `ForwardIndexUpdate` and `ChangesProducer` interfaces -- which reduce number of different impls & increase code reuse

GitOrigin-RevId: 78f4e1d271e8157ddc33c07d7c25ae673489432f
2024-08-23 18:56:50 +00:00
Ruslan Cheremin
cab4169964 [cleanup] sorted MapReduceIndex fields, made them private, added accessors
+ made all the fields private, add accessors for some that are accessed outside the class
+ sorted fields in groups by semantics

GitOrigin-RevId: fe345f916dbdf87fbf25048afee44b64f9c28c84
2024-08-22 10:33:22 +00:00
Ruslan Cheremin
8da349af18 [cleanup] renaming, comments (no functional changes)
GitOrigin-RevId: b1d1dcaf465b1d03481e241c0ce6c4aaea9d6ceb
2024-08-22 10:33:22 +00:00
Ruslan Cheremin
992d09eb5b [cleanup] remove useless MapReduceIndexBase.updateWithMap() override
GitOrigin-RevId: a6aed3bcd46d0e55ca48c12a2b6dde64f1d2525e
2024-08-22 10:33:22 +00:00
Ruslan Cheremin
db1fd77e0b [cleanup] renaming, comments (no functional changes)
GitOrigin-RevId: fd33f1ba806e2dfcf9bd6a69baa663d4d2c1eeac
2024-08-22 10:33:22 +00:00
Max Medvedev
4c98e42ef8 cleanup SerializedStubTree
GitOrigin-RevId: 00902db69ebc378815a85d70e26059347ffb4f32
2024-08-17 10:06:10 +00:00
Max Medvedev
b118f713ab introduce StorageUpdate interface
GitOrigin-RevId: 610d4ba045deab0724494d3422b2373b3e18ded8
2024-08-17 10:06:10 +00:00
Max Medvedev
5fc0e29380 cleanup StubUpdatingIndex
GitOrigin-RevId: db9fd031568e77e32aaf76f7b2a3a2c2031b6199
2024-08-17 10:06:10 +00:00
Max Medvedev
bacd3a22fc StubUpdatingIndex: move field to the top, before constructors
GitOrigin-RevId: c26b428250ecedad4f31c9a1211d46374ee4b542
2024-08-17 10:06:10 +00:00
Max Medvedev
c74fbea2df remove final keywords in StubIndexEx
GitOrigin-RevId: e0381eb5c72434d30332845f3b5a9aa8a68eb696
2024-08-17 10:06:10 +00:00
Max Medvedev
d86af8912b cleanup StubIndexEx
GitOrigin-RevId: da4a439ee281d173c0d2df18ef24ad805ccf8ec1
2024-08-17 10:06:10 +00:00
Max Medvedev
f3617188e6 cleanup UpdatableIndex
GitOrigin-RevId: d17b63ca5116aa00a91d918cd7b6f25bd1546f23
2024-08-17 10:06:09 +00:00
Ruslan Cheremin
ba668f4bea [indexes]IJP-1365: IndexWriter iface, re-implement IndexUpdateWriter with it
`IndexWriter` interface:
+ re-implement `IndexUpdateWriter` as `IndexWriter` same-thread and parallel threads implementations
+ add coroutine-based `IndexWriter` impl

Minors:
+ remove `GLOBAL_INDEXING_SEMAPHORE`
+ renamed `FileIndexesValuesApplier` -> `FileIndexingResult` (better fit)
+ OTel metrics reported: time spent on index writing, time slept by indexers due to writer queue overload

GitOrigin-RevId: be4eb4ed2b26c3b6e02b334d7ef448fa0e1d983c
2024-08-15 17:14:33 +00:00
Max Medvedev
c33a6ef01d ignore V class from fastutil
GitOrigin-RevId: 964f1632b57f609af53949f7aee6ee4a398fa5c5
2024-08-14 14:05:32 +00:00
Ruslan Cheremin
126c33c51c [indexes] IJP-1365: larger cache for Id & TrigramIndex
+ IdIndex & TrigramIndex are the largest indexes -> increase cache size 64x for them

GitOrigin-RevId: 48dbd0915dd07f5a9c20da05bd08bafe5c927102
2024-08-13 14:22:35 +00:00
Ruslan Cheremin
df59e37c6d [indexes] minor improvements in IdIndexEntryMapExternalizer
GitOrigin-RevId: dc222163aaf5ab691eceb6683719439129e837fd
2024-08-13 14:22:35 +00:00
Ruslan Cheremin
4325094326 [indexes] refactoring: CustomInputMapIndexExtension
+ CustomInputMapIndexExtension is a replacement for CustomInputsIndexFileBasedIndexExtension, to create fully-customized externalizer for Map<Key,Value> -- planned to be used for IdIndex optimization
+ code cleanup and comments along the way

GitOrigin-RevId: f2e3cf5f35aebc61bddcfe2fb749fd72631d75a4
2024-08-13 10:55:50 +00:00
Ruslan Cheremin
01527ec451 [indexes] IJP-1365: utilise ValueLessInputMapExternalizer for ScalarIndexExtension by default
+ defaultMapExternalizerFor(): default factory for input map externalizers, reused by different index storage impls

GitOrigin-RevId: ec3a324630fdb8c200985ebc185d1e305d365fba
2024-08-13 10:55:50 +00:00
Ruslan Cheremin
c7234a3b81 [indexes] IJP-1365: comments and minor code cleanup
+ no functional changes

GitOrigin-RevId: 43366a676a34cdc2eb46cb4ca41f38f1242397cf
2024-08-13 10:55:50 +00:00
Ruslan Cheremin
4518cb1caf [indexes] IJP-1365: utilize specialized serializer for IdIndex inputMap in DurableStorageLayout
GitOrigin-RevId: 22271011313a8c031cc24405301573a30537746c
2024-08-13 10:55:50 +00:00
Andrei.Kuznetsov
058da6c4ef Comment
GitOrigin-RevId: d82370fe9745510d317d5c50ca58ed5e2fa36fd4
2024-08-12 14:13:19 +00:00
Max Medvedev
8e5516d5de cleanup nanoxml
GitOrigin-RevId: 58f3de6ecb1b6049ce22d8658f0ca6a6b3a99b0a
2024-08-09 20:25:05 +00:00
Ruslan Cheremin
edb733a038 [indexes] refactor IdDataConsumer
+ extract explicit int->int map wrapper (IdEntryToScopeMapImpl), extract an IdEntryToScopeMap interface for it

GitOrigin-RevId: cd2cc114425121a945bdf82e3e0ff7a5769a52b4
2024-08-09 18:26:38 +00:00
Ruslan Cheremin
9e32ecca88 [indexes][refactoring] move common .invalidateCachedMappings() method up to IndexStorage
+ added TODO about method name ambiguity

GitOrigin-RevId: cd07a47aae29a2f4e0a8a003f8af3cd9550ef457
2024-08-09 14:05:38 +00:00
Roman Shevchenko
127ee4ec2f Cleanup (minor optimization; typos; formatting)
GitOrigin-RevId: 0ee801a8ecd200f01c8a033aa0a9c26ba8b81661
2024-08-08 21:44:30 +00:00
Max Medvedev
8f91400df3 PsiSearchHelperImpl: doc update
GitOrigin-RevId: 2461012fcfb19f0ad3395997117b8de356ed4131
2024-08-05 13:15:57 +00:00
Max Medvedev
d51f732634 PsiSearchHelperImpl: remove double negation from check
GitOrigin-RevId: f985b2b220c1abfb81ed8e15f03a4a67c4f0757c
2024-08-05 13:15:57 +00:00
Max Medvedev
befac5539c cleanup TrigramTextSearchService and TextSearchService
GitOrigin-RevId: 6467400b889654eae5c12f1845bd3618113376b1
2024-08-05 13:15:57 +00:00
Liudmila Kornilova
cd97efcf15 IJPL-159372 Freeze caused by IndexableFilesCollector (don't iterate over all files inside one big RA)
Replace EmptyProgressIndicator with null

GitOrigin-RevId: 8427c2343de5e2a173eaf126ed3f854976ee6b45
2024-07-31 10:43:08 +00:00