ML 4 Completion Performance Plugin

Update experiment configuration

Add fixed generator execution order

Fix some build issues

Remove redundant ArtifactGenerator and update related classes

Enable performance only on Nightly users

Enable performance only on Nightly users

Remove unused interfaces and update related classes

Fix Kotlin style-related comments

Fix references in documentation

Allow other contributors to run

Improve code organization and add documentation to turboComplete

Merge remote-tracking branch 'origin/master' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	community/plugins/completion-ml-ranking/resources/experiment.json

Run completion contributors after live templates

Remove unnecessary functions from performance parameters

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Update experiment list for Kotlin

Increase experiment version number

Clarify usage of local debug model

Register A/B experiment

Merge branch 'master' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	build/src/org/jetbrains/intellij/build/UltimateLibraryLicenses.kt
#	community/plugins/completion-ml-ranking/src/com/intellij/completion/ml/performance/MLCompletionPerformanceTracker.kt

Remove unnecessary nullability indicators

Remove redundant class

Make the opener internal

Revert change in modules.xml

Utilize delegation when necessary

Merge branch 'master' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	community/plugins/completion-ml-ranking/src/com/intellij/completion/ml/performance/MLCompletionPerformanceTracker.kt

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Merge branch 'master' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	community/platform/analysis-api/src/com/intellij/codeInsight/completion/CompletionParameters.java
#	community/plugins/completion-ml-ranking/src/com/intellij/completion/ml/performance/MLCompletionPerformanceTracker.kt

Remove unnecessary symbols from Kotlin's kind generation session

Disable performance on TeamCity

Fix some style issues

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Restore EP name

Add dependency on completion ml ranking plugin

Change scope of kotlin's completion dependency

Accord API core package name with the naming convention

Accord API core package name with the naming convention

Revert non-standard-root-packages.txt to default

Remove java performance from community modules

Remove duplicating logic

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Reorganize plugin's structure

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Fix some comments & build issues

Fix some build issues

Fix some build issues

Fix some build issues

Fix some build issues

Fix some build issues

Fix some build issues

Merge remote-tracking branch 'origin/master' into marin/plugin-ml-4-completion-performace

Revert unintended change

Fix some style issues

Rename .java to .kt

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Add core to main and ultimate iml files

Fix test

Use model with updated kinds names

Avoid unwanted generator execution for artifact generation

Fix JavaKindVariety

Revert not indented changes

Actualize kotlin completion usage

Merge branch 'master' into marin/plugin-ml-4-completion-performace

Initialize completion kind name with Enum

Add core module to CommunityRepositoryModules

Update model & remove legacy support

Support legacy completion kinds ranking

Fix kind features computation for ordering

Fix style

Reintroduce CompletionKind

Fix fast lookup appearance

Rename main interfaces

Remove plugin's core dependency from completion-ml-ranking

Cache completion kind features in lookup

Bundle core into community

Merge branch 'master' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	intellij.idea.ultimate.main.iml

Add todo

Add completion kind features

Try fixing build schema

Try fixing build schema

Try fixing build schema

Add performance logging

Add kotlin module dependency to build

Add performance to the community

Put early lookup opener to the experiment

Add dependency on Kotlin by default

Add plugin to the build

Split java generator into the most frequent kinds

Rename api module to core

Add plugin to the build

Add performance completion to community

Update Kotlin's ML model version

Initialize completion performance for Java

Merge remote-tracking branch 'origin/marin/plugin-ml-4-completion-performace' into marin/plugin-ml-4-completion-performace

# Conflicts:
#	community/plugins/turboComplete/src/com/intellij/codeInsight/completion/KindExecutingCompletionContributor.kt
#	intellij.idea.ultimate.main.iml

Change default policy to buffering

Use local models for testing

Record actual performance status to context

Refactor plugin structure

Fix completion kind collection

Fix completion kind collection

Add registry configuration options

Implement kind collection & split reference for Kotlin

Implement fast completion for Kotlin

Initialize performance plugin

Use local models for testing

Record actual performance status to context

Refactor plugin structure

Fix completion kind collection

Fix completion kind collection

Add registry configuration options

Implement kind collection & split reference for Kotlin

Implement fast completion for Kotlin

Initialize performance plugin

Merge-request: IJ-MR-103789
Merged-by: Gleb Marin <Gleb.Marin@jetbrains.com>

GitOrigin-RevId: 1859b97a729b6c123fa22d6d9b6518e836ac6dec
This commit is contained in:
Gleb.Marin
2023-06-09 18:15:57 +00:00
committed by intellij-monorepo-bot
parent e1b2d92857
commit a71d5fc374
93 changed files with 4019 additions and 67 deletions

3
.idea/modules.xml generated
View File

@@ -1053,6 +1053,9 @@
<module fileurl="file://$PROJECT_DIR$/jps/standalone-builder/intellij.tools.jps.build.standalone.iml" filepath="$PROJECT_DIR$/jps/standalone-builder/intellij.tools.jps.build.standalone.iml" />
<module fileurl="file://$PROJECT_DIR$/tools/launcher-generator/intellij.tools.launcherGenerator.iml" filepath="$PROJECT_DIR$/tools/launcher-generator/intellij.tools.launcherGenerator.iml" />
<module fileurl="file://$PROJECT_DIR$/platform/testFramework/bootstrap/intellij.tools.testsBootstrap.iml" filepath="$PROJECT_DIR$/platform/testFramework/bootstrap/intellij.tools.testsBootstrap.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/turboComplete/intellij.turboComplete.iml" filepath="$PROJECT_DIR$/plugins/turboComplete/intellij.turboComplete.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/turboComplete/languages/kotlin/intellij.turboComplete.languages.kotlin.iml" filepath="$PROJECT_DIR$/plugins/turboComplete/languages/kotlin/intellij.turboComplete.languages.kotlin.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/turboComplete/intellij.turboComplete.tests.iml" filepath="$PROJECT_DIR$/plugins/turboComplete/intellij.turboComplete.tests.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/ui-designer-core/intellij.uiDesigner.iml" filepath="$PROJECT_DIR$/plugins/ui-designer-core/intellij.uiDesigner.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/changeReminder/intellij.vcs.changeReminder.iml" filepath="$PROJECT_DIR$/plugins/changeReminder/intellij.vcs.changeReminder.iml" />
<module fileurl="file://$PROJECT_DIR$/plugins/git4idea/intellij.vcs.git.iml" filepath="$PROJECT_DIR$/plugins/git4idea/intellij.vcs.git.iml" />

View File

@@ -101,6 +101,7 @@ val IDEA_BUNDLED_PLUGINS: PersistentList<String> = DEFAULT_BUNDLED_PLUGINS + per
"intellij.keymap.visualStudio",
"intellij.keymap.netbeans",
"intellij.performanceTesting",
"intellij.turboComplete",
)
val CE_CLASS_VERSIONS: PersistentMap<String, String> = BASE_CLASS_VERSIONS.putAll(persistentHashMapOf(

View File

@@ -282,6 +282,12 @@ object CommunityRepositoryModules {
},
plugin("intellij.editorconfig") { spec ->
spec.withProjectLibrary("ec4j-core")
},
plugin(
"intellij.turboComplete",
) { spec ->
spec.bundlingRestrictions.includeInEapOnly = true
spec.withModule("intellij.turboComplete.languages.kotlin")
}
)

View File

@@ -0,0 +1,32 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.codeInsight.completion
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.openapi.application.ApplicationManager
import com.intellij.openapi.application.invokeLater
import org.jetbrains.annotations.ApiStatus
/**
* A singleton that provides the functionality to open the code completion lookups pop-up.
*
* @see [LookupImpl]
* @see [CompletionProgressIndicator]
*/
@ApiStatus.Internal
object CompletionLookupOpener {
private fun LookupImpl.shownTimestampInitialized() = this.shownTimestampMillis != 0L
/**
* Schedules a request to open lookup on the AWT event dispatching thread under Write Intent lock.
* The request expires, if the lookup is already opened.
*/
fun showLookup(parameters: CompletionParameters) {
val process = parameters.process
if (process !is CompletionProgressIndicator) {
return
}
val lookup = process.lookup
if (lookup.shownTimestampInitialized()) return
ApplicationManager.getApplication().invokeLater(process::showLookup) { lookup.shownTimestampInitialized() }
}
}

View File

@@ -0,0 +1,95 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.codeInsight.completion
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.openapi.progress.ProgressManager
import com.intellij.patterns.ElementPattern
import com.intellij.util.Consumer
import org.jetbrains.annotations.ApiStatus
@ApiStatus.Internal
class FilteringResultSet(
private val base: CompletionResultSet,
private val filter: (CompletionContributor) -> Boolean
) : CompletionResultSet(base.prefixMatcher, base.consumer, base.myContributor) {
override fun addElement(element: LookupElement) {
base.addElement(element)
}
override fun withPrefixMatcher(matcher: PrefixMatcher): CompletionResultSet {
return FilteringResultSet(base.withPrefixMatcher(matcher), filter)
}
override fun withPrefixMatcher(prefix: String): CompletionResultSet {
return FilteringResultSet(base.withPrefixMatcher(prefix), filter)
}
override fun withRelevanceSorter(sorter: CompletionSorter): CompletionResultSet {
return FilteringResultSet(base.withRelevanceSorter(sorter), filter)
}
override fun addLookupAdvertisement(text: String) {
base.addLookupAdvertisement(text)
}
override fun caseInsensitive(): CompletionResultSet {
return FilteringResultSet(base.caseInsensitive(), filter)
}
override fun restartCompletionOnPrefixChange(prefixCondition: ElementPattern<String>?) {
base.restartCompletionWhenNothingMatches()
}
override fun restartCompletionWhenNothingMatches() {
base.restartCompletionWhenNothingMatches()
}
override fun runRemainingContributors(parameters: CompletionParameters,
consumer: Consumer<in CompletionResult>,
stop: Boolean,
customSorter: CompletionSorter?) {
if (stop) {
stopHere()
}
val batchConsumer = object : BatchConsumer<CompletionResult?> {
override fun startBatch() {
this@FilteringResultSet.startBatch()
}
override fun endBatch() {
this@FilteringResultSet.endBatch()
}
override fun consume(result: CompletionResult?) {
consumer.consume(result)
}
}
myCompletionService.getVariantsFromContributors(parameters, myContributor, prefixMatcher, batchConsumer, customSorter, filter)
}
companion object {
private fun CompletionService.getVariantsFromContributors(parameters: CompletionParameters,
from: CompletionContributor,
matcher: PrefixMatcher,
consumer: Consumer<in CompletionResult?>,
customSorter: CompletionSorter?,
filter: (CompletionContributor) -> Boolean) {
val contributors = CompletionContributor.forParameters(parameters)
for (i in contributors.indexOf(from) + 1 until contributors.size) {
ProgressManager.checkCanceled()
val contributor = contributors[i]
if (!filter(contributor)) {
continue
}
var result: CompletionResultSet = FilteringResultSet(createResultSet(parameters, consumer, contributor, matcher), filter)
if (customSorter != null) {
result = result.withRelevanceSorter(customSorter)
}
getVariantsFromContributor(parameters, contributor, result)
if (result.isStopped) {
return
}
}
}
}
}

View File

@@ -0,0 +1,59 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.codeInsight.completion
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.patterns.ElementPattern
import org.jetbrains.annotations.ApiStatus
@ApiStatus.Internal
class PolicyObeyingResultSet(
private val originalResult: CompletionResultSet,
private val policyHolder: () -> ElementsAddingPolicy
) : CompletionResultSet(originalResult.prefixMatcher, originalResult.consumer, originalResult.myContributor) {
override fun addElement(element: LookupElement) {
policyHolder().addElement(originalResult, element)
}
override fun addAllElements(elements: MutableIterable<LookupElement>) {
policyHolder().addAllElements(originalResult, elements)
}
override fun withPrefixMatcher(matcher: PrefixMatcher): CompletionResultSet {
return PolicyObeyingResultSet(originalResult.withPrefixMatcher(matcher), policyHolder)
}
override fun withPrefixMatcher(prefix: String): CompletionResultSet {
return PolicyObeyingResultSet(originalResult.withPrefixMatcher(prefix), policyHolder)
}
override fun withRelevanceSorter(sorter: CompletionSorter): CompletionResultSet {
return PolicyObeyingResultSet(originalResult.withRelevanceSorter(sorter), policyHolder)
}
override fun addLookupAdvertisement(text: String) {
originalResult.addLookupAdvertisement(text)
}
override fun caseInsensitive(): CompletionResultSet {
return PolicyObeyingResultSet(originalResult.caseInsensitive(), policyHolder)
}
override fun restartCompletionOnPrefixChange(prefixCondition: ElementPattern<String>?) {
originalResult.restartCompletionOnPrefixChange(prefixCondition)
}
override fun restartCompletionWhenNothingMatches() {
originalResult.restartCompletionWhenNothingMatches()
}
override fun isStopped(): Boolean {
return originalResult.isStopped
}
override fun stopHere() {
policyHolder().onResultStop(originalResult)
originalResult.stopHere()
}
}

View File

@@ -0,0 +1,42 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.codeInsight.completion.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.lookup.LookupElement
import org.jetbrains.annotations.ApiStatus
/**
* Policy, that controls how exactly elements should be
* added to the result set
*/
@ApiStatus.Internal
interface ElementsAddingPolicy {
/**
* Called when the policy should come into effect
*
* @see #onDeactivate
*/
fun onActivate(result: CompletionResultSet) {}
/**
* Called when result's {@link com.intellij.codeInsight.completion.CompletionResultSet#stopHere} was called
*/
fun onResultStop(result: CompletionResultSet) {}
/**
* Called when another [element] should be added to [result]
*/
fun addElement(result: CompletionResultSet, element: LookupElement)
/**
* Called when all [elements] should be added to [result]
*/
fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>)
/**
* Called when the policy should end its action
*
* @see #onActivate
*/
fun onDeactivate(result: CompletionResultSet) {}
}

View File

@@ -0,0 +1,74 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.codeInsight.completion.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.PolicyObeyingResultSet
import org.jetbrains.annotations.ApiStatus
import com.intellij.util.containers.Stack
/**
* An intermediary instance, that controls the policy of in which manner
* should completion results be added to the [originalResult]
*/
@ApiStatus.Internal
class PolicyController(protected val originalResult: CompletionResultSet) : () -> ElementsAddingPolicy {
private val policies: Stack<ElementsAddingPolicy> = Stack()
/**
* Make the [policy] rule how elements are added to the [originalResult]
* If there is already an active policy A in the controller, than it
* will be put on the stack. So that when the newly added policy will
* be popped, the policy A will be in action again.
*
* @see popPolicy
*/
fun pushPolicy(policy: ElementsAddingPolicy) {
policies.push(policy)
policy.onActivate(originalResult)
}
/**
* Revoke currently active policy
*
* @throws NoActivePolicyException if there is no active policy
* @see [pushPolicy]
*/
fun popPolicy() {
verifyNotEmptyStack()
val policyToDeactivate = policies.pop()
policyToDeactivate.onDeactivate(originalResult)
}
/**
* @return A result set, that will be obeying to this controller
*/
fun getObeyingResultSet(): CompletionResultSet {
return PolicyObeyingResultSet(originalResult, this)
}
/**
* Invoke the given action
*/
fun <T> invokeWithPolicy(policy: ElementsAddingPolicy, action: () -> T): T {
pushPolicy(policy)
try {
return action()
}
finally {
popPolicy()
}
}
override fun invoke(): ElementsAddingPolicy {
verifyNotEmptyStack()
return policies.peek()!!
}
private fun verifyNotEmptyStack() {
if (policies.isEmpty()) {
throw NoActivePolicyException()
}
}
public class NoActivePolicyException : Exception("ElementsAddingPolicyController does not have an active policy")
}

View File

@@ -20,5 +20,6 @@
<orderEntry type="module" module-name="intellij.platform.testFramework" scope="TEST" />
<orderEntry type="library" scope="TEST" name="assertJ" level="project" />
<orderEntry type="module" module-name="intellij.platform.util.text.matching" />
<orderEntry type="module" module-name="intellij.platform.lang.impl" />
</component>
</module>

View File

@@ -0,0 +1,11 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
/**
* Data class representing a kind of [SuggestionGenerator]'s suggestions.
*
* Each completion kind belongs to a kind variety.
* The completion kind's name is defined statically, it should be unique among
* the corresponding [KindVariety].
*/
data class CompletionKind(val name: Enum<*>, val variety: KindVariety)

View File

@@ -0,0 +1,20 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
class ConditionalConsumer(
base: SuggestionGeneratorConsumer,
private val shouldExecuteKind: (SuggestionGenerator) -> Boolean,
) : DelegatingConsumer(base) {
override fun pass(suggestionGenerator: SuggestionGenerator) {
val generator = SuggestionGenerator.withApplicability(
suggestionGenerator.kind,
suggestionGenerator.result,
suggestionGenerator.parameters,
suggestionGenerator::generateCompletionVariants
) {
shouldExecuteKind(suggestionGenerator)
}
super.pass(generator)
}
}

View File

@@ -0,0 +1,4 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
open class DelegatingConsumer(private val base: SuggestionGeneratorConsumer) : SuggestionGeneratorConsumer by base

View File

@@ -0,0 +1,53 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.openapi.extensions.ExtensionPointName
/**
* Collects all [SuggestionGenerator] of the same variety,
* without executing them.
*
* The purpose of this class it to just acknowledge existence of
* the generators, so then we can reorder the generators, and
* run them in the order we want.
*/
interface KindCollector {
/**
* The variety of completion suggestions generated by collected [SuggestionGenerator]s
*
* An analogue of [com.intellij.codeInsight.completion.CompletionContributor],
* but instead of executing all suggestion generators immediately, it delegates their execution
*/
val kindVariety: KindVariety
/**
* Checks if this collector is relevant within the given parameters
*/
fun shouldBeCalled(parameters: CompletionParameters): Boolean
/**
* Collects and adds [SuggestionGenerator]s to the consumer
*
* An analogue of [com.intellij.codeInsight.completion.CompletionContributor.fillCompletionVariants]
*
* @param parameters The completion parameters containing information about the current completion request.
* @param generatorConsumer The suggestion generator consumer used to collect [SuggestionGenerator]s.
* @param result The completion result set to which the suggestions will be added.
*/
fun collectKinds(parameters: CompletionParameters,
generatorConsumer: SuggestionGeneratorConsumer,
result: CompletionResultSet,
resultPolicyController: PolicyController)
companion object {
@JvmStatic
val EP_NAME: ExtensionPointName<KindCollector> = ExtensionPointName.create(
"com.intellij.turboComplete.kindCollector")
fun forParameters(parameters: CompletionParameters): List<KindCollector> {
return EP_NAME.extensionList.filter { it.shouldBeCalled(parameters) }
}
}
}

View File

@@ -0,0 +1,50 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
/**
* Listens how [SuggestionGenerator]s are executed
* Be aware, that each listener is not reinitialized, but
* the same instance used to convey information about generators'
* execution each time.
*
* The functions are called in their declaration order.
*/
interface KindExecutionListener {
/**
* Code completion was just called
*/
fun onInitialize(parameters: CompletionParameters)
/**
* Called before any [KindCollector] has collected any kinds,
* so the collection process just began
*/
fun onCollectionStarted() {}
/**
* A [SuggestionGenerator] had been collected.
* There could be multiple suggestion generators, hence,
* the function is called as many times.
*/
fun onGeneratorCollected(suggestionGenerator: SuggestionGenerator) {}
/**
* All kinds have been collected
*/
fun onCollectionFinished() {}
/**
* The generator started generating variants,
* i.e. [SuggestionGenerator.generateCompletionVariants] was called
*/
fun onGenerationStarted(suggestionGenerator: SuggestionGenerator) {}
/**
* The generator finished generating variants,
* i.e. [SuggestionGenerator.generateCompletionVariants] finished
* (either by an exception, or without)
*/
fun onGenerationFinished(suggestionGenerator: SuggestionGenerator) {}
}

View File

@@ -0,0 +1,26 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
/**
* Represents a family of completion kinds.
* One [KindCollector] collects kinds from the same family (from Java, Kotlin K1, Kotlin K2, etc.)
*/
interface KindVariety {
/**
* Checks, if the kind variety can be collected withing the given parameters
*/
fun kindsCorrespondToParameters(parameters: CompletionParameters): Boolean
/**
* Temporary workaround
*
* Currently, [SuggestionGenerator] is a "fixed"
* version of a [com.intellij.codeInsight.completion.CompletionContributor].
* We can't dynamically remove one contributor, so we need to filter it out, so we don't have duplicates
* (suggestions from the contributor, and from the duplicating suggestion generator).
* And to do this, we must know, what is the actual contributor, that we are filtering out.
*/
val actualCompletionContributorClass: Class<*>
}

View File

@@ -0,0 +1,75 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.openapi.util.Key
/**
* Generates suggestions for code completion. A simplified version of
* [com.intellij.codeInsight.completion.CompletionContributor]
*
* Instances of this interface are generated by [KindCollector],
* and then executed
*/
interface SuggestionGenerator {
/**
* Kind of completion variants, which will be suggested by this particular generator
*/
val kind: CompletionKind
/**
* Result set, to where the generated completion variants will be stored after
* [generateCompletionVariants] has been called
*/
val result: CompletionResultSet
/**
* Parameters, which the generated suggestions will be satisfying
*/
val parameters: CompletionParameters
/**
* Generates completion variants, which correspond to [parameters] and puts them
* to [result]
*/
fun generateCompletionVariants()
companion object {
@JvmStatic
val LOOKUP_ELEMENT_SUGGESTION_GENERATOR = Key<SuggestionGenerator>("SuggestionGenerator which generated the lookup element")
@JvmStatic
fun fromGenerator(kind: CompletionKind,
parameters: CompletionParameters,
result: CompletionResultSet,
generateVariants: () -> Unit): SuggestionGenerator {
return object : SuggestionGenerator {
override val kind = kind
override val result = result
override val parameters = parameters
override fun generateCompletionVariants() {
generateVariants()
}
}
}
@JvmStatic
fun withApplicability(kind: CompletionKind,
result: CompletionResultSet,
parameters: CompletionParameters,
generator: () -> Unit,
isApplicable: () -> Boolean): SuggestionGenerator {
return fromGenerator(
kind,
parameters,
result
) { if (isApplicable()) generator() else Unit }
}
val LookupElement.suggestionGenerator
get() = this.getUserData(LOOKUP_ELEMENT_SUGGESTION_GENERATOR)
}
}

View File

@@ -0,0 +1,10 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
/**
* An interface for classes that consume suggestion generators,
* generated by [KindCollector]
*/
interface SuggestionGeneratorConsumer {
fun pass(suggestionGenerator: SuggestionGenerator)
}

View File

@@ -0,0 +1,62 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.platform.ml.impl.turboComplete.addingPolicy.CollectionFillingPolicy
import com.intellij.platform.ml.impl.turboComplete.addingPolicy.PassDirectlyPolicy
/**
* Generates completion suggestions for a certain type of completion kind and stores
* a cached artifact for later use.
*
* For example, we have two [SuggestionGenerator]s A and B.
* While working, A generates an artifact, that is used later by B.
* And the ML model decided, that B must be executed earlier.
*
* If we execute B first, then we want to preserve the recommended order of the
* generators, and we don't want A to add completion variants to the lookup first.
* Instead, A could be a [SuggestionGeneratorWithArtifact]. Then
* 1. B asks A to create the artifact - [getArtifact]
* 2. B generates completion variants
* (it's A's order now to generate variants)
* 3. A will only collect put cached lookup elements to the result set
*/
abstract class SuggestionGeneratorWithArtifact<T>(override val kind: CompletionKind,
override val result: CompletionResultSet,
private val resultPolicyController: PolicyController,
override val parameters: CompletionParameters) : SuggestionGenerator {
private var cachedArtifact: T? = null
private var cachedLookupElements: MutableList<LookupElement>? = null
fun getArtifact(): T {
return cachedArtifact ?: run {
val generatedLookupElements = mutableListOf<LookupElement>()
val generatedArtifact = resultPolicyController.invokeWithPolicy(CollectionFillingPolicy(generatedLookupElements)) {
generateVariantsAndArtifact()
}
cachedLookupElements = generatedLookupElements
cachedArtifact = generatedArtifact
generatedArtifact
}
}
override fun generateCompletionVariants() {
cachedLookupElements?.let {
if (it.isEmpty()) return
resultPolicyController.invokeWithPolicy(PassDirectlyPolicy()) {
result.addAllElements(it)
}
it.clear()
} ?: run {
cachedArtifact = resultPolicyController.invokeWithPolicy(PassDirectlyPolicy()) {
generateVariantsAndArtifact()
}
}
}
abstract fun generateVariantsAndArtifact(): T
}

View File

@@ -0,0 +1,20 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
/**
* Fill the given collection, when an element is added
*/
class CollectionFillingPolicy(private val addedElements: MutableCollection<LookupElement>) : ElementsAddingPolicy {
override fun addElement(result: CompletionResultSet, element: LookupElement) {
addedElements.add(element)
}
override fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>) {
addedElements.addAll(elements)
}
}

View File

@@ -0,0 +1,19 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
/**
* Pass all elements directly to the result set
*/
class PassDirectlyPolicy : ElementsAddingPolicy {
override fun addElement(result: CompletionResultSet, element: LookupElement) {
result.addElement(element)
}
override fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>) {
result.addAllElements(elements)
}
}

View File

@@ -0,0 +1,25 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete.ranking
/**
* Listens to the ranking process of [com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator]
*
* The same instance of the listener used during all the application lifetime.
* The callbacks are called in the order of their declaration
*/
interface KindRankingListener {
/**
* The ranking process started
*/
fun onRankingStarted() {}
/**
* The kinds were ranked
*/
fun onRanked(ranked: List<RankedKind>) {}
/**
* The ranking process finished
*/
fun onRankingFinished() {}
}

View File

@@ -0,0 +1,20 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.platform.ml.impl.turboComplete.ranking
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
data class RankedKind(
val kind: CompletionKind,
val relevance: Double?,
) {
companion object {
fun fromWeights(
kindWeights: Iterable<Pair<CompletionKind, Double>>,
negateWeight: Boolean,
): List<RankedKind> {
return kindWeights
.sortedBy { (_, weight) -> if (negateWeight) -weight else weight }
.map { (kind, weight) -> RankedKind(kind, weight) }
}
}
}

View File

@@ -57,6 +57,27 @@
"useMLRanking": true,
"showArrows": false,
"calculateFeatures": true
},
{
"number": 17,
"description": "Completion performance (early lookup & MLRanking)",
"useMLRanking": true,
"showArrows": false,
"calculateFeatures": true
},
{
"number": 18,
"description": "Completion performance (early lookup)",
"useMLRanking": false,
"showArrows": false,
"calculateFeatures": true
},
{
"number": 19,
"description": "Completion performance",
"useMLRanking": false,
"showArrows": false,
"calculateFeatures": true
}
],
"languages": [
@@ -75,8 +96,8 @@
{
"id": "kotlin",
"experimentBucketsCount": 8,
"includeGroups": [ 7, 8, 11, 12, 13 ],
"shouldLogElementFeatures": false
"includeGroups": [ 7, 8, 8, 12, 13, 17, 17, 17 ],
"shouldLogElementFeatures": true
},
{
"id": "scala",

View File

@@ -7,6 +7,7 @@ import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.ElementFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.openapi.util.Key
class CommonElementLocationFeatures : ElementFeatureProvider {
override fun getName(): String = "common"
@@ -34,9 +35,15 @@ class CommonElementLocationFeatures : ElementFeatureProvider {
}
element.getUserData(BaseCompletionService.LOOKUP_ELEMENT_CONTRIBUTOR)?.let {
result["contributor"] = MLFeatureValue.className(it::class.java)
val actualCompletionContributor: Class<*>? = element.getUserData(LOOKUP_ORIGINAL_ELEMENT_CONTRIBUTOR_TYPE)
result["contributor"] = MLFeatureValue.className(actualCompletionContributor ?: it::class.java)
}
return result
}
companion object {
// For the TurboComplete plugin, to not confuse the ML model with new unknown contributors
val LOOKUP_ORIGINAL_ELEMENT_CONTRIBUTOR_TYPE = Key<Class<*>>("original contributor of the element")
}
}

View File

@@ -40,8 +40,6 @@ class CommonLocationFeatures : ContextFeatureProvider {
"chars_in_line_after_caret" to MLFeatureValue.float(lineEndOffset - caretOffset)
)
result["is_completion_performance_mode"] = MLFeatureValue.binary(false)
if (DumbService.isDumb(lookup.project)) {
result["dumb_mode"] = MLFeatureValue.binary(true)
}

View File

@@ -10,12 +10,25 @@ import java.util.concurrent.atomic.LongAdder
class MLCompletionPerformanceTracker {
private val tracker: MeasuredTracker = MeasuredTracker(OTelTracker())
private val measuredTracker = MeasuredTracker()
private val tracker = DelegatingTracker(listOf(OTelTracker(), measuredTracker))
private val elementProvidersMeasurer: ConcurrentHashMap<String, LongAdder> = ConcurrentHashMap()
private var sortingCount = 0
private var totalMlContribution: Long = 0L
private val metricCollectors: MutableList<PerformanceMetricCollector> = mutableListOf()
fun addMetricCollector(metricCollectorFactory: PerformanceMetricCollectorFactory) {
val performanceTracker = object : PerformanceTracker {
override fun addByKey(key: String, timeMs: Long) {
tracker.addByKey("${metricCollectorFactory.performanceMetricName}.$key", timeMs)
}
}
val metricCollector = metricCollectorFactory.createMetricCollector(performanceTracker)
metricCollectors.add(metricCollector)
}
fun totalMLTimeContribution(): Long = totalMlContribution
fun sortingPerformed(itemsCount: Int, totalTime: Long) {
@@ -55,8 +68,9 @@ class MLCompletionPerformanceTracker {
}
fun measurements(): Map<String, Long> {
metricCollectors.forEach { it.onFinishCollecting() }
flushElementProvidersContribution()
return tracker.measurements()
return measuredTracker.measurements()
}
private fun flushElementProvidersContribution() {
@@ -70,11 +84,7 @@ class MLCompletionPerformanceTracker {
}
}
private interface PerfTracker {
fun addByKey(key: String, timeMs: Long)
}
private class OTelTracker : PerfTracker {
private class OTelTracker : PerformanceTracker {
private val meter = TelemetryTracer.getMeter(CompletionRanking)
private val key2counter: MutableMap<String, LongCounter> = mutableMapOf()
override fun addByKey(key: String, timeMs: Long) {
@@ -84,10 +94,15 @@ class MLCompletionPerformanceTracker {
}
}
private class MeasuredTracker(private val delegate: PerfTracker) : PerfTracker {
private class DelegatingTracker(private val trackers: List<PerformanceTracker>) : PerformanceTracker {
override fun addByKey(key: String, timeMs: Long) {
trackers.forEach { it.addByKey(key, timeMs) }
}
}
private class MeasuredTracker : PerformanceTracker {
private val measurements: ConcurrentHashMap<String, LongAdder> = ConcurrentHashMap()
override fun addByKey(key: String, timeMs: Long) {
delegate.addByKey(key, timeMs)
measurements.computeIfAbsent(key) { LongAdder() }.add(timeMs)
}

View File

@@ -0,0 +1,6 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.completion.ml.performance
interface PerformanceMetricCollector {
fun onFinishCollecting() {}
}

View File

@@ -0,0 +1,8 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.completion.ml.performance
interface PerformanceMetricCollectorFactory {
val performanceMetricName: String
fun createMetricCollector(tracker: PerformanceTracker): PerformanceMetricCollector
}

View File

@@ -0,0 +1,6 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.completion.ml.performance
interface PerformanceTracker {
fun addByKey(key: String, timeMs: Long)
}

View File

@@ -2,17 +2,17 @@
package com.intellij.completion.ml.ranker.local
import com.intellij.internal.ml.DecisionFunction
import com.intellij.openapi.diagnostic.Logger
import com.intellij.openapi.diagnostic.logger
import com.intellij.openapi.util.registry.Registry
import com.intellij.util.concurrency.SequentialTaskExecutor
import java.util.*
import java.util.concurrent.Future
import java.util.zip.ZipFile
object MLCompletionLocalModelsUtil {
private const val REGISTRY_PATH_KEY = "completion.ml.path.to.zip.model"
private val LOG = Logger.getInstance(MLCompletionLocalModelsUtil::class.java)
class MLCompletionLocalModelsLoader(private val registryPathKey: String) {
private val executor = SequentialTaskExecutor.createSequentialApplicationPoolExecutor("MLCompletionTxtModelsUtil pool executor")
@Volatile private var localModel: LocalModalInfo? = null
@Volatile
private var localModel: LocalModalInfo? = null
fun getModel(languageId: String): DecisionFunction? {
if (!isPathToTheModelSet()) {
@@ -25,7 +25,7 @@ object MLCompletionLocalModelsUtil {
}
val resLocalModel = localModel ?: return null
return if (languageId.toLowerCase() in resLocalModel.languages) {
return if (languageId.lowercase(Locale.getDefault()) in resLocalModel.languages) {
resLocalModel.decisionFunction
}
else {
@@ -38,14 +38,14 @@ object MLCompletionLocalModelsUtil {
*/
private fun scheduleInitModel(): Future<*> = executor.submit { initModelFromPathToZipSynchronously() }
private fun isPathToTheModelSet() = Registry.get(REGISTRY_PATH_KEY).isChangedFromDefault
private fun isPathToTheModelSet() = Registry.get(registryPathKey).isChangedFromDefault
private fun isPathToTheModelChanged() = Registry.stringValue(REGISTRY_PATH_KEY) != localModel?.path
private fun isPathToTheModelChanged() = Registry.stringValue(registryPathKey) != localModel?.path
private fun initModelFromPathToZipSynchronously() {
localModel = null
val startTime = System.currentTimeMillis()
val pathToZip = Registry.stringValue(REGISTRY_PATH_KEY)
val pathToZip = Registry.stringValue(registryPathKey)
localModel = loadModel(pathToZip)
val endTime = System.currentTimeMillis()
LOG.info("ML Completion local model initialization took: ${endTime - startTime} ms.")
@@ -58,11 +58,16 @@ object MLCompletionLocalModelsUtil {
val (decisionFunction, languages) = loader.loadModel(file)
return LocalModalInfo(decisionFunction, pathToZip, languages.toSet())
}
} catch (t: Throwable) {
}
catch (t: Throwable) {
LOG.error(t)
return null
}
}
private data class LocalModalInfo(val decisionFunction: DecisionFunction, val path: String, val languages: Set<String>)
companion object {
private val LOG = logger<MLCompletionLocalModelsLoader>()
}
}

View File

@@ -0,0 +1,45 @@
package com.intellij.completion.ml.sorting
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.ml.CompletionEnvironment
import com.intellij.codeInsight.completion.ml.ContextFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.Lookup
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.completion.ml.storage.MutableLookupStorage
import com.intellij.openapi.extensions.impl.ExtensionProcessingHelper
import com.intellij.openapi.progress.ProgressManager
import com.intellij.openapi.util.UserDataHolderBase
import java.util.concurrent.TimeUnit
import kotlin.system.measureTimeMillis
object ContextFactorCalculator {
fun calculateContextFactors(lookup: LookupImpl, parameters: CompletionParameters, storage: MutableLookupStorage) {
val environment = MyEnvironment(lookup, parameters)
val contextFeatures = mutableMapOf<String, MLFeatureValue>()
ExtensionProcessingHelper.forEachExtensionSafe(ContextFeatureProvider.forLanguage(storage.language)) { provider ->
ProgressManager.checkCanceled()
val providerName = provider.name
val timeSpent = measureTimeMillis {
val features = provider.calculateFeatures(environment)
for ((featureName, value) in features) {
contextFeatures["ml_ctx_${providerName}_$featureName"] = value
}
}
storage.performanceTracker.contextFeaturesCalculated(providerName, TimeUnit.NANOSECONDS.toMillis(timeSpent))
}
for (contextFeatureProvider in AdditionalContextFeatureProvider.forLanguage(storage.language)) {
contextFeatures.putAll(contextFeatureProvider.calculateFeatures(contextFeatures))
}
storage.initContextFactors(contextFeatures, environment)
}
}
private class MyEnvironment(
private val lookup: LookupImpl,
private val parameters: CompletionParameters
) : CompletionEnvironment, UserDataHolderBase() {
override fun getLookup(): Lookup = lookup
override fun getParameters(): CompletionParameters = parameters
}

View File

@@ -4,19 +4,11 @@ package com.intellij.completion.ml.sorting
import com.intellij.codeInsight.completion.CompletionContributor
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.ml.CompletionEnvironment
import com.intellij.codeInsight.completion.ml.ContextFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.Lookup
import com.intellij.codeInsight.lookup.LookupManager
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.completion.ml.CompletionMLPolicy
import com.intellij.openapi.progress.ProgressManager
import com.intellij.openapi.project.DumbAware
import com.intellij.openapi.util.UserDataHolderBase
import com.intellij.completion.ml.storage.MutableLookupStorage
import com.intellij.openapi.extensions.impl.ExtensionProcessingHelper
import java.util.concurrent.TimeUnit
class ContextFeaturesContributor : CompletionContributor(), DumbAware {
override fun fillCompletionVariants(parameters: CompletionParameters, result: CompletionResultSet) {
@@ -29,42 +21,10 @@ class ContextFeaturesContributor : CompletionContributor(), DumbAware {
storage.disableReRanking()
}
if (storage.shouldComputeFeatures() && !storage.isContextFactorsInitialized()) {
calculateContextFactors(lookup, parameters, storage)
ContextFactorCalculator.calculateContextFactors(lookup, parameters, storage)
}
}
}
super.fillCompletionVariants(parameters, result)
}
private fun calculateContextFactors(lookup: LookupImpl, parameters: CompletionParameters, storage: MutableLookupStorage) {
val environment = MyEnvironment(lookup, parameters)
val contextFeatures = mutableMapOf<String, MLFeatureValue>()
ExtensionProcessingHelper.forEachExtensionSafe(ContextFeatureProvider.forLanguage(storage.language)) { provider ->
ProgressManager.checkCanceled()
val providerName = provider.name
val start = System.nanoTime()
val features = provider.calculateFeatures(environment)
for ((featureName, value) in features) {
contextFeatures["ml_ctx_${providerName}_$featureName"] = value
}
val timeSpent = System.nanoTime() - start
storage.performanceTracker.contextFeaturesCalculated(providerName, TimeUnit.NANOSECONDS.toMillis(timeSpent))
}
for (contextFeatureProvider in AdditionalContextFeatureProvider.forLanguage(storage.language)) {
contextFeatures.putAll(contextFeatureProvider.calculateFeatures(contextFeatures))
}
storage.initContextFactors(contextFeatures, environment)
}
private class MyEnvironment(
private val lookup: LookupImpl,
private val parameters: CompletionParameters
) : CompletionEnvironment, UserDataHolderBase() {
override fun getLookup(): Lookup = lookup
override fun getParameters(): CompletionParameters = parameters
}
}

View File

@@ -4,7 +4,7 @@ package com.intellij.completion.ml.sorting
import com.intellij.completion.ml.experiment.ExperimentStatus
import com.intellij.completion.ml.ranker.ExperimentModelProvider
import com.intellij.completion.ml.ranker.ExperimentModelProvider.Companion.match
import com.intellij.completion.ml.ranker.local.MLCompletionLocalModelsUtil
import com.intellij.completion.ml.ranker.local.MLCompletionLocalModelsLoader
import com.intellij.completion.ml.settings.CompletionMLRankingSettings
import com.intellij.internal.ml.completion.DecoratingItemsPolicy
import com.intellij.internal.ml.completion.RankingModelProvider
@@ -18,9 +18,11 @@ import org.jetbrains.annotations.TestOnly
object RankingSupport {
private val LOG = logger<RankingSupport>()
private var enabledInTests: Boolean = false
private val localDebugModelLoader = MLCompletionLocalModelsLoader("completion.ml.path.to.zip.model")
fun getRankingModel(language: Language): RankingModelWrapper? {
MLCompletionLocalModelsUtil.getModel(language.id)?.let { return LanguageRankingModel(it, DecoratingItemsPolicy.DISABLED) }
tryLoadLocalDebugModel(language)?.let { return it }
val provider = findProviderSafe(language)
return if (provider != null && shouldSortByML(language, provider)) tryGetModel(provider) else null
}
@@ -73,6 +75,12 @@ object RankingSupport {
return shouldSort
}
private fun tryLoadLocalDebugModel(language: Language): LanguageRankingModel? {
return localDebugModelLoader.getModel(language.id)?.let {
return LanguageRankingModel(it, DecoratingItemsPolicy.DISABLED)
}
}
@TestOnly
fun enableInTests(parentDisposable: Disposable) {
enabledInTests = true

View File

@@ -38,5 +38,6 @@
<orderEntry type="module" module-name="intellij.java.impl" />
<orderEntry type="module" module-name="kotlin.code-insight.api" />
<orderEntry type="module" module-name="intellij.platform.util.text.matching" />
<orderEntry type="module" module-name="intellij.platform.ml.impl" />
</component>
</module>

View File

@@ -0,0 +1,173 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package org.jetbrains.kotlin.idea.completion
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.CompletionType
import com.intellij.codeInsight.completion.PrefixMatcher
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.codeInsight.completion.impl.CamelHumpMatcher
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.openapi.extensions.InternalIgnoreDependencyViolation
import com.intellij.openapi.util.ThrowableComputable
import com.intellij.openapi.util.registry.Registry
import com.intellij.patterns.PsiJavaPatterns.elementType
import com.intellij.patterns.PsiJavaPatterns.psiElement
import com.intellij.platform.ml.impl.turboComplete.ConditionalConsumer
import com.intellij.platform.ml.impl.turboComplete.KindCollector
import com.intellij.platform.ml.impl.turboComplete.SuggestionGeneratorConsumer
import com.intellij.psi.PsiComment
import com.intellij.util.indexing.DumbModeAccessType
import org.jetbrains.kotlin.idea.completion.implCommon.stringTemplates.StringTemplateCompletion
import org.jetbrains.kotlin.idea.completion.smart.SmartCompletionSession
import org.jetbrains.kotlin.idea.completion.stringTemplates.wrapLookupElementForStringTemplateAfterDotCompletion
import org.jetbrains.kotlin.lexer.KtTokens
import org.jetbrains.kotlin.psi.KtFile
import org.jetbrains.kotlin.psi.KtNameReferenceExpression
import org.jetbrains.kotlin.psi.psiUtil.getNonStrictParentOfType
/**
* Collects [com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator]s for Kotlin K1 code completion.
* It is an analogue of [KotlinCompletionContributor]
*/
@InternalIgnoreDependencyViolation
class KotlinKindCollector : KindCollector {
override val kindVariety = KotlinKindVariety
private val AFTER_NUMBER_LITERAL = psiElement().afterLeafSkipping(
psiElement().withText(""),
psiElement().withElementType(elementType().oneOf(KtTokens.FLOAT_LITERAL, KtTokens.INTEGER_LITERAL))
)
private val AFTER_INTEGER_LITERAL_AND_DOT = psiElement().afterLeafSkipping(
psiElement().withText("."),
psiElement().withElementType(elementType().oneOf(KtTokens.INTEGER_LITERAL))
)
override fun shouldBeCalled(parameters: CompletionParameters): Boolean {
val position = parameters.position
val parametersOriginFile = parameters.originalFile
return position.containingFile is KtFile && parametersOriginFile is KtFile
}
override fun collectKinds(
parameters: CompletionParameters,
generatorConsumer: SuggestionGeneratorConsumer,
result: CompletionResultSet,
resultPolicyController: PolicyController
) {
if (!shouldBeCalled(parameters)) return
StringTemplateCompletion.correctParametersForInStringTemplateCompletion(parameters)?.let { correctedParameters ->
generateCompletionKinds(correctedParameters, generatorConsumer, result, resultPolicyController,
::wrapLookupElementForStringTemplateAfterDotCompletion)
return
}
DumbModeAccessType.RELIABLE_DATA_ONLY.ignoreDumbMode(ThrowableComputable {
generateCompletionKinds(parameters, generatorConsumer, result, resultPolicyController, null)
})
}
private fun generateCompletionKinds(parameters: CompletionParameters,
suggestionGeneratorConsumer: SuggestionGeneratorConsumer,
result: CompletionResultSet,
resultPolicyController: PolicyController,
lookupElementPostProcessor: ((LookupElement) -> LookupElement)?
) {
val position = parameters.position
if (position.getNonStrictParentOfType<PsiComment>() != null) {
// don't stop here, allow other contributors to run
return
}
if (shouldSuppressCompletion(parameters, result.prefixMatcher)) {
result.stopHere()
return
}
if (PackageDirectiveCompletion.perform(parameters, result)) {
result.stopHere()
return
}
for (extension in KotlinCompletionExtension.EP_NAME.extensionList) {
if (extension.perform(parameters, result)) return
}
fun addPostProcessor(session: CompletionSession) {
if (lookupElementPostProcessor != null) {
session.addLookupElementPostProcessor(lookupElementPostProcessor)
}
}
result.restartCompletionWhenNothingMatches()
val configuration = CompletionSessionConfiguration(parameters)
if (parameters.completionType == CompletionType.BASIC) {
val session = BasicCompletionKindGenerationSession(configuration, parameters, resultPolicyController,
suggestionGeneratorConsumer)
addPostProcessor(session)
if (parameters.isAutoPopup && session.shouldDisableAutoPopup()) {
result.stopHere()
return
}
val primaryKindsAddedSomething = session.complete()
if (!primaryKindsAddedSomething && parameters.invocationCount < 2) {
val newConfiguration = CompletionSessionConfiguration(
useBetterPrefixMatcherForNonImportedClasses = false,
nonAccessibleDeclarations = false,
javaGettersAndSetters = true,
javaClassesNotToBeUsed = false,
staticMembers = parameters.invocationCount > 0,
dataClassComponentFunctions = true,
excludeEnumEntries = configuration.excludeEnumEntries,
)
val newSession = BasicCompletionKindGenerationSession(
newConfiguration, parameters, resultPolicyController,
ConditionalConsumer(suggestionGeneratorConsumer) {
session.somethingAddedToResult
}
)
addPostProcessor(newSession)
newSession.complete()
}
}
else {
val session = SmartCompletionSession(configuration, parameters, result)
addPostProcessor(session)
session.complete()
}
}
private fun shouldSuppressCompletion(parameters: CompletionParameters, prefixMatcher: PrefixMatcher): Boolean {
val position = parameters.position
val invocationCount = parameters.invocationCount
if (prefixMatcher is CamelHumpMatcher && prefixMatcher.isTypoTolerant) return true
// no completion inside number literals
if (AFTER_NUMBER_LITERAL.accepts(position)) return true
// no completion auto-popup after integer and dot
if (invocationCount == 0 && prefixMatcher.prefix.isEmpty() && AFTER_INTEGER_LITERAL_AND_DOT.accepts(position)) return true
if (invocationCount == 0 && Registry.`is`("kotlin.disable.auto.completion.inside.expression", false)) {
val originalPosition = parameters.originalPosition
val originalExpression = originalPosition?.getNonStrictParentOfType<KtNameReferenceExpression>()
val expression = position.getNonStrictParentOfType<KtNameReferenceExpression>()
if (expression != null && originalExpression != null &&
!expression.getReferencedName().startsWith(originalExpression.getReferencedName())
) {
return true
}
}
return false
}
}

View File

@@ -0,0 +1,40 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package org.jetbrains.kotlin.idea.completion
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import org.jetbrains.kotlin.idea.KotlinLanguage
enum class KotlinCompletionKindName {
DSL_FUNCTION,
SMART_ADDITIONAL_ITEM,
REFERENCE_BASIC,
REFERENCE_EXTENSION,
PACKAGE_NAME,
NAMED_ARGUMENT,
EXTENSION_FUNCTION_TYPE_VALUE,
CONTEXT_VARIABLE_TYPE_SC,
CONTEXT_VARIABLE_TYPE_REFERENCE,
STATIC_MEMBER_FROM_IMPORTS,
NON_IMPORTED,
DEBUGGER_VARIANTS,
STATIC_MEMBER_OBJECT_MEMBER,
STATIC_MEMBER_EXPLICIT_INHERITED,
STATIC_MEMBER_INACCESSIBLE,
KEYWORD_ONLY,
OPERATOR_NAME,
DECLARATION_NAME,
TOP_LEVEL_CLASS_NAME,
SUPER_QUALIFIER,
DECLARATION_NAME_FROM_UNRESOLVED_OVERRIDE,
PARAMETER_OR_VAR_NAME_AND_TYPE,
}
object KotlinKindVariety : KindVariety {
override fun kindsCorrespondToParameters(parameters: CompletionParameters): Boolean {
return parameters.position.language == KotlinLanguage.INSTANCE
}
override val actualCompletionContributorClass: Class<*>
get() = KotlinCompletionContributor::class.java
}

View File

@@ -119,6 +119,27 @@ class ReferenceVariantsCollector(
consumer(extensions)
}
data class ReferenceVariantsCollectors(
val basic: Lazy<ReferenceVariants>,
val extensions: Lazy<ReferenceVariants>
)
fun makeReferenceVariantsCollectors(descriptorKindFilter: DescriptorKindFilter): ReferenceVariantsCollectors {
val config = configuration(descriptorKindFilter)
val basic = lazy {
assert(!isCollectingFinished)
collectBasicVariants(config).fixDescriptors()
}
val extensions = lazy {
assert(!isCollectingFinished)
collectExtensionVariants(config, basic.value).fixDescriptors()
}
return ReferenceVariantsCollectors(basic, extensions)
}
private fun collectBasicVariants(filterConfiguration: FilterConfiguration): ReferenceVariants {
val variants = doCollectBasicVariants(filterConfiguration)
collectedImported += variants.imported

View File

@@ -0,0 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/resources" type="java-resource" />
<sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="kotlin-stdlib" level="project" />
<orderEntry type="module" module-name="intellij.platform.analysis" />
<orderEntry type="module" module-name="intellij.platform.analysis.impl" />
<orderEntry type="module" module-name="intellij.platform.lang" />
<orderEntry type="module" module-name="intellij.platform.statistics" />
<orderEntry type="module" module-name="intellij.completionMlRanking" />
<orderEntry type="module" module-name="intellij.platform.lang.impl" />
<orderEntry type="module" module-name="intellij.platform.ml.impl" />
</component>
</module>

View File

@@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$/test">
<sourceFolder url="file://$MODULE_DIR$/test" isTestSource="true" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" scope="TEST" name="JUnit5" level="project" />
<orderEntry type="module" module-name="intellij.turboComplete" scope="TEST" />
<orderEntry type="library" scope="TEST" name="kotlin-stdlib" level="project" />
<orderEntry type="module" module-name="intellij.platform.analysis" scope="TEST" />
<orderEntry type="module" module-name="intellij.platform.ml.impl" scope="TEST" />
</component>
</module>

View File

@@ -0,0 +1,29 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/resources" type="java-resource" />
<sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="kotlin-stdlib" level="project" />
<orderEntry type="module" module-name="kotlin.completion.impl-k1" scope="PROVIDED" />
<orderEntry type="module" module-name="intellij.platform.ml.impl" />
<orderEntry type="module-library">
<library name="completion-performance-kotlin" type="repository">
<properties maven-id="org.jetbrains.intellij.deps.completion:performance-kotlin:0.0.6" />
<CLASSES>
<root url="jar://$MAVEN_REPOSITORY$/org/jetbrains/intellij/deps/completion/performance-kotlin/0.0.6/performance-kotlin-0.0.6.jar!/" />
</CLASSES>
<JAVADOC />
<SOURCES>
<root url="jar://$MAVEN_REPOSITORY$/org/jetbrains/intellij/deps/completion/performance-kotlin/0.0.6/performance-kotlin-0.0.6-sources.jar!/" />
</SOURCES>
</library>
</orderEntry>
<orderEntry type="module" module-name="intellij.turboComplete" />
<orderEntry type="module" module-name="intellij.platform.ide.impl" />
</component>
</module>

View File

@@ -0,0 +1,13 @@
<idea-plugin package="com.intellij.turboComplete.languages.kotlin">
<dependencies>
<plugin id="org.jetbrains.kotlin"/>
</dependencies>
<extensions defaultExtensionNs="com.intellij.turboComplete">
<kindCollector implementation="org.jetbrains.kotlin.idea.completion.KotlinKindCollector"/>
<suggestionGeneratorExecutorProvider implementation="com.intellij.turboComplete.languages.kotlin.MLKotlinSuggestionGeneratorExecutorProvider"/>
</extensions>
<extensions defaultExtensionNs="com.intellij">
<registryKey key="ml.completion.performance.localModel.kotlin" defaultValue="no path" description="Use local model for kotlin completion kind ordering"/>
</extensions>
</idea-plugin>

View File

@@ -0,0 +1,2 @@
completion.kind.sorter.ml.kotlin=Kotlin
presentation.tail.for.0.in.1="\ for {0} in {1}"

View File

@@ -0,0 +1,37 @@
package com.intellij.turboComplete.languages.kotlin
import com.intellij.internal.ml.catboost.CatBoostJarCompletionModelProvider
import com.intellij.lang.Language
import com.intellij.turboComplete.SortingExecutorPreferences
import com.intellij.turboComplete.SortingExecutorProvider
import com.intellij.turboComplete.ranking.MLKindSorterProvider
import com.intellij.turboComplete.ranking.provideLocalIfAny
import org.jetbrains.kotlin.idea.completion.KotlinKindVariety
class MLKotlinSuggestionGeneratorExecutorProvider : SortingExecutorProvider(
SortingExecutorPreferences(
policyForMostRelevant = SortingExecutorPreferences.MostRelevantKindPolicy.PASS_TO_RESULT,
policyForNoneKind = SortingExecutorPreferences.NoneKindPolicy.PASS_TO_RESULT,
executeMostRelevantWhenPassed = 3
)
) {
override val sorterProvider = run {
val mlSorterProvider = MLKindSorterProvider(KotlinKindVariety) { MlKotlinMultipleReferencesSorterProvider().model }
val maybeLocalSorterProvider = mlSorterProvider
.provideLocalIfAny("kotlin", "ml.completion.performance.localModel.kotlin")
maybeLocalSorterProvider
}
}
class MlKotlinMultipleReferencesSorterProvider : CatBoostJarCompletionModelProvider(
TurboCompleteKotlinBundle.message(
"completion.kind.sorter.ml.kotlin"),
"performance_kotlin_features", "performance_kotlin_model") {
override fun isLanguageSupported(language: Language): Boolean = language.id.compareTo("kotlin", ignoreCase = true) == 0
override fun isEnabledByDefault(): Boolean {
return true
}
}

View File

@@ -0,0 +1,20 @@
package com.intellij.turboComplete.languages.kotlin
import com.intellij.AbstractBundle
import com.intellij.DynamicBundle
import org.jetbrains.annotations.Nls
import org.jetbrains.annotations.PropertyKey
class TurboCompleteKotlinBundle : DynamicBundle(TURBO_COMPLETE_JAVA_BUNDLE) {
companion object {
private val ourInstance: AbstractBundle = TurboCompleteKotlinBundle()
private const val TURBO_COMPLETE_JAVA_BUNDLE = "messages.TurboCompleteKotlin"
@Nls
@JvmStatic
fun message(key: @PropertyKey(resourceBundle = TURBO_COMPLETE_JAVA_BUNDLE) String, vararg params: Any): String {
return ourInstance.getMessage(key, *params)
}
}
}

View File

@@ -0,0 +1,83 @@
<idea-plugin>
<id>com.intellij.turboComplete</id>
<name>Turbo Complete</name>
<vendor>JetBrains</vendor>
<category>Other Tools</category>
<description>
<![CDATA[
Experimental approach to speed up code completion.
<br><br>
The plugin implements a <a href="https://youtrack.jetbrains.com/issue/MLP-17/ML-for-IDE-Performance">new approach</a>
to improve code completion's performance perception.
Currently, it works only for Kotlin.
If you are experiencing problems with code completion, you could disable
the plugin.
]]>
</description>
<resource-bundle>messages.TurboComplete</resource-bundle>
<extensionPoints>
<extensionPoint
interface="com.intellij.platform.ml.impl.turboComplete.KindCollector"
dynamic="true"
name="kindCollector"/>
<extensionPoint
interface="com.intellij.turboComplete.analysis.PipelineListener"
dynamic="true"
name="analysis.pipelineListener"/>
<extensionPoint
interface="com.intellij.turboComplete.SuggestionGeneratorExecutorProvider"
dynamic="true"
name="suggestionGeneratorExecutorProvider"/>
<extensionPoint
interface="com.intellij.turboComplete.features.kind.KindFeatureProvider"
dynamic="true"
name="features.kind.provider"/>
</extensionPoints>
<extensions defaultExtensionNs="com.intellij">
<completion.contributor language="any"
id="kind_ordering_factors"
implementationClass="com.intellij.turboComplete.platform.contributor.KindOrderingFeaturesContributor"
order="first, after liveTemplates"/>
<completion.contributor language="any"
id="kind_executor"
implementationClass="com.intellij.turboComplete.platform.contributor.KindExecutingCompletionContributor"
order="first, after kind_ordering_factors, before duplicate_remover"/>
<completion.contributor language="any"
id="duplicate_remover"
implementationClass="com.intellij.turboComplete.platform.contributor.FilteringCompletionContributor"
order="first"/>
<completion.ml.contextFeatures language="" implementationClass="com.intellij.turboComplete.features.context.AllKindsUsageFeatures"/>
<completion.ml.contextFeatures language=""
implementationClass="com.intellij.turboComplete.features.context.CompletionPerformanceStatusFeatures"/>
<completion.ml.elementFeatures language="" implementationClass="com.intellij.turboComplete.features.element.ElementsKindFeatures"/>
<turboComplete.features.kind.provider implementation="com.intellij.turboComplete.features.kind.KindCommonFeatures"/>
<turboComplete.features.kind.provider implementation="com.intellij.turboComplete.features.kind.KindUsageFeatures"/>
<turboComplete.analysis.pipelineListener implementation="com.intellij.turboComplete.analysis.KindPerformanceRecorder"/>
<turboComplete.analysis.pipelineListener implementation="com.intellij.turboComplete.analysis.PipelinePerformanceRecorder"/>
<turboComplete.analysis.pipelineListener
implementation="com.intellij.turboComplete.analysis.usage.KindVarietyUsageTracker$UsagePipelineListener"/>
<turboComplete.analysis.pipelineListener implementation="com.intellij.turboComplete.platform.EarlyLookupOpener"/>
<registryKey key="ml.completion.performance.enable" defaultValue="true" description="Enable completion performance"/>
<registryKey key="ml.completion.performance.showLookupEarly" defaultValue="true"
description="Show lookup as soon as the first CompletionKind finished working"/>
<registryKey key="ml.completion.performance.experiment" defaultValue="true"
description="Perform an A/B experiment turning on and off performance"/>
<registryKey key="ml.completion.performance.executeImmediately" defaultValue="false"
description="Immediately execute suggestion generators"/>
</extensions>
<content>
<module name="intellij.turboComplete.languages.kotlin"/>
</content>
<depends>com.intellij.completion.ml.ranking</depends>
</idea-plugin>

View File

@@ -0,0 +1,82 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.completion.ml.common.CurrentProjectInfo
import com.intellij.completion.ml.experiment.ExperimentStatus
import com.intellij.openapi.util.registry.Registry
import com.intellij.platform.ml.impl.turboComplete.KindCollector
data class CompletionPerformanceParameters(
val enabled: Boolean,
val showLookupEarly: Boolean,
val fixedGeneratorsOrder: Boolean,
) {
companion object {
private fun parametrizeNoPerformance() = CompletionPerformanceParameters(false, false, false)
private fun parametrizeForUnitTesting() = CompletionPerformanceParameters(true, false, false)
private val isTeamCity = System.getenv("TEAMCITY_VERSION") != null
private enum class Experiment(val number: Int) {
PERFORMANCE_LOOKUP_RANKING(17),
PERFORMANCE_LOOKUP(18),
ONLY_PERFORMANCE(19);
companion object {
fun numbers() = Experiment.values().map { it.number }
fun fromVersion(version: Int) = Experiment.values().find { it.number == version }
}
}
private fun registryParameters(): CompletionPerformanceParameters? {
if (Registry.`is`("ml.completion.performance.experiment")) return null
val enablePerformance = Registry.`is`("ml.completion.performance.enable")
val showLookupEarly = Registry.`is`("ml.completion.performance.showLookupEarly")
val executeImmediately = Registry.`is`("ml.completion.performance.executeImmediately")
return CompletionPerformanceParameters(
enablePerformance,
enablePerformance && showLookupEarly,
executeImmediately,
)
}
private fun experimentParameters(parameters: CompletionParameters): CompletionPerformanceParameters {
if (!CurrentProjectInfo.getInstance(parameters.position.project).isIdeaProject) {
return parametrizeNoPerformance()
}
val status = ExperimentStatus.getInstance().forLanguage(parameters.position.language)
if (!status.inExperiment || status.version !in Experiment.numbers()) {
return parametrizeNoPerformance()
}
return when (Experiment.fromVersion(status.version)) {
Experiment.PERFORMANCE_LOOKUP_RANKING, Experiment.PERFORMANCE_LOOKUP -> CompletionPerformanceParameters(true, true, false)
Experiment.ONLY_PERFORMANCE -> CompletionPerformanceParameters(true, false, false)
else -> parametrizeNoPerformance()
}
}
private fun canEnablePerformance(parameters: CompletionParameters): Boolean {
val hasKindExecutorProvider = { SuggestionGeneratorExecutorProvider.hasAnyToCall(parameters) }
val hasAtLeastOneGenerator = { KindCollector.forParameters(parameters).any() }
return hasKindExecutorProvider() && hasAtLeastOneGenerator()
}
fun fromCompletionPreferences(parameters: CompletionParameters): CompletionPerformanceParameters {
if (isTeamCity || !canEnablePerformance(parameters)) {
return parametrizeNoPerformance()
}
if (parameters.isTestingMode) {
return parametrizeForUnitTesting()
}
return registryParameters() ?: experimentParameters(parameters)
}
}
}

View File

@@ -0,0 +1,27 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.KindExecutionListener
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
interface DelegatingKindExecutionListener<T : KindExecutionListener> : KindExecutionListener {
val delegatedListeners: MutableList<T>
override fun onInitialize(parameters: CompletionParameters) = delegatedListeners.forEach { it.onInitialize(parameters) }
override fun onCollectionStarted() = delegatedListeners.forEach { it.onCollectionStarted() }
override fun onGeneratorCollected(suggestionGenerator: SuggestionGenerator) = delegatedListeners.forEach {
it.onGeneratorCollected(suggestionGenerator)
}
override fun onGenerationStarted(suggestionGenerator: SuggestionGenerator) = delegatedListeners.forEach {
it.onGenerationStarted(suggestionGenerator)
}
override fun onGenerationFinished(suggestionGenerator: SuggestionGenerator) = delegatedListeners.forEach {
it.onGenerationFinished(suggestionGenerator)
}
override fun onCollectionFinished() = delegatedListeners.forEach { it.onCollectionFinished() }
}

View File

@@ -0,0 +1,17 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.addingPolicy.PassDirectlyPolicy
class ImmediateExecutor(override val parameters: CompletionParameters,
override val policyController: PolicyController) : SuggestionGeneratorExecutor {
override fun createNoneKindPolicy() = PassDirectlyPolicy()
override fun executeAll() {}
override fun pass(suggestionGenerator: SuggestionGenerator) {
suggestionGenerator.generateCompletionVariants()
}
}

View File

@@ -0,0 +1,14 @@
package com.intellij.turboComplete
import com.intellij.platform.ml.impl.turboComplete.KindVariety
enum class NullableKindName {
NONE_KIND
}
class NullableKindVariety(private val baseKindVariety: KindVariety) : KindVariety by baseKindVariety {
companion object {
fun KindVariety.withNullableKind() = NullableKindVariety(this)
}
}

View File

@@ -0,0 +1,72 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.platform.ml.impl.turboComplete.KindCollector
import com.intellij.platform.ml.impl.turboComplete.KindExecutionListener
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.SuggestionGeneratorConsumer
import com.intellij.turboComplete.analysis.PipelineListener
private class GenerationReportingSuggestionGenerator(
private val suggestionGenerator: SuggestionGenerator,
private val listener: KindExecutionListener,
) : SuggestionGenerator by suggestionGenerator {
override fun generateCompletionVariants() {
listener.onGenerationStarted(suggestionGenerator)
try {
suggestionGenerator.generateCompletionVariants()
}
finally {
listener.onGenerationFinished(suggestionGenerator)
}
}
}
class ReportingSuggestionGeneratorExecutor(
private val baseExecutor: SuggestionGeneratorExecutor,
private val listener: KindExecutionListener,
) : SuggestionGeneratorExecutor by baseExecutor {
fun reportGenerateCompletionKinds(generator: KindCollector,
parameters: CompletionParameters,
completionKindsConsumer: SuggestionGeneratorConsumer,
result: CompletionResultSet,
resultPolicyController: PolicyController) {
listener.onCollectionStarted()
try {
generator.collectKinds(parameters, completionKindsConsumer, result, resultPolicyController)
}
finally {
listener.onCollectionFinished()
}
}
override fun pass(suggestionGenerator: SuggestionGenerator) {
val reportingCompletionKind = GenerationReportingSuggestionGenerator(suggestionGenerator, listener)
listener.onGeneratorCollected(suggestionGenerator)
baseExecutor.pass(reportingCompletionKind)
}
companion object {
fun initialize(
parameters: CompletionParameters,
policyController: PolicyController,
executorProvider: SuggestionGeneratorExecutorProvider,
): ReportingSuggestionGeneratorExecutor {
val listener = object : DelegatingKindExecutionListener<KindExecutionListener> {
override val delegatedListeners: MutableList<KindExecutionListener> = PipelineListener.EP_NAME.extensionList.toMutableList()
}
listener.onInitialize(parameters)
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(parameters)
val generatorExecutor = if (performanceParameters.fixedGeneratorsOrder)
ImmediateExecutor(parameters, policyController)
else
executorProvider.createExecutor(parameters, policyController)
return ReportingSuggestionGeneratorExecutor(generatorExecutor, listener)
}
}
}

View File

@@ -0,0 +1,84 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.openapi.components.service
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.addingPolicy.PassDirectlyPolicy
import com.intellij.turboComplete.analysis.usage.KindVarietyUsageTracker
import com.intellij.turboComplete.platform.addingPolicy.BufferingPolicy
import com.intellij.turboComplete.platform.addingPolicy.ConvertToCompletionKindPolicy
import com.intellij.turboComplete.platform.addingPolicy.addingActualContributor
import com.intellij.turboComplete.platform.addingPolicy.addingCompletionKind
import com.intellij.turboComplete.ranking.KindRelevanceSorter
class SortingExecutor(
private val sorter: KindRelevanceSorter,
override val parameters: CompletionParameters,
override val policyController: PolicyController,
private val executionPreferences: SortingExecutorPreferences
) : SuggestionGeneratorExecutor {
private val executedGenerators: MutableSet<SuggestionGenerator> = mutableSetOf()
private val nonExecutedGenerators: MutableSet<SuggestionGenerator> = mutableSetOf()
private val mostRelevantKinds: MutableList<CompletionKind> = run {
val usageTracker = service<KindVarietyUsageTracker>()
val onceGeneratedKinds = usageTracker.trackedKinds(sorter.kindVariety)
sorter.sort(onceGeneratedKinds, parameters).map { it.kind }
.take(executionPreferences.executeMostRelevantWhenPassed)
.toMutableList()
}
override fun createNoneKindPolicy() = when (executionPreferences.policyForNoneKind) {
SortingExecutorPreferences.NoneKindPolicy.BUFFER -> BufferingPolicy()
SortingExecutorPreferences.NoneKindPolicy.PASS_TO_RESULT -> PassDirectlyPolicy()
SortingExecutorPreferences.NoneKindPolicy.CREATE_NONE_KIND -> ConvertToCompletionKindPolicy(
this,
CompletionKind(NullableKindName.NONE_KIND, sorter.kindVariety),
parameters
)
}
private fun makeMostRelevantKindPolicy() = when (executionPreferences.policyForMostRelevant) {
SortingExecutorPreferences.MostRelevantKindPolicy.BUFFER -> BufferingPolicy()
SortingExecutorPreferences.MostRelevantKindPolicy.PASS_TO_RESULT -> PassDirectlyPolicy()
}
private fun makeKindPolicy(suggestionGenerator: SuggestionGenerator): ElementsAddingPolicy {
val plainPolicy =
if (executedGenerators.isEmpty()) makeMostRelevantKindPolicy()
else PassDirectlyPolicy()
return plainPolicy
.addingActualContributor(suggestionGenerator)
.addingCompletionKind(suggestionGenerator)
}
private fun invokeCompletionKind(suggestionGenerator: SuggestionGenerator) {
val policy = makeKindPolicy(suggestionGenerator)
policyController.invokeWithPolicy(policy) {
suggestionGenerator.generateCompletionVariants()
}
executedGenerators.add(suggestionGenerator)
nonExecutedGenerators.remove(suggestionGenerator)
}
override fun executeAll() {
if (nonExecutedGenerators.isEmpty()) return
val executionOrder = sorter.sortGenerators(nonExecutedGenerators.toList(), parameters)
executionOrder.forEach {
invokeCompletionKind(it)
}
}
override fun pass(suggestionGenerator: SuggestionGenerator) {
nonExecutedGenerators.add(suggestionGenerator)
if (mostRelevantKinds.getOrNull(0) == suggestionGenerator.kind) {
invokeCompletionKind(suggestionGenerator)
mostRelevantKinds.removeFirst()
}
}
}

View File

@@ -0,0 +1,18 @@
package com.intellij.turboComplete
data class SortingExecutorPreferences(
val policyForMostRelevant: MostRelevantKindPolicy,
val policyForNoneKind: NoneKindPolicy,
val executeMostRelevantWhenPassed: Int,
) {
enum class MostRelevantKindPolicy {
BUFFER,
PASS_TO_RESULT,
}
enum class NoneKindPolicy {
BUFFER,
PASS_TO_RESULT,
CREATE_NONE_KIND,
}
}

View File

@@ -0,0 +1,34 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.turboComplete.analysis.DelegatingPipelineListener
import com.intellij.turboComplete.analysis.PipelineListener
import com.intellij.turboComplete.ranking.KindRelevanceSorter
import com.intellij.turboComplete.ranking.KindSorterProvider
import com.intellij.turboComplete.ranking.ReportingKindSorter
abstract class SortingExecutorProvider(
private val executionPreferences: SortingExecutorPreferences
) : SuggestionGeneratorExecutorProvider {
abstract val sorterProvider: KindSorterProvider
override fun shouldBeCalled(parameters: CompletionParameters): Boolean {
return sorterProvider.kindVariety.kindsCorrespondToParameters(parameters)
}
private fun createSorter(): KindRelevanceSorter {
val rankingListeners: MutableList<PipelineListener> = PipelineListener.EP_NAME.extensionList.toMutableList()
val delegatingListener = object : DelegatingPipelineListener {
override val delegatedListeners: MutableList<PipelineListener>
get() = rankingListeners
}
return ReportingKindSorter(delegatingListener, sorterProvider)
}
override fun createExecutor(
parameters: CompletionParameters,
policyController: PolicyController,
): SuggestionGeneratorExecutor {
return SortingExecutor(createSorter(), parameters, policyController, executionPreferences)
}
}

View File

@@ -0,0 +1,15 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.platform.ml.impl.turboComplete.SuggestionGeneratorConsumer
interface SuggestionGeneratorExecutor : SuggestionGeneratorConsumer {
val parameters: CompletionParameters
val policyController: PolicyController
fun createNoneKindPolicy(): ElementsAddingPolicy
fun executeAll()
}

View File

@@ -0,0 +1,32 @@
package com.intellij.turboComplete
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.openapi.extensions.ExtensionPointName
interface SuggestionGeneratorExecutorProvider {
fun shouldBeCalled(parameters: CompletionParameters): Boolean
fun createExecutor(
parameters: CompletionParameters,
policyController: PolicyController,
): SuggestionGeneratorExecutor
companion object {
private val EP_NAME: ExtensionPointName<SuggestionGeneratorExecutorProvider> =
ExtensionPointName("com.intellij.turboComplete.suggestionGeneratorExecutorProvider")
fun hasAnyToCall(parameters: CompletionParameters): Boolean {
return EP_NAME.extensionList.any { it.shouldBeCalled(parameters) }
}
fun findOneMatching(parameters: CompletionParameters): SuggestionGeneratorExecutorProvider {
val allExecutorProviders = EP_NAME.extensionList.filter { it.shouldBeCalled(parameters) }
if (allExecutorProviders.size > 1) {
throw IllegalStateException(
"Found more than one matching CompletionKindExecutorProvider: ${allExecutorProviders.map { it.javaClass.name }}")
}
return allExecutorProviders[0]
}
}
}

View File

@@ -0,0 +1,9 @@
package com.intellij.turboComplete.analysis
import com.intellij.turboComplete.DelegatingKindExecutionListener
import com.intellij.turboComplete.ranking.DelegatingKindRankingListener
interface DelegatingPipelineListener
: PipelineListener,
DelegatingKindRankingListener<PipelineListener>,
DelegatingKindExecutionListener<PipelineListener>

View File

@@ -0,0 +1,18 @@
package com.intellij.turboComplete.analysis
import com.intellij.completion.ml.performance.PerformanceTracker
class KindPerformanceRecorder : PerformanceRecorder<SingleLifetimeKindsRecorder>() {
override val performanceMetricName = "turboComplete.kindPerf"
override fun createPipelineListener() = SingleLifetimeKindsRecorder()
override fun capturePerformance(pipelineListener: SingleLifetimeKindsRecorder, tracker: PerformanceTracker) {
val completionKindLifetimes = pipelineListener.captureCompletionKindLifetimeDurations()
completionKindLifetimes.entries.forEach { (completionKind, performance) ->
tracker.addByKey("init.${completionKind.name}", performance.created)
performance.executed?.let { tracker.addByKey("run.${completionKind.name}", it) }
performance.executedAsFirst?.let { tracker.addByKey("runFirst.${completionKind.name}", it) }
}
}
}

View File

@@ -0,0 +1,41 @@
package com.intellij.turboComplete.analysis
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.codeInsight.lookup.impl.LookupManagerImpl
import com.intellij.completion.ml.performance.PerformanceMetricCollector
import com.intellij.completion.ml.performance.PerformanceMetricCollectorFactory
import com.intellij.completion.ml.performance.PerformanceTracker
import com.intellij.completion.ml.storage.MutableLookupStorage
abstract class PerformanceRecorder<T : PipelineListener>
: DelegatingPipelineListener,
PerformanceMetricCollectorFactory {
private lateinit var currentListener: T
abstract fun createPipelineListener(): T
abstract fun capturePerformance(pipelineListener: T, tracker: PerformanceTracker)
override val delegatedListeners: MutableList<PipelineListener>
get() = mutableListOf(currentListener)
override fun onInitialize(parameters: CompletionParameters) {
val lookup = LookupManagerImpl.getActiveLookup(parameters.editor) as? LookupImpl ?: return
val lookupStorage = MutableLookupStorage.get(lookup)!!
lookupStorage.performanceTracker.addMetricCollector(this)
currentListener = createPipelineListener()
super.onInitialize(parameters)
}
override fun createMetricCollector(tracker: PerformanceTracker): PerformanceMetricCollector {
return CompletionKindMetricsCollector(tracker)
}
inner class CompletionKindMetricsCollector(private val tracker: PerformanceTracker) : PerformanceMetricCollector {
override fun onFinishCollecting() {
capturePerformance(currentListener, tracker)
}
}
}

View File

@@ -0,0 +1,13 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.turboComplete.analysis
import com.intellij.openapi.extensions.ExtensionPointName
import com.intellij.platform.ml.impl.turboComplete.KindExecutionListener
import com.intellij.platform.ml.impl.turboComplete.ranking.KindRankingListener
interface PipelineListener : KindExecutionListener, KindRankingListener {
companion object {
val EP_NAME: ExtensionPointName<PipelineListener> = ExtensionPointName.create(
"com.intellij.turboComplete.analysis.pipelineListener")
}
}

View File

@@ -0,0 +1,15 @@
package com.intellij.turboComplete.analysis
import com.intellij.completion.ml.performance.PerformanceTracker
class PipelinePerformanceRecorder : PerformanceRecorder<SinglePipelineRecorder>() {
override val performanceMetricName = "turboComplete.pipeline"
override fun createPipelineListener() = SinglePipelineRecorder()
override fun capturePerformance(pipelineListener: SinglePipelineRecorder, tracker: PerformanceTracker) {
val chronologyDurations = pipelineListener.captureChronologyDurations()
tracker.addByKey("gen", chronologyDurations.generation)
tracker.addByKey("rank", chronologyDurations.ranking)
}
}

View File

@@ -0,0 +1,64 @@
package com.intellij.turboComplete.analysis
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
import kotlin.properties.Delegates
class SingleLifetimeKindsRecorder : PipelineListener {
private var collectionStarted by Delegates.notNull<Long>()
private var generatorCollected: MutableMap<CompletionKind, Long?> = mutableMapOf()
private var kindRanked: MutableMap<CompletionKind, Long?> = mutableMapOf()
private var generationStarted: MutableMap<CompletionKind, Long?> = mutableMapOf()
private var generationFinished: MutableMap<CompletionKind, Long?> = mutableMapOf()
private var firstExecutedKind: CompletionKind? = null
override fun onInitialize(parameters: CompletionParameters) {}
override fun onCollectionStarted() {
collectionStarted = System.currentTimeMillis()
}
override fun onGeneratorCollected(suggestionGenerator: SuggestionGenerator) {
generatorCollected[suggestionGenerator.kind] = System.currentTimeMillis()
}
override fun onRanked(ranked: List<RankedKind>) {
val time = System.currentTimeMillis()
ranked.forEach { kindRanked[it.kind] = time }
}
override fun onGenerationStarted(suggestionGenerator: SuggestionGenerator) {
generationStarted[suggestionGenerator.kind] = System.currentTimeMillis()
if (firstExecutedKind == null) {
firstExecutedKind = suggestionGenerator.kind
}
}
override fun onGenerationFinished(suggestionGenerator: SuggestionGenerator) {
generationFinished[suggestionGenerator.kind] = System.currentTimeMillis()
}
fun captureCompletionKindLifetimeDurations(): Map<CompletionKind, GeneratorLifetimeDurations> {
fun durationBetween(start: Long?, finish: Long?): Long? {
return finish?.let { notNullFinish -> start?.let { notNullStart -> notNullFinish - notNullStart } }
}
return generatorCollected.keys.associateWith {
GeneratorLifetimeDurations(
created = durationBetween(collectionStarted, generatorCollected[it])!!,
executed = durationBetween(generationStarted[it], generationFinished[it]),
executedAsFirst = if (it == firstExecutedKind)
durationBetween(generationStarted[it], generationFinished[it])
else
null
)
}
}
}
data class GeneratorLifetimeDurations(
val created: Long,
val executed: Long?,
val executedAsFirst: Long?
)

View File

@@ -0,0 +1,60 @@
package com.intellij.turboComplete.analysis
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import kotlin.properties.Delegates
class SinglePipelineRecorder : PipelineListener {
private var collectionStarted by Delegates.notNull<Long>()
private var collectionFinished by Delegates.notNull<Long>()
private val generatorCollected: MutableMap<CompletionKind, Long> = mutableMapOf()
private val generatorStarted: MutableMap<CompletionKind, Long> = mutableMapOf()
private val generatorFinished: MutableMap<CompletionKind, Long> = mutableMapOf()
private var rankingDurations: MutableList<Long> = mutableListOf()
private var rankingStarted: Long? = null
override fun onInitialize(parameters: CompletionParameters) {
}
override fun onCollectionStarted() {
collectionStarted = System.currentTimeMillis()
}
override fun onGeneratorCollected(suggestionGenerator: SuggestionGenerator) {
generatorCollected[suggestionGenerator.kind] = System.currentTimeMillis()
}
override fun onCollectionFinished() {
collectionFinished = System.currentTimeMillis()
}
override fun onGenerationStarted(suggestionGenerator: SuggestionGenerator) {
generatorStarted[suggestionGenerator.kind] = System.currentTimeMillis()
}
override fun onGenerationFinished(suggestionGenerator: SuggestionGenerator) {
generatorFinished[suggestionGenerator.kind] = System.currentTimeMillis()
}
fun captureChronologyDurations(): PipelineChronologyDurations {
return PipelineChronologyDurations(
generation = collectionFinished - collectionStarted,
ranking = rankingDurations.sum(),
)
}
override fun onRankingStarted() {
rankingStarted = System.currentTimeMillis()
}
override fun onRankingFinished() {
rankingDurations.add(System.currentTimeMillis() - rankingStarted!!)
rankingStarted = null
}
data class PipelineChronologyDurations(
val generation: Long,
val ranking: Long,
)
}

View File

@@ -0,0 +1,52 @@
package com.intellij.turboComplete.analysis.usage
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
data class CombinedKindUsageStatistics(
val created: Int,
val generated: KindStatistics,
val generatedInRow: KindStatistics,
val recentGenerated: ValuePerPeriod<KindStatistics>,
) {
val recentProbGenerateCorrect: ValuePerPeriod<Double>
get() {
return if (recentGenerated.period != 0)
ValuePerPeriod(
recentGenerated.period,
recentGenerated.value.correct.toDouble() / recentGenerated.period
)
else {
ValuePerPeriod(0, 0.0)
}
}
}
class CombinedKindUsageTracker(override val kind: CompletionKind,
recentWindowSize: Int,
maxUsageWindowSize: Int) : KindUsageTracker<CombinedKindUsageStatistics> {
private val createdTracker = KindCreatedTracker(kind)
private val generatedTracker = KindGenerationTracker(kind)
private val generationInRawTracker = KindGenerationInRawTracker(kind, maxUsageWindowSize)
private val recentGeneratedTracker = KindRecentUsageTracker(kind, recentWindowSize) {
KindGenerationTracker(it)
}
private val allTrackers = listOf(
createdTracker,
generatedTracker,
generationInRawTracker,
recentGeneratedTracker,
)
override fun trackCreated() = allTrackers.forEach { it.trackCreated() }
override fun trackGenerated(correct: Boolean) = allTrackers.forEach { it.trackGenerated(correct) }
override fun getSummary(): CombinedKindUsageStatistics {
return CombinedKindUsageStatistics(
created = createdTracker.getSummary(),
generated = generatedTracker.getSummary(),
generatedInRow = generationInRawTracker.getSummary(),
recentGenerated = recentGeneratedTracker.getSummary(),
)
}
}

View File

@@ -0,0 +1,64 @@
package com.intellij.turboComplete.analysis.usage
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import java.util.*
data class ValuePerPeriod<T>(
val period: Int,
val value: T,
)
private data class RecentUsage<T>(
val isCreation: Boolean,
val action: (KindUsageTracker<T>) -> Unit,
)
class KindRecentUsageTracker<T>(override val kind: CompletionKind,
private val windowSize: Int,
val baseTrackerProvider: (CompletionKind) -> KindUsageTracker<T>
) : KindUsageTracker<ValuePerPeriod<T>> {
private val recentUsages: Queue<RecentUsage<T>> = LinkedList()
private var recentCreations: Int = 0
private var recentUsagesStatistics: T? = null
private fun popCreationAndFollowingActions() {
assert(recentUsages.peek().isCreation)
assert(recentUsages.remove().isCreation)
recentCreations -= 1
while (recentUsages.isNotEmpty() && !recentUsages.peek().isCreation) {
recentUsages.remove()
}
}
private fun trackRecentUsage(action: RecentUsage<T>) {
recentUsagesStatistics = null
recentUsages.add(action)
if (action.isCreation) {
recentCreations += 1
}
while (recentCreations > windowSize) {
popCreationAndFollowingActions()
}
}
override fun trackCreated() {
trackRecentUsage(RecentUsage(true) { it.trackCreated() })
}
override fun trackGenerated(correct: Boolean) {
trackRecentUsage(RecentUsage(false) { it.trackGenerated(correct) })
}
private fun computeSummary(): T {
val recentUsagesTracker = baseTrackerProvider(kind)
recentUsages.forEach { it.action(recentUsagesTracker) }
return recentUsagesTracker.getSummary()
}
override fun getSummary(): ValuePerPeriod<T> {
if (recentUsagesStatistics == null) {
recentUsagesStatistics = computeSummary()
}
return ValuePerPeriod(recentCreations, recentUsagesStatistics!!)
}
}

View File

@@ -0,0 +1,65 @@
package com.intellij.turboComplete.analysis.usage
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import kotlin.math.min
interface KindUsageTracker<T> {
val kind: CompletionKind
fun trackCreated() {}
fun trackGenerated(correct: Boolean) {}
fun getSummary(): T
}
class KindCreatedTracker(override val kind: CompletionKind) : KindUsageTracker<Int> {
private var created = 0
override fun trackCreated() {
created += 1
}
override fun getSummary() = created
}
data class KindStatistics(
val correct: Int,
val notCorrect: Int
)
class KindGenerationTracker(override val kind: CompletionKind) : KindUsageTracker<KindStatistics> {
private var generatedCorrect = 0
private var generatedNotCorrect = 0
override fun trackGenerated(correct: Boolean) {
if (correct)
generatedCorrect += 1
else
generatedNotCorrect += 1
}
override fun getSummary() = KindStatistics(generatedCorrect, generatedNotCorrect)
}
class KindGenerationInRawTracker(override val kind: CompletionKind, private val maxPeriod: Int?) : KindUsageTracker<KindStatistics> {
private var generatedCorrectInRow = 0
private var generatedNotCorrectInRow = 0
override fun trackGenerated(correct: Boolean) {
if (correct) {
generatedCorrectInRow += 1
generatedNotCorrectInRow = 0
}
else {
generatedCorrectInRow = 0
generatedNotCorrectInRow += 1
}
}
override fun getSummary(): KindStatistics {
return maxPeriod?.let {
KindStatistics(min(it, generatedCorrectInRow), min(it, generatedNotCorrectInRow))
} ?: KindStatistics(generatedCorrectInRow, generatedNotCorrectInRow)
}
}

View File

@@ -0,0 +1,63 @@
package com.intellij.turboComplete.analysis.usage
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.lookup.LookupEvent
import com.intellij.codeInsight.lookup.LookupListener
import com.intellij.codeInsight.lookup.LookupManager
import com.intellij.openapi.components.Service
import com.intellij.openapi.components.service
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator.Companion.suggestionGenerator
import com.intellij.turboComplete.analysis.PipelineListener
@Service
class KindVarietyUsageTracker {
private val kindStatisticsPerVariety: MutableMap<CompletionKind, CombinedKindUsageTracker> = mutableMapOf()
private val recentPeriodLength = 30
fun kindStatistics(kind: CompletionKind): CombinedKindUsageStatistics {
return kindStatisticsPerVariety
.getOrDefault(kind, CombinedKindUsageTracker(kind, recentPeriodLength, recentPeriodLength))
.getSummary()
}
fun trackedKinds(variety: KindVariety): List<CompletionKind> {
return kindStatisticsPerVariety.keys.filter { it.variety == variety }
}
fun trackedKinds(parameters: CompletionParameters): List<CompletionKind> {
return kindStatisticsPerVariety.keys.filter { it.variety.kindsCorrespondToParameters(parameters) }
}
private fun kindStatisticsTracker(kind: CompletionKind): CombinedKindUsageTracker {
return kindStatisticsPerVariety.getOrPut(kind) {
CombinedKindUsageTracker(kind, recentPeriodLength, recentPeriodLength)
}
}
class UsagePipelineListener : PipelineListener {
override fun onInitialize(parameters: CompletionParameters) {
LookupManager.getActiveLookup(parameters.editor)?.addLookupListener(UsageLookupListener())
}
override fun onGeneratorCollected(suggestionGenerator: SuggestionGenerator) {
service<KindVarietyUsageTracker>()
.kindStatisticsTracker(suggestionGenerator.kind)
.trackCreated()
}
}
class UsageLookupListener : LookupListener {
override fun itemSelected(event: LookupEvent) {
val tracker = service<KindVarietyUsageTracker>()
val correctSuggestionGenerator = event.item?.suggestionGenerator ?: return
tracker.kindStatisticsPerVariety.forEach { (kind, kindStatisticsTracker) ->
kindStatisticsTracker.trackGenerated(correct = kind == correctSuggestionGenerator.kind)
}
}
}
}

View File

@@ -0,0 +1,32 @@
package com.intellij.turboComplete.features.context
import com.intellij.codeInsight.completion.ml.CompletionEnvironment
import com.intellij.codeInsight.completion.ml.ContextFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.openapi.components.service
import com.intellij.turboComplete.analysis.usage.KindVarietyUsageTracker
class AllKindsUsageFeatures : ContextFeatureProvider {
override fun getName() = "kind_usage"
override fun calculateFeatures(environment: CompletionEnvironment): Map<String, MLFeatureValue> {
val usageTracker = service<KindVarietyUsageTracker>()
val features = mutableMapOf<String, MLFeatureValue>()
val trackedKinds = usageTracker.trackedKinds(environment.parameters)
features.putAll(
trackedKinds
.map { kind -> kind to usageTracker.kindStatistics(kind) }
.filter { it.second.generated.correct > 0 }
.flatMap { (kind, usage) ->
listOf(
"correct_prob_${kind.name}" to MLFeatureValue.float(usage.recentProbGenerateCorrect.value),
"correct_in_row_${kind.name}" to MLFeatureValue.numerical(usage.generatedInRow.correct),
"incorrect_in_row_${kind.name}" to MLFeatureValue.numerical(usage.generatedInRow.notCorrect),
)
}
)
return features
}
}

View File

@@ -0,0 +1,19 @@
package com.intellij.turboComplete.features.context
import com.intellij.codeInsight.completion.ml.CompletionEnvironment
import com.intellij.codeInsight.completion.ml.ContextFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.turboComplete.CompletionPerformanceParameters
class CompletionPerformanceStatusFeatures : ContextFeatureProvider {
override fun getName() = "common_completion_kind"
override fun calculateFeatures(environment: CompletionEnvironment): Map<String, MLFeatureValue> {
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(environment.parameters)
return mutableMapOf(
"performance_enabled" to MLFeatureValue.binary(performanceParameters.enabled),
"show_lookup_early" to MLFeatureValue.binary(performanceParameters.showLookupEarly),
"fixed_generators_order" to MLFeatureValue.binary(performanceParameters.fixedGeneratorsOrder)
)
}
}

View File

@@ -0,0 +1,20 @@
package com.intellij.turboComplete.features.element
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.ElementFeatureProvider
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator.Companion.suggestionGenerator
import com.intellij.turboComplete.features.kind.FeaturesComputer
class ElementsKindFeatures : ElementFeatureProvider {
override fun getName() = "completion_kind"
override fun calculateFeatures(element: LookupElement,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
val suggestionGenerator = element.suggestionGenerator ?: return mutableMapOf()
return FeaturesComputer.getKindFeatures(suggestionGenerator.kind, location, contextFeatures).toMutableMap()
}
}

View File

@@ -0,0 +1,41 @@
package com.intellij.turboComplete.features.kind
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.openapi.util.Key
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
class CompletionKindFeaturesManager {
private val computedFeatures: MutableMap<CompletionKind, Map<String, MLFeatureValue>> = mutableMapOf()
private fun computeKindFeatures(kind: CompletionKind,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
return KindFeatureProvider.EP_NAME.extensionList.flatMap { provider ->
provider.calculateFeatures(kind, location, contextFeatures).entries.map { (featureName, featureValue) ->
"${provider.getName()}_$featureName" to featureValue
}
}.toMap()
}
fun getOrCompute(kind: CompletionKind,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
return computedFeatures.getOrPut(kind) {
computeKindFeatures(kind, location, contextFeatures)
}
}
companion object {
private val LOOKUP_COMPLETION_KIND_FEATURES_MANAGER = Key<CompletionKindFeaturesManager>("Manager of CompletionKinds' features")
val LookupImpl.completionKindFeaturesManager: CompletionKindFeaturesManager
get() = this.getUserData(LOOKUP_COMPLETION_KIND_FEATURES_MANAGER) ?: run {
val manager = CompletionKindFeaturesManager()
this.putUserData(LOOKUP_COMPLETION_KIND_FEATURES_MANAGER, manager)
manager
}
}
}

View File

@@ -0,0 +1,17 @@
package com.intellij.turboComplete.features.kind
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.codeInsight.lookup.LookupManager
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.turboComplete.features.kind.CompletionKindFeaturesManager.Companion.completionKindFeaturesManager
object FeaturesComputer {
fun getKindFeatures(kind: CompletionKind, location: CompletionLocation, contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
val lookup = LookupManager.getActiveLookup(location.completionParameters.editor) as? LookupImpl ?: return emptyMap()
val featuresManager = lookup.completionKindFeaturesManager
return featuresManager.getOrCompute(kind, location, contextFeatures)
}
}

View File

@@ -0,0 +1,18 @@
package com.intellij.turboComplete.features.kind
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
class KindCommonFeatures : KindFeatureProvider {
override fun getName() = "common"
override fun calculateFeatures(kind: CompletionKind,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
return mutableMapOf(
"is_name" to MLFeatureValue.categorical(kind.name)
)
}
}

View File

@@ -0,0 +1,20 @@
package com.intellij.turboComplete.features.kind
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.openapi.extensions.ExtensionPointName
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
interface KindFeatureProvider {
fun getName(): String
fun calculateFeatures(kind: CompletionKind,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue>
companion object {
val EP_NAME: ExtensionPointName<KindFeatureProvider> = ExtensionPointName.create(
"com.intellij.turboComplete.features.kind.provider")
}
}

View File

@@ -0,0 +1,24 @@
package com.intellij.turboComplete.features.kind
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.ml.ContextFeatures
import com.intellij.codeInsight.completion.ml.MLFeatureValue
import com.intellij.openapi.components.service
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.turboComplete.analysis.usage.KindVarietyUsageTracker
class KindUsageFeatures : KindFeatureProvider {
override fun getName() = "usage"
override fun calculateFeatures(kind: CompletionKind,
location: CompletionLocation,
contextFeatures: ContextFeatures): Map<String, MLFeatureValue> {
val usageTracker = service<KindVarietyUsageTracker>()
val usage = usageTracker.kindStatistics(kind)
return mutableMapOf(
"correct_prob" to MLFeatureValue.float(usage.recentProbGenerateCorrect.value),
"correct_in_row" to MLFeatureValue.numerical(usage.generatedInRow.correct),
"incorrect_in_row" to MLFeatureValue.numerical(usage.generatedInRow.notCorrect),
)
}
}

View File

@@ -0,0 +1,35 @@
package com.intellij.turboComplete.platform
import com.intellij.codeInsight.completion.CompletionLookupOpener
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
import com.intellij.turboComplete.CompletionPerformanceParameters
import com.intellij.turboComplete.analysis.PipelineListener
class EarlyLookupOpener : PipelineListener {
private val order: MutableList<CompletionKind> = mutableListOf()
override fun onInitialize(parameters: CompletionParameters) {
}
override fun onRanked(ranked: List<RankedKind>) {
order.addAll(ranked.map { it.kind })
}
override fun onGenerationFinished(suggestionGenerator: SuggestionGenerator) {
val rankIndex = order.indexOf(suggestionGenerator.kind)
if (rankIndex != 0) return
showLookup(suggestionGenerator.parameters)
order.clear()
}
companion object {
fun showLookup(parameters: CompletionParameters) {
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(parameters)
if (!performanceParameters.showLookupEarly) return
CompletionLookupOpener.showLookup(parameters)
}
}
}

View File

@@ -0,0 +1,20 @@
package com.intellij.turboComplete.platform.addingPolicy
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.completion.ml.common.CommonElementLocationFeatures
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
class ActualContributorPuttingPolicy(
base: ElementsAddingPolicy,
private val actualCompletionContributorClass: Class<*>,
) : ElementDecoratingPolicy(base) {
override fun decorate(element: LookupElement) {
element.putUserData(CommonElementLocationFeatures.LOOKUP_ORIGINAL_ELEMENT_CONTRIBUTOR_TYPE,
actualCompletionContributorClass)
}
}
fun ElementsAddingPolicy.addingActualContributor(suggestionGenerator: SuggestionGenerator): ActualContributorPuttingPolicy {
return ActualContributorPuttingPolicy(this, suggestionGenerator.kind.variety.actualCompletionContributorClass)
}

View File

@@ -0,0 +1,30 @@
package com.intellij.turboComplete.platform.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
class BufferingPolicy : ElementsAddingPolicy {
private val buffer: MutableList<LookupElement> = mutableListOf()
override fun onResultStop(result: CompletionResultSet) {
flushBuffer(result)
}
override fun addElement(result: CompletionResultSet, element: LookupElement) {
buffer.add(element)
}
override fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>) {
buffer.addAll(elements)
}
override fun onDeactivate(result: CompletionResultSet) {
flushBuffer(result)
}
private fun flushBuffer(result: CompletionResultSet) {
result.addAllElements(buffer)
buffer.clear()
}
}

View File

@@ -0,0 +1,48 @@
package com.intellij.turboComplete.platform.addingPolicy
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.SuggestionGeneratorConsumer
class ConvertToCompletionKindPolicy(
private val suggestionGeneratorConsumer: SuggestionGeneratorConsumer,
private val convertedKind: CompletionKind,
private val parameters: CompletionParameters
) : ElementsAddingPolicy {
private val buffer = mutableListOf<LookupElement>()
override fun onResultStop(result: CompletionResultSet) {
createCompletionKindFromBuffer(result)
}
override fun addElement(result: CompletionResultSet, element: LookupElement) {
buffer.add(element)
}
override fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>) {
buffer.addAll(elements)
}
override fun onDeactivate(result: CompletionResultSet) {
createCompletionKindFromBuffer(result)
}
private fun createCompletionKindFromBuffer(result: CompletionResultSet) {
if (buffer.isEmpty()) return
val bufferCopy = mutableListOf<LookupElement>()
bufferCopy.addAll(buffer)
buffer.clear()
suggestionGeneratorConsumer.pass(SuggestionGenerator.fromGenerator(
convertedKind,
parameters,
result
) {
result.addAllElements(bufferCopy)
})
}
}

View File

@@ -0,0 +1,25 @@
package com.intellij.turboComplete.platform.addingPolicy
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
abstract class ElementDecoratingPolicy(protected val base: ElementsAddingPolicy) : ElementsAddingPolicy {
override fun addElement(result: CompletionResultSet, element: LookupElement) {
decorate(element)
base.addElement(result, element)
}
override fun addAllElements(result: CompletionResultSet, elements: MutableIterable<LookupElement>) {
elements.forEach { decorate(it) }
base.addAllElements(result, elements)
}
override fun onActivate(result: CompletionResultSet) = base.onActivate(result)
override fun onDeactivate(result: CompletionResultSet) = base.onDeactivate(result)
abstract fun decorate(element: LookupElement)
override fun onResultStop(result: CompletionResultSet) = base.onResultStop(result)
}

View File

@@ -0,0 +1,19 @@
// Copyright 2000-2023 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
package com.intellij.turboComplete.platform.addingPolicy
import com.intellij.codeInsight.completion.addingPolicy.ElementsAddingPolicy
import com.intellij.codeInsight.lookup.LookupElement
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
class SuggestionGeneratorPuttingPolicy(
base: ElementsAddingPolicy,
private val suggestionGenerator: SuggestionGenerator,
) : ElementDecoratingPolicy(base) {
override fun decorate(element: LookupElement) {
element.putUserData(SuggestionGenerator.LOOKUP_ELEMENT_SUGGESTION_GENERATOR, suggestionGenerator)
}
}
fun ElementsAddingPolicy.addingCompletionKind(suggestionGenerator: SuggestionGenerator): ElementsAddingPolicy {
return SuggestionGeneratorPuttingPolicy(this, suggestionGenerator)
}

View File

@@ -0,0 +1,25 @@
package com.intellij.turboComplete.platform.contributor
import com.intellij.codeInsight.completion.*
import com.intellij.platform.ml.impl.turboComplete.KindCollector
import com.intellij.turboComplete.CompletionPerformanceParameters
class FilteringCompletionContributor : CompletionContributor() {
private val filter: (CompletionContributor) -> Boolean = RejectedListFilter(KindCollector.EP_NAME.extensionList.map { it.kindVariety.actualCompletionContributorClass })
override fun fillCompletionVariants(parameters: CompletionParameters, result: CompletionResultSet) {
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(parameters)
if (!performanceParameters.enabled) return
// TODO: Verify if that works
FilteringResultSet(result, filter).runRemainingContributors(parameters,true)
result.stopHere()
}
}
private class RejectedListFilter(private val rejectedTypes: List<Class<*>>) : (CompletionContributor) -> Boolean {
override fun invoke(contributor: CompletionContributor): Boolean {
return contributor.javaClass !in rejectedTypes
}
}

View File

@@ -0,0 +1,41 @@
package com.intellij.turboComplete.platform.contributor
import com.intellij.codeInsight.completion.CompletionContributor
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.completion.PolicyObeyingResultSet
import com.intellij.codeInsight.completion.addingPolicy.PolicyController
import com.intellij.openapi.util.ThrowableComputable
import com.intellij.platform.ml.impl.turboComplete.KindCollector
import com.intellij.turboComplete.CompletionPerformanceParameters
import com.intellij.turboComplete.ReportingSuggestionGeneratorExecutor
import com.intellij.turboComplete.SuggestionGeneratorExecutorProvider
import com.intellij.util.indexing.DumbModeAccessType
class KindExecutingCompletionContributor : CompletionContributor() {
override fun fillCompletionVariants(parameters: CompletionParameters, result: CompletionResultSet) {
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(parameters)
if (!performanceParameters.enabled) return
val executorProvider = SuggestionGeneratorExecutorProvider.findOneMatching(parameters)
val policyController = PolicyController(result)
val obeyingResult = PolicyObeyingResultSet(result, policyController)
val executor = ReportingSuggestionGeneratorExecutor.initialize(
parameters, policyController, executorProvider
)
for (generator in KindCollector.forParameters(parameters)) {
val policyWhileGenerating = executor.createNoneKindPolicy()
policyController.invokeWithPolicy(policyWhileGenerating) {
executor.reportGenerateCompletionKinds(generator, parameters, executor, obeyingResult, policyController)
}
if (obeyingResult.isStopped) break
}
DumbModeAccessType.RELIABLE_DATA_ONLY.ignoreDumbMode(ThrowableComputable {
executor.executeAll()
})
}
}

View File

@@ -0,0 +1,30 @@
package com.intellij.turboComplete.platform.contributor
import com.intellij.codeInsight.completion.CompletionContributor
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.completion.CompletionResultSet
import com.intellij.codeInsight.lookup.LookupManager
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.completion.ml.sorting.ContextFactorCalculator
import com.intellij.completion.ml.storage.MutableLookupStorage
import com.intellij.openapi.project.DumbAware
import com.intellij.turboComplete.CompletionPerformanceParameters
class KindOrderingFeaturesContributor : CompletionContributor(), DumbAware {
override fun fillCompletionVariants(parameters: CompletionParameters, result: CompletionResultSet) {
val performanceParameters = CompletionPerformanceParameters.fromCompletionPreferences(parameters)
if (!performanceParameters.enabled) return
val lookup = LookupManager.getActiveLookup(parameters.editor) as? LookupImpl
if (lookup != null) {
val storage = MutableLookupStorage.get(lookup)
if (storage != null) {
MutableLookupStorage.saveAsUserData(parameters, storage)
if (!storage.isContextFactorsInitialized()) {
ContextFactorCalculator.calculateContextFactors(lookup, parameters, storage)
}
}
}
super.fillCompletionVariants(parameters, result)
}
}

View File

@@ -0,0 +1,23 @@
package com.intellij.turboComplete.ranking
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import com.intellij.platform.ml.impl.turboComplete.SuggestionGenerator
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
interface KindRelevanceSorter {
val kindVariety: KindVariety
fun sort(kinds: List<CompletionKind>, parameters: CompletionParameters): List<RankedKind>
fun sortGenerators(suggestionGenerators: List<SuggestionGenerator>, parameters: CompletionParameters): List<SuggestionGenerator> {
val completionKindNames = suggestionGenerators.map { it.kind }
val order = sort(completionKindNames, parameters)
.map { ranked ->
val actualCompletionKind = suggestionGenerators.find { it.kind == ranked.kind }!!
actualCompletionKind
}
return order
}
}

View File

@@ -0,0 +1,20 @@
package com.intellij.turboComplete.ranking
import com.intellij.platform.ml.impl.turboComplete.ranking.KindRankingListener
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
interface DelegatingKindRankingListener<T : KindRankingListener> : KindRankingListener {
val delegatedListeners: MutableList<T>
override fun onRankingStarted() {
delegatedListeners.forEach { it.onRankingStarted() }
}
override fun onRanked(ranked: List<RankedKind>) {
delegatedListeners.forEach { it.onRanked(ranked) }
}
override fun onRankingFinished() {
delegatedListeners.forEach { it.onRankingFinished() }
}
}

View File

@@ -0,0 +1,9 @@
package com.intellij.turboComplete.ranking
import com.intellij.platform.ml.impl.turboComplete.KindVariety
interface KindSorterProvider {
val kindVariety: KindVariety
fun createSorter(): KindRelevanceSorter
}

View File

@@ -0,0 +1,22 @@
package com.intellij.turboComplete.ranking
import com.intellij.completion.ml.ranker.local.MLCompletionLocalModelsLoader
import com.intellij.internal.ml.DecisionFunction
import com.intellij.platform.ml.impl.turboComplete.KindVariety
class LocalOrDefaultSorterProvider(
private val languageId: String,
modelRegistryKey: String,
private val defaultDecisionFunctionProvider: () -> DecisionFunction,
override val kindVariety: KindVariety,
) : KindSorterProvider {
private val localModelLoader = MLCompletionLocalModelsLoader(modelRegistryKey)
override fun createSorter(): KindRelevanceSorter {
return MLKindSorter(loadLocalIfAny(languageId, defaultDecisionFunctionProvider), kindVariety)
}
private fun loadLocalIfAny(languageId: String, default: () -> DecisionFunction): DecisionFunction {
return localModelLoader.getModel(languageId) ?: default()
}
}

View File

@@ -0,0 +1,56 @@
package com.intellij.turboComplete.ranking
import com.intellij.codeInsight.completion.CompletionLocation
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.codeInsight.lookup.LookupManager
import com.intellij.codeInsight.lookup.impl.LookupImpl
import com.intellij.completion.ml.features.ContextFeaturesStorage
import com.intellij.completion.ml.sorting.LanguageRankingModel
import com.intellij.completion.ml.sorting.RankingFeatures
import com.intellij.completion.ml.storage.MutableLookupStorage
import com.intellij.internal.ml.DecisionFunction
import com.intellij.internal.ml.completion.DecoratingItemsPolicy
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
import com.intellij.turboComplete.features.kind.FeaturesComputer
class MLKindSorter(decisionFunction: DecisionFunction, override val kindVariety: KindVariety) : KindRelevanceSorter {
private val model = LanguageRankingModel(decisionFunction, DecoratingItemsPolicy.DISABLED)
override fun sort(kinds: List<CompletionKind>,
parameters: CompletionParameters): List<RankedKind> {
val lookup = LookupManager.getActiveLookup(parameters.editor) as? LookupImpl
requireNotNull(lookup) { "Unable to perform Completion Kind ordering when Lookup is not LookupImpl" }
val lookupStorage = MutableLookupStorage.get(lookup)
requireNotNull(lookupStorage)
require(lookupStorage.isContextFactorsInitialized())
val kindsWeights = kinds.map { it to predict(it, lookupStorage, parameters) }
return RankedKind.fromWeights(
kindsWeights,
true
)
}
private fun predict(kind: CompletionKind, lookupStorage: MutableLookupStorage, parameters: CompletionParameters): Double {
val rankingFeatures = RankingFeatures(
lookupStorage.userFactors,
lookupStorage.contextFactors,
emptyMap(), emptyMap(), emptySet()
)
val kindFeatures = FeaturesComputer.getKindFeatures(kind, CompletionLocation(parameters), lookupStorage.contextProvidersResult())
return model.score(rankingFeatures.withElementFeatures(
ContextFeaturesStorage(kindFeatures).asMap(),
emptyMap(),
))
}
}

View File

@@ -0,0 +1,17 @@
package com.intellij.turboComplete.ranking
import com.intellij.internal.ml.DecisionFunction
import com.intellij.platform.ml.impl.turboComplete.KindVariety
class MLKindSorterProvider(
override val kindVariety: KindVariety,
val decisionFunctionProvider: () -> DecisionFunction,
) : KindSorterProvider {
override fun createSorter(): KindRelevanceSorter {
return MLKindSorter(decisionFunctionProvider(), kindVariety)
}
}
fun MLKindSorterProvider.provideLocalIfAny(languageId: String, registryKey: String): KindSorterProvider {
return LocalOrDefaultSorterProvider(languageId, registryKey, this.decisionFunctionProvider, this.kindVariety)
}

View File

@@ -0,0 +1,28 @@
package com.intellij.turboComplete.ranking
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import com.intellij.platform.ml.impl.turboComplete.ranking.KindRankingListener
import com.intellij.platform.ml.impl.turboComplete.ranking.RankedKind
class ReportingKindSorter(
private val listener: KindRankingListener,
private val sorterProvider: KindSorterProvider
) : KindRelevanceSorter {
override val kindVariety: KindVariety
get() = sorterProvider.kindVariety
override fun sort(kinds: List<CompletionKind>, parameters: CompletionParameters): List<RankedKind> {
val sorter = sorterProvider.createSorter()
listener.onRankingStarted()
return try {
val kindsSorted = sorter.sort(kinds, parameters)
listener.onRanked(kindsSorted)
kindsSorted
}
finally {
listener.onRankingFinished()
}
}
}

View File

@@ -0,0 +1,196 @@
package com.intellij.codeInsight.completion.ml.performance.analysis.usage
import com.intellij.codeInsight.completion.CompletionParameters
import com.intellij.platform.ml.impl.turboComplete.CompletionKind
import com.intellij.platform.ml.impl.turboComplete.KindVariety
import com.intellij.turboComplete.NullableKindName
import com.intellij.turboComplete.analysis.usage.CombinedKindUsageStatistics
import com.intellij.turboComplete.analysis.usage.CombinedKindUsageTracker
import com.intellij.turboComplete.analysis.usage.KindStatistics
import com.intellij.turboComplete.analysis.usage.ValuePerPeriod
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test
class CombinedKindUsageTrackerTest {
private enum class Event {
CREATED,
GENERATED_CORRECT,
GENERATED_NOT_CORRECT,
}
private fun makeMockCompletionKind() = CompletionKind(
NullableKindName.NONE_KIND,
object : KindVariety {
override fun kindsCorrespondToParameters(parameters: CompletionParameters) = true
override val actualCompletionContributorClass: Class<*>
get() = throw NotImplementedError()
}
)
private fun getEventsStatistics(windowSize: Int, events: List<Event>): CombinedKindUsageStatistics {
val kind = makeMockCompletionKind()
val tracker = CombinedKindUsageTracker(kind, windowSize, windowSize)
events.forEach {
when (it) {
Event.CREATED -> tracker.trackCreated()
Event.GENERATED_CORRECT -> tracker.trackGenerated(true)
Event.GENERATED_NOT_CORRECT -> tracker.trackGenerated(false)
}
}
return tracker.getSummary()
}
private fun validateStates(windowSize: Int, events: List<Pair<Event, CombinedKindUsageStatistics>>) {
val kind = makeMockCompletionKind()
val tracker = CombinedKindUsageTracker(kind, windowSize, windowSize)
events.forEach { (event, expectedStatistics) ->
when (event) {
Event.CREATED -> tracker.trackCreated()
Event.GENERATED_CORRECT -> tracker.trackGenerated(true)
Event.GENERATED_NOT_CORRECT -> tracker.trackGenerated(false)
}
assertEquals(expectedStatistics, tracker.getSummary())
}
}
@Test
fun `test total created`() {
val windowSize = 3
fun assertCreated(expected: Int, events: List<Event>) {
assertEquals(expected, getEventsStatistics(windowSize, events).created)
}
assertCreated(0, listOf())
assertCreated(1, listOf(Event.CREATED))
assertCreated(2, listOf(Event.CREATED, Event.CREATED))
assertCreated(15, List(15) { Event.CREATED })
assertCreated(1, listOf(Event.CREATED, Event.GENERATED_CORRECT))
assertCreated(1, listOf(Event.CREATED, Event.GENERATED_CORRECT, Event.GENERATED_NOT_CORRECT))
}
@Test
fun `test total generated correct`() {
val windowSize = 3
fun assertCorrect(expected: Int, events: List<Event>) {
assertEquals(expected, getEventsStatistics(windowSize, events).generated.correct)
}
assertCorrect(0, listOf())
assertCorrect(0, listOf(Event.CREATED))
assertCorrect(0, listOf(Event.GENERATED_NOT_CORRECT))
assertCorrect(1, listOf(Event.GENERATED_CORRECT))
assertCorrect(2, listOf(Event.GENERATED_NOT_CORRECT, Event.GENERATED_CORRECT, Event.CREATED, Event.CREATED, Event.GENERATED_CORRECT))
}
@Test
fun `test total generated not correct`() {
val windowSize = 3
fun assertCorrect(expected: Int, events: List<Event>) {
assertEquals(expected, getEventsStatistics(windowSize, events).generated.notCorrect)
}
assertCorrect(0, listOf())
assertCorrect(0, listOf(Event.CREATED))
assertCorrect(1, listOf(Event.GENERATED_NOT_CORRECT))
assertCorrect(0, listOf(Event.GENERATED_CORRECT))
assertCorrect(1, listOf(Event.GENERATED_NOT_CORRECT, Event.GENERATED_CORRECT, Event.CREATED, Event.CREATED, Event.GENERATED_CORRECT))
}
@Test
fun `test total not correct in row`() {
val windowSize = 3
fun assertNotCorrectInRow(expected: Int, events: List<Event>) {
assertEquals(expected, getEventsStatistics(windowSize, events).generatedInRow.notCorrect)
}
assertNotCorrectInRow(0, listOf())
assertNotCorrectInRow(0, listOf(Event.CREATED))
assertNotCorrectInRow(0, listOf(Event.GENERATED_CORRECT))
assertNotCorrectInRow(1, listOf(Event.GENERATED_NOT_CORRECT))
assertNotCorrectInRow(0, listOf(Event.GENERATED_NOT_CORRECT, Event.GENERATED_CORRECT, Event.CREATED, Event.CREATED,
Event.GENERATED_CORRECT))
assertNotCorrectInRow(3, List(4) { Event.GENERATED_NOT_CORRECT })
assertNotCorrectInRow(2, List(2) { Event.GENERATED_NOT_CORRECT } + listOf(Event.CREATED))
assertNotCorrectInRow(3, List(2) { Event.GENERATED_NOT_CORRECT } + listOf(Event.CREATED, Event.GENERATED_NOT_CORRECT))
assertNotCorrectInRow(0, List(4) { Event.GENERATED_NOT_CORRECT } + listOf(Event.CREATED, Event.GENERATED_CORRECT))
assertNotCorrectInRow(2, listOf(
Event.CREATED,
Event.CREATED,
Event.GENERATED_CORRECT,
Event.GENERATED_NOT_CORRECT,
Event.CREATED,
Event.GENERATED_NOT_CORRECT,
Event.GENERATED_NOT_CORRECT,
Event.GENERATED_CORRECT,
Event.GENERATED_NOT_CORRECT,
Event.GENERATED_NOT_CORRECT,
))
}
@Test
fun `test recently generated correct`() {
val windowSize = 2
fun assertCreated(expected: Int, events: List<Event>) {
assertEquals(expected, getEventsStatistics(windowSize, events).recentGenerated.value.correct)
}
assertCreated(0, listOf())
assertCreated(0, listOf(Event.CREATED))
assertCreated(1, listOf(Event.CREATED, Event.GENERATED_CORRECT))
assertCreated(3, listOf(Event.CREATED, Event.GENERATED_CORRECT, Event.GENERATED_CORRECT, Event.GENERATED_CORRECT))
assertCreated(0, listOf(Event.CREATED, Event.GENERATED_CORRECT, Event.GENERATED_CORRECT, Event.CREATED, Event.CREATED))
assertCreated(0, listOf(Event.CREATED, Event.GENERATED_CORRECT, Event.GENERATED_CORRECT, Event.CREATED, Event.CREATED, Event.CREATED))
}
@Test
fun `test event sequence`() {
validateStates(2, listOf(
Event.CREATED to CombinedKindUsageStatistics(
1, generated = KindStatistics(0, 0), generatedInRow = KindStatistics(0, 0), ValuePerPeriod(1, KindStatistics(0, 0))
),
Event.GENERATED_CORRECT to CombinedKindUsageStatistics(
1, generated = KindStatistics(1, 0), generatedInRow = KindStatistics(1, 0), ValuePerPeriod(1, KindStatistics(1, 0))
),
Event.CREATED to CombinedKindUsageStatistics(
2, generated = KindStatistics(1, 0), generatedInRow = KindStatistics(1, 0), ValuePerPeriod(2, KindStatistics(1, 0))
),
Event.GENERATED_CORRECT to CombinedKindUsageStatistics(
2, generated = KindStatistics(2, 0), generatedInRow = KindStatistics(2, 0), ValuePerPeriod(2, KindStatistics(2, 0))
),
Event.CREATED to CombinedKindUsageStatistics(
3, generated = KindStatistics(2, 0), generatedInRow = KindStatistics(2, 0), ValuePerPeriod(2, KindStatistics(1, 0))
),
Event.GENERATED_CORRECT to CombinedKindUsageStatistics(
3, generated = KindStatistics(3, 0), generatedInRow = KindStatistics(2, 0), ValuePerPeriod(2, KindStatistics(2, 0))
),
Event.CREATED to CombinedKindUsageStatistics(
4, generated = KindStatistics(3, 0), generatedInRow = KindStatistics(2, 0), ValuePerPeriod(2, KindStatistics(1, 0))
),
Event.GENERATED_NOT_CORRECT to CombinedKindUsageStatistics(
4, generated = KindStatistics(3, 1), generatedInRow = KindStatistics(0, 1), ValuePerPeriod(2, KindStatistics(1, 1))
),
Event.CREATED to CombinedKindUsageStatistics(
5, generated = KindStatistics(3, 1), generatedInRow = KindStatistics(0, 1), ValuePerPeriod(2, KindStatistics(0, 1))
),
Event.GENERATED_CORRECT to CombinedKindUsageStatistics(
5, generated = KindStatistics(4, 1), generatedInRow = KindStatistics(1, 0), ValuePerPeriod(2, KindStatistics(1, 1))
),
Event.CREATED to CombinedKindUsageStatistics(
6, generated = KindStatistics(4, 1), generatedInRow = KindStatistics(1, 0), ValuePerPeriod(2, KindStatistics(1, 0))
),
Event.GENERATED_CORRECT to CombinedKindUsageStatistics(
6, generated = KindStatistics(5, 1), generatedInRow = KindStatistics(2, 0), ValuePerPeriod(2, KindStatistics(2, 0))
),
))
}
}