mirror of
https://gitflic.ru/project/openide/openide.git
synced 2026-09-27 10:03:11 +07:00
IJPL-222388 IJ-MR-184993 fix codeserver product and improve validation
GitOrigin-RevId: 1fe406284a60933466a666258d5361c0e1913370
This commit is contained in:
Vladimir Krivosheev
committed by
intellij-monorepo-bot
intellij-monorepo-bot
parent
26218580ab
commit
9625a6c660
@@ -169,182 +169,9 @@ But does NOT duplicate these content modules (already in `fleetMinimal()`):
|
||||
|
||||
## Module Set Validation
|
||||
|
||||
### Two-Tier Validation System
|
||||
|
||||
The build system uses a **two-tier validation approach** to ensure module dependencies are resolvable while avoiding false positives:
|
||||
|
||||
#### Tier 1: Product-Level Validation
|
||||
|
||||
**What it validates**: All products with their complete module composition
|
||||
|
||||
**How it works**:
|
||||
- Collects all modules from a product's complete module set hierarchy
|
||||
- Validates that each module's dependencies are available within that product context
|
||||
- Reports errors per-product with clear affected modules
|
||||
|
||||
**Why this tier exists**: Products are the actual deployment units. Most module sets (like `debugger()`, `vcs()`, `xml()`) are **composable building blocks** designed to work together. Validating them in isolation would produce false positives because they intentionally depend on modules from other sets (e.g., `debugger()` depends on `intellij.platform.core` from `essential()`).
|
||||
|
||||
**Example**:
|
||||
- Product `GoLand` uses: `ide.ultimate`, `ssh`, `rd.common`
|
||||
- Validation checks: Can all modules in `ide.ultimate` + `ssh` + `rd.common` resolve their dependencies within this combined set?
|
||||
|
||||
#### Tier 2: Self-Contained Validation
|
||||
|
||||
**What it validates**: Module sets marked with `selfContained = true`
|
||||
|
||||
**How it works**:
|
||||
- Validates the module set in isolation without considering other module sets
|
||||
- Ensures all dependencies are resolvable within the set itself
|
||||
- Reports errors specific to that module set
|
||||
|
||||
**Why this tier exists**: Some module sets are designed to be **standalone/self-contained** and used directly by products without composition with other sets. These sets must have all their dependencies available internally.
|
||||
|
||||
**When to use `selfContained = true`**:
|
||||
- Module set is used directly by products as the primary/only module set
|
||||
- Module set represents a complete, independent runtime environment
|
||||
- You want to enforce that the set doesn't leak dependencies on external sets
|
||||
|
||||
**When NOT to use `selfContained = true`**:
|
||||
- Module set is designed to be composed with other sets (debugger, vcs, xml, ssh)
|
||||
- Module set intentionally depends on modules from base sets like `essential()`
|
||||
- Module set is a specialized feature addition to a larger base
|
||||
|
||||
### Example: core.platform (Self-Contained)
|
||||
|
||||
```kotlin
|
||||
/**
|
||||
* Core platform modules without IDE or language support.
|
||||
* Used by: CodeServer (analysis tool)
|
||||
*/
|
||||
fun corePlatform(): ModuleSet = moduleSet(
|
||||
name = "core.platform",
|
||||
selfContained = true, // ✅ Must be resolvable in isolation
|
||||
outputModule = "intellij.platform.ide.core"
|
||||
) {
|
||||
moduleSet(librariesPlatform())
|
||||
moduleSet(rpcMinimal()) // Provides kernel + fleet deps
|
||||
|
||||
embeddedModule("intellij.platform.core", includeDependencies = true)
|
||||
embeddedModule("intellij.platform.ide.core", includeDependencies = true)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Why selfContained**: CodeServer uses `core.platform` alone without other module sets. It must contain everything needed for the platform runtime.
|
||||
|
||||
### Example: debugger() (Composable, Not Self-Contained)
|
||||
|
||||
```kotlin
|
||||
/**
|
||||
* Debugger platform modules.
|
||||
* Used by: All IDE products via essential()
|
||||
*/
|
||||
fun debugger(): ModuleSet = moduleSet(
|
||||
name = "debugger"
|
||||
// ❌ NOT selfContained - designed to compose with essential()
|
||||
) {
|
||||
module("intellij.platform.debugger.impl.backend")
|
||||
embeddedModule("intellij.platform.debugger")
|
||||
// Depends on intellij.platform.core from essential() ✅ OK
|
||||
}
|
||||
```
|
||||
|
||||
**Why NOT selfContained**: Debugger is always used together with `essential()` which provides core platform modules. Validating debugger in isolation would fail because it depends on `intellij.platform.core`, but that's correct by design.
|
||||
|
||||
### Validation Error Examples
|
||||
|
||||
#### Product-Level Error
|
||||
|
||||
```
|
||||
❌ Unresolvable dependencies in products
|
||||
|
||||
Product: GoLand
|
||||
✗ Missing: 'intellij.platform.polySymbols'
|
||||
Needed by: intellij.platform.vcs.impl
|
||||
Chain: intellij.platform.vcs.impl → intellij.platform.polySymbols
|
||||
```
|
||||
|
||||
**What this means**: The product `GoLand` is missing a module that one of its included modules depends on. Fix by adding the missing module or a module set that contains it.
|
||||
|
||||
#### Self-Contained Error
|
||||
|
||||
```
|
||||
❌ Module set 'core.platform' is marked selfContained but has unresolvable dependencies
|
||||
|
||||
✗ Missing: 'fleet.kernel'
|
||||
Needed by: intellij.platform.kernel
|
||||
Suggestion: Include fleet() or add fleet.kernel directly
|
||||
```
|
||||
|
||||
**What this means**: A self-contained module set is missing a dependency. Fix by adding the missing module/set to make it truly self-contained.
|
||||
|
||||
### Troubleshooting Dependency Errors
|
||||
|
||||
When you encounter a dependency validation error, **resist the temptation to simply add the missing module to the failing module set**. This creates technical debt and breaks the isolation principle.
|
||||
|
||||
#### ❌ Wrong Approach
|
||||
|
||||
```kotlin
|
||||
// BAD: Adding a library directly to debugger module set
|
||||
fun debugger(): ModuleSet = moduleSet("debugger") {
|
||||
module("intellij.platform.debugger.impl")
|
||||
embeddedModule("intellij.libraries.kotlinx.serialization.core") // ❌ Wrong!
|
||||
}
|
||||
```
|
||||
|
||||
**Why it's wrong**: The debugger module set should contain only debugger-related modules. Adding serialization libraries pollutes its domain and creates maintenance burden.
|
||||
|
||||
#### ✅ Correct Approach
|
||||
|
||||
1. **Identify the dependency chain**: Use PMA MCP to trace dependencies
|
||||
```
|
||||
mcp__PluginModelAnalyzer__find_dependency_path(
|
||||
fromModule="intellij.platform.debugger.impl",
|
||||
toModule="intellij.libraries.kotlinx.serialization.core"
|
||||
)
|
||||
```
|
||||
|
||||
2. **Find which module set provides the dependency**: Check existing module sets
|
||||
```
|
||||
mcp__PluginModelAnalyzer__suggest_module_set_for_modules(
|
||||
moduleNames=["intellij.libraries.kotlinx.serialization.core"]
|
||||
)
|
||||
```
|
||||
|
||||
3. **Analyze the product's module set hierarchy**: Trace what the product includes
|
||||
- Does `essentialMinimal()` → `coreLang()` → `coreIde()` → `corePlatform()` → `librariesPlatform()` include it?
|
||||
- Is there a missing link in the chain?
|
||||
|
||||
4. **Fix at the appropriate level**:
|
||||
- If a base module set is missing a nested set → add the nested set
|
||||
- If a product is missing a required module set → add to product
|
||||
- If the module truly belongs in the domain → then add it directly (rare)
|
||||
|
||||
#### Example: Serialization Library Missing
|
||||
|
||||
**Error**: `debugger` module set missing `intellij.libraries.kotlinx.serialization.core`
|
||||
|
||||
**Investigation**:
|
||||
- `kotlinx.serialization.core` is in `librariesPlatform()` module set
|
||||
- `librariesPlatform()` is nested in `corePlatform()`
|
||||
- `corePlatform()` is nested in `coreIde()` → `coreLang()` → `essentialMinimal()`
|
||||
- Product uses `essentialMinimal()` ✅ so it should have the library
|
||||
|
||||
**Root cause options**:
|
||||
1. Product doesn't include `essentialMinimal()` - add it
|
||||
2. The chain is broken somewhere - fix the nesting
|
||||
3. Validation logic has a bug - investigate the validator
|
||||
|
||||
**Key insight**: The fix is rarely "add module X to module set Y". It's usually "ensure the module set hierarchy correctly provides module X through proper nesting".
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Start without `selfContained`**: Most module sets should NOT be self-contained. Only flag sets that truly need isolation.
|
||||
|
||||
2. **Keep self-contained sets minimal**: If a set needs `selfContained = true`, try to minimize dependencies to keep it lightweight.
|
||||
|
||||
3. **Document why self-contained**: When using `selfContained = true`, add a KDoc comment explaining why this set needs isolation.
|
||||
|
||||
4. **Use product-level validation as guide**: If validation suggests adding many dependencies to a self-contained set, consider whether it should really be self-contained.
|
||||
|
||||
5. **Test with real products**: After changes, run "Generate Product Layouts" to validate both tiers work correctly.
|
||||
See [Validation Documentation](validation.md) for comprehensive coverage of:
|
||||
- Two-tier validation system (product-level and self-contained)
|
||||
- Cross-plugin dependency validation
|
||||
- Loading attribute semantics (`embedded`/`required` vs `optional`/`on_demand`)
|
||||
- `allowMissingDependencies` usage
|
||||
- Troubleshooting validation errors
|
||||
@@ -767,7 +767,8 @@ Before committing changes:
|
||||
|
||||
## See Also
|
||||
|
||||
- [Module Sets Documentation](module-sets.md)
|
||||
- [Module Sets Documentation](module-sets.md) - How module sets work and composition
|
||||
- [Validation Documentation](validation.md) - Dependency validation and troubleshooting
|
||||
- `ProductModulesContentSpec` class documentation
|
||||
- `ModuleSet` and `ContentModule` classes
|
||||
- Example: `GatewayProperties.getProductContentModules()`
|
||||
@@ -9,6 +9,11 @@ import kotlinx.coroutines.awaitAll
|
||||
import kotlinx.coroutines.coroutineScope
|
||||
import org.jetbrains.intellij.build.ModuleOutputProvider
|
||||
import org.jetbrains.intellij.build.findFileInModuleSources
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.MissingDependenciesError
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.ValidationError
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.formatProductDependencyErrorsFooter
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.formatProductDependencyErrorsHeader
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.formatValidationErrors
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.validateProductModuleSets
|
||||
import org.jetbrains.intellij.build.productLayout.analysis.validateSelfContainedModuleSets
|
||||
import java.nio.file.Files
|
||||
@@ -45,10 +50,9 @@ internal suspend fun generateModuleDescriptorDependencies(
|
||||
coreModuleSets: List<ModuleSet> = emptyList(),
|
||||
moduleOutputProvider: ModuleOutputProvider,
|
||||
productSpecs: List<Pair<String, ProductModulesContentSpec?>> = emptyList(),
|
||||
embeddedModulesDeferred: Deferred<Set<String>>? = null,
|
||||
pluginContentJobs: Map<String, Deferred<PluginContentInfo?>> = emptyMap(),
|
||||
): DependencyGenerationResult = coroutineScope {
|
||||
val allModuleSets = communityModuleSets + coreModuleSets + ultimateModuleSets
|
||||
val embeddedModules = embeddedModulesDeferred?.await() ?: emptySet()
|
||||
val modulesToProcess = collectModulesToProcess(allModuleSets)
|
||||
if (modulesToProcess.isEmpty()) {
|
||||
return@coroutineScope DependencyGenerationResult(emptyList())
|
||||
@@ -56,18 +60,33 @@ internal suspend fun generateModuleDescriptorDependencies(
|
||||
|
||||
val cache = ModuleDescriptorCache(moduleOutputProvider)
|
||||
|
||||
// Collect all validation errors
|
||||
val errors = mutableListOf<ValidationError>()
|
||||
|
||||
// Validate self-contained module sets in isolation
|
||||
// Module sets marked with selfContained=true must be resolvable without other sets
|
||||
validateSelfContainedModuleSets(allModuleSets, cache)
|
||||
errors.addAll(validateSelfContainedModuleSets(allModuleSets, cache))
|
||||
|
||||
// Tier 2: Validate product-level dependencies
|
||||
// This ensures all products can load without missing dependency errors
|
||||
validateProductModuleSets(
|
||||
errors.addAll(validateProductModuleSets(
|
||||
allModuleSets = allModuleSets,
|
||||
productSpecs = productSpecs,
|
||||
descriptorCache = cache,
|
||||
precomputedEmbeddedModules = embeddedModules,
|
||||
)
|
||||
pluginContentJobs = pluginContentJobs,
|
||||
))
|
||||
|
||||
// Report all errors at once
|
||||
if (errors.isNotEmpty()) {
|
||||
val hasMissingDependencies = errors.any { it is MissingDependenciesError }
|
||||
error(buildString {
|
||||
if (hasMissingDependencies) {
|
||||
formatProductDependencyErrorsHeader(this)
|
||||
}
|
||||
append(formatValidationErrors(errors))
|
||||
if (hasMissingDependencies) {
|
||||
formatProductDependencyErrorsFooter(this)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Write XML files in parallel
|
||||
val results = modulesToProcess.map { moduleName ->
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
// Copyright 2000-2025 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license.
|
||||
@file:Suppress("ReplaceGetOrSet")
|
||||
|
||||
package org.jetbrains.intellij.build.productLayout
|
||||
|
||||
import kotlinx.coroutines.Deferred
|
||||
import kotlinx.coroutines.async
|
||||
import kotlinx.coroutines.awaitAll
|
||||
import kotlinx.coroutines.coroutineScope
|
||||
import org.jetbrains.intellij.build.ModuleOutputProvider
|
||||
import org.jetbrains.intellij.build.PLUGIN_XML_RELATIVE_PATH
|
||||
import org.jetbrains.intellij.build.findFileInModuleSources
|
||||
import java.nio.file.Files
|
||||
|
||||
/**
|
||||
@@ -14,13 +14,16 @@ import java.nio.file.Files
|
||||
* Uses same logic as ModuleDescriptorDependencyGenerator but for plugin.xml files.
|
||||
*
|
||||
* For each bundled plugin module:
|
||||
* 1. Finds META-INF/plugin.xml in module sources
|
||||
* 2. Gets JPS production dependencies that have XML descriptors (content modules)
|
||||
* 1. Uses pre-extracted content from shared jobs (path, content, JPS deps - avoids duplicate lookups)
|
||||
* 2. Filters JPS production dependencies to those with XML descriptors
|
||||
* 3. Updates the `<dependencies>` section with generated `<module name="..."/>` entries
|
||||
*
|
||||
* @param pluginContentJobs Pre-launched async jobs containing all plugin info.
|
||||
* Multiple consumers can await the same Deferred - extraction runs only once per plugin.
|
||||
*/
|
||||
internal suspend fun generatePluginDependencies(
|
||||
plugins: List<String>,
|
||||
moduleOutputProvider: ModuleOutputProvider,
|
||||
pluginContentJobs: Map<String, Deferred<PluginContentInfo?>>,
|
||||
descriptorCache: ModuleDescriptorCache,
|
||||
dependencyFilter: (String) -> Boolean,
|
||||
): PluginDependencyGenerationResult = coroutineScope {
|
||||
@@ -32,7 +35,7 @@ internal suspend fun generatePluginDependencies(
|
||||
async {
|
||||
generatePluginDependency(
|
||||
pluginModuleName = pluginModuleName,
|
||||
moduleOutputProvider = moduleOutputProvider,
|
||||
pluginContentJobs = pluginContentJobs,
|
||||
descriptorCache = descriptorCache,
|
||||
dependencyFilter = dependencyFilter,
|
||||
)
|
||||
@@ -45,49 +48,38 @@ internal suspend fun generatePluginDependencies(
|
||||
/**
|
||||
* Generates dependencies for a single plugin module.
|
||||
*
|
||||
* @return PluginDependencyFileResult or null if plugin.xml not found
|
||||
* @param pluginContentJobs Pre-launched async jobs containing all plugin info.
|
||||
* @return PluginDependencyFileResult or null if plugin.xml not found or has module refs with '/'
|
||||
*/
|
||||
private fun generatePluginDependency(
|
||||
private suspend fun generatePluginDependency(
|
||||
pluginModuleName: String,
|
||||
moduleOutputProvider: ModuleOutputProvider,
|
||||
pluginContentJobs: Map<String, Deferred<PluginContentInfo?>>,
|
||||
descriptorCache: ModuleDescriptorCache,
|
||||
dependencyFilter: (String) -> Boolean,
|
||||
): PluginDependencyFileResult? {
|
||||
val jpsModule = moduleOutputProvider.findModule(pluginModuleName) ?: return null
|
||||
// All data from shared jobs - NO additional lookups needed
|
||||
val info = pluginContentJobs.get(pluginModuleName)?.await() ?: return null
|
||||
|
||||
val pluginXmlPath = findFileInModuleSources(module = jpsModule, relativePath = PLUGIN_XML_RELATIVE_PATH, onlyProductionSources = true) ?: return null
|
||||
|
||||
// Read file once and extract content modules (these should be excluded from dependencies)
|
||||
val pluginXmlContent = Files.readString(pluginXmlPath)
|
||||
// null means plugin has content module references (modules with '/'), skip it
|
||||
val contentModules = extractContentModulesFromText(pluginXmlContent) ?: return null
|
||||
|
||||
// Get JPS dependencies that have XML descriptors (content modules)
|
||||
// Skip if:
|
||||
// 1. Module is in <content> section of this plugin.xml
|
||||
// 2. Module is filtered out by dependencyFilter
|
||||
// 3. Module doesn't have a descriptor
|
||||
val deps = mutableListOf<String>()
|
||||
for (dep in jpsModule.getProductionModuleDependencies(withTests = false)) {
|
||||
val depName = dep.moduleReference.moduleName
|
||||
if (depName in contentModules) {
|
||||
continue
|
||||
// Filter JPS dependencies: exclude content modules, apply filter, require descriptor
|
||||
val dependencies = info.jpsDependencies()
|
||||
.filter { depName ->
|
||||
depName !in info.contentModules &&
|
||||
dependencyFilter(depName) &&
|
||||
descriptorCache.hasDescriptor(depName)
|
||||
}
|
||||
if (!dependencyFilter(depName)) {
|
||||
continue
|
||||
}
|
||||
if (!descriptorCache.hasDescriptor(depName)) {
|
||||
continue
|
||||
}
|
||||
deps.add(depName)
|
||||
}
|
||||
.distinct()
|
||||
.sorted()
|
||||
|
||||
val dependencies = deps.distinct().sorted()
|
||||
val status = updateXmlDependencies(path = pluginXmlPath, content = pluginXmlContent, moduleDependencies = dependencies, preserveExistingModule = { !dependencyFilter(it) })
|
||||
val status = updateXmlDependencies(
|
||||
path = info.pluginXmlPath,
|
||||
content = info.pluginXmlContent,
|
||||
moduleDependencies = dependencies,
|
||||
preserveExistingModule = { !dependencyFilter(it) },
|
||||
)
|
||||
|
||||
// Also process content modules - generate dependencies for their module descriptors
|
||||
val contentModuleResults = mutableListOf<DependencyFileResult>()
|
||||
for (contentModuleName in contentModules) {
|
||||
for (contentModuleName in info.contentModules) {
|
||||
val result = generateContentModuleDependencies(contentModuleName = contentModuleName, descriptorCache = descriptorCache, dependencyFilter = dependencyFilter)
|
||||
if (result != null) {
|
||||
contentModuleResults.add(result)
|
||||
@@ -96,7 +88,7 @@ private fun generatePluginDependency(
|
||||
|
||||
return PluginDependencyFileResult(
|
||||
pluginModuleName = pluginModuleName,
|
||||
pluginXmlPath = pluginXmlPath,
|
||||
pluginXmlPath = info.pluginXmlPath,
|
||||
status = status,
|
||||
dependencyCount = dependencies.size,
|
||||
contentModuleResults = contentModuleResults,
|
||||
|
||||
@@ -141,6 +141,13 @@ class ProductModulesContentSpec(
|
||||
*/
|
||||
@JvmField val bundledPlugins: List<String> = emptyList(),
|
||||
|
||||
/**
|
||||
* Modules that are allowed to be missing during validation.
|
||||
* These are typically provided by plugin layouts rather than module sets.
|
||||
* Example: CIDR modules for CLion/AppCode that come from plugin bundles.
|
||||
*/
|
||||
@JvmField val allowedMissingDependencies: Set<String> = emptySet(),
|
||||
|
||||
/**
|
||||
* Composition graph tracking how this spec was assembled.
|
||||
* Records all include(), moduleSet(), and other composition operations.
|
||||
@@ -166,6 +173,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
private val moduleSets = mutableListOf<ModuleSetWithOverrides>()
|
||||
private val additionalModules = mutableListOf<ContentModule>()
|
||||
private val bundledPlugins = mutableListOf<String>()
|
||||
private val allowedMissingDeps = LinkedHashSet<String>()
|
||||
|
||||
// Composition tracking
|
||||
private val compositionGraph = mutableListOf<SpecComposition>()
|
||||
@@ -230,6 +238,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
xmlIncludes.addAll(spec.deprecatedXmlIncludes)
|
||||
moduleSets.addAll(spec.moduleSets)
|
||||
additionalModules.addAll(spec.additionalModules)
|
||||
allowedMissingDeps.addAll(spec.allowedMissingDependencies)
|
||||
|
||||
// Also preserve the nested spec's composition graph for deep analysis
|
||||
compositionGraph.addAll(spec.compositionGraph)
|
||||
@@ -310,7 +319,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
type = CompositionType.DIRECT_MODULE,
|
||||
reference = name,
|
||||
path = pathStack.toList(),
|
||||
sourceLocation = null
|
||||
sourceLocation = null,
|
||||
))
|
||||
}
|
||||
|
||||
@@ -323,7 +332,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
type = CompositionType.DIRECT_MODULE,
|
||||
reference = name,
|
||||
path = pathStack.toList(),
|
||||
sourceLocation = null
|
||||
sourceLocation = null,
|
||||
))
|
||||
}
|
||||
|
||||
@@ -351,6 +360,16 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
bundledPlugins.addAll(pluginModules)
|
||||
}
|
||||
|
||||
/**
|
||||
* Allow specific modules to be missing during validation.
|
||||
* Use for modules provided by plugin layouts rather than module sets.
|
||||
*
|
||||
* @param modules Module names that are allowed to be missing
|
||||
*/
|
||||
fun allowMissingDependencies(vararg modules: String) {
|
||||
allowedMissingDeps.addAll(modules)
|
||||
}
|
||||
|
||||
@PublishedApi
|
||||
internal fun build(): ProductModulesContentSpec {
|
||||
return ProductModulesContentSpec(
|
||||
@@ -360,6 +379,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() {
|
||||
moduleSets = java.util.List.copyOf(moduleSets),
|
||||
additionalModules = java.util.List.copyOf(additionalModules),
|
||||
bundledPlugins = java.util.List.copyOf(bundledPlugins),
|
||||
allowedMissingDependencies = java.util.Set.copyOf(allowedMissingDeps),
|
||||
compositionGraph = java.util.List.copyOf(compositionGraph),
|
||||
metadata = metadata,
|
||||
)
|
||||
|
||||
@@ -2,192 +2,117 @@
|
||||
package org.jetbrains.intellij.build.productLayout.analysis
|
||||
|
||||
import com.intellij.platform.plugins.parser.impl.elements.ModuleLoadingRuleValue
|
||||
import kotlinx.coroutines.Deferred
|
||||
import org.jetbrains.intellij.build.productLayout.AnsiColors
|
||||
import org.jetbrains.intellij.build.productLayout.ModuleDescriptorCache
|
||||
import org.jetbrains.intellij.build.productLayout.ModuleSet
|
||||
import org.jetbrains.intellij.build.productLayout.PluginContentInfo
|
||||
import org.jetbrains.intellij.build.productLayout.ProductModulesContentSpec
|
||||
|
||||
/**
|
||||
* Known missing dependencies that are temporarily allowed.
|
||||
* These are typically provided by plugin layouts rather than module sets.
|
||||
* TODO: Move these to proper module sets or fix product definitions
|
||||
*/
|
||||
internal val KNOWN_MISSING_DEPENDENCIES = setOf(
|
||||
"intellij.cidr.core",
|
||||
"intellij.cidr.util.execution",
|
||||
"intellij.cidr.debugger.core",
|
||||
"intellij.cidr.debugger.backend",
|
||||
"intellij.cidr.runner"
|
||||
)
|
||||
// region Validation Result Types
|
||||
|
||||
internal sealed interface ValidationError {
|
||||
val context: String
|
||||
}
|
||||
|
||||
internal data class SelfContainedValidationError(
|
||||
override val context: String,
|
||||
val missingDependencies: Map<String, Set<String>>,
|
||||
) : ValidationError
|
||||
|
||||
internal data class MissingModuleSetsError(
|
||||
override val context: String,
|
||||
val missingModuleSets: Set<String>,
|
||||
) : ValidationError
|
||||
|
||||
internal data class DuplicateModulesError(
|
||||
override val context: String,
|
||||
val duplicates: Map<String, Int>,
|
||||
) : ValidationError
|
||||
|
||||
internal data class MissingDependenciesError(
|
||||
override val context: String,
|
||||
val missingModules: Map<String, Set<String>>,
|
||||
val allModuleSets: List<ModuleSet>,
|
||||
) : ValidationError
|
||||
|
||||
// endregion
|
||||
|
||||
// region Validation Functions
|
||||
|
||||
/**
|
||||
* Validates module sets marked with selfContained=true in isolation.
|
||||
*
|
||||
*
|
||||
* Self-contained module sets must be resolvable without other module sets.
|
||||
* This ensures they have all their dependencies available internally.
|
||||
*
|
||||
*
|
||||
* Example: core.platform is self-contained because CodeServer uses it alone
|
||||
* without other module sets, so it must contain everything needed.
|
||||
*
|
||||
* Only validates self-contained sets to avoid false positives from composable
|
||||
* module sets like debugger(), vcs(), xml() that depend on modules from other sets.
|
||||
*
|
||||
* **Key difference from normal validation**: For self-contained sets, ALL modules
|
||||
* in the entire hierarchy (including nested sets) can see ALL other modules.
|
||||
* This matches runtime behavior where order doesn't matter (topological sort).
|
||||
*
|
||||
* @return List of validation errors (empty if all self-contained sets are valid)
|
||||
*/
|
||||
internal fun validateSelfContainedModuleSets(
|
||||
allModuleSets: List<ModuleSet>,
|
||||
descriptorCache: ModuleDescriptorCache
|
||||
) {
|
||||
// Collect all self-contained module sets (recursively check nested sets too)
|
||||
val selfContainedSets = mutableListOf<ModuleSet>()
|
||||
fun collectSelfContained(moduleSet: ModuleSet) {
|
||||
if (moduleSet.selfContained) {
|
||||
selfContainedSets.add(moduleSet)
|
||||
}
|
||||
for (nestedSet in moduleSet.nestedSets) {
|
||||
collectSelfContained(nestedSet)
|
||||
}
|
||||
}
|
||||
for (moduleSet in allModuleSets) {
|
||||
collectSelfContained(moduleSet)
|
||||
}
|
||||
|
||||
descriptorCache: ModuleDescriptorCache,
|
||||
): List<SelfContainedValidationError> {
|
||||
val selfContainedSets = collectSelfContainedSets(allModuleSets)
|
||||
if (selfContainedSets.isEmpty()) {
|
||||
return
|
||||
return emptyList()
|
||||
}
|
||||
|
||||
// Validate each self-contained set in isolation
|
||||
|
||||
val errors = mutableListOf<SelfContainedValidationError>()
|
||||
|
||||
for (moduleSet in selfContainedSets) {
|
||||
// Collect ALL modules from the entire hierarchy (flatten nested sets)
|
||||
val allModulesInSet = mutableSetOf<String>()
|
||||
fun collectAllModules(ms: ModuleSet) {
|
||||
for (module in ms.modules) {
|
||||
allModulesInSet.add(module.name)
|
||||
}
|
||||
for (nestedSet in ms.nestedSets) {
|
||||
collectAllModules(nestedSet)
|
||||
}
|
||||
}
|
||||
collectAllModules(moduleSet)
|
||||
|
||||
// Collect modules with descriptors
|
||||
val modulesWithDescriptors = allModulesInSet.mapNotNull { moduleName ->
|
||||
descriptorCache.getOrAnalyze(moduleName)?.let { moduleName to it }
|
||||
}
|
||||
|
||||
if (modulesWithDescriptors.isEmpty()) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Validate dependencies: each module must be able to reach its dependencies
|
||||
// within the flattened set (all modules can see all other modules)
|
||||
val missingDeps = mutableMapOf<String, MutableSet<String>>()
|
||||
|
||||
for ((moduleName, info) in modulesWithDescriptors) {
|
||||
// Check direct and transitive dependencies
|
||||
val visited = mutableSetOf(moduleName)
|
||||
val queue = ArrayDeque(info.dependencies.map { it to listOf(moduleName) })
|
||||
|
||||
while (queue.isNotEmpty()) {
|
||||
val (dep, chain) = queue.removeFirst()
|
||||
|
||||
if (dep in visited) {
|
||||
continue
|
||||
}
|
||||
visited.add(dep)
|
||||
|
||||
// Check if dependency is in the flattened set
|
||||
if (dep !in allModulesInSet) {
|
||||
missingDeps.getOrPut(dep) { mutableSetOf() }.add(chain.first())
|
||||
}
|
||||
|
||||
// Add transitive dependencies
|
||||
val depInfo = descriptorCache.getOrAnalyze(dep)
|
||||
if (depInfo != null) {
|
||||
for (transitiveDep in depInfo.dependencies) {
|
||||
queue.add(transitiveDep to (chain + dep))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val allModulesInSet = ModuleSetTraversal.collectAllModuleNames(moduleSet)
|
||||
val missingDeps = findMissingTransitiveDependencies(
|
||||
modules = allModulesInSet,
|
||||
availableModules = allModulesInSet,
|
||||
descriptorCache = descriptorCache,
|
||||
)
|
||||
|
||||
if (missingDeps.isNotEmpty()) {
|
||||
error(buildString {
|
||||
appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Module set '${moduleSet.name}' is marked selfContained but has unresolvable dependencies${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
|
||||
for ((dep, needingModules) in missingDeps.entries.sortedByDescending { it.value.size }) {
|
||||
appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$dep'${AnsiColors.RESET}")
|
||||
appendLine(" Needed by: ${needingModules.sorted().joinToString(", ")}")
|
||||
}
|
||||
|
||||
appendLine()
|
||||
appendLine("${AnsiColors.YELLOW}💡 To fix:${AnsiColors.RESET}")
|
||||
appendLine("1. Add the missing modules/sets to '${moduleSet.name}' to make it truly self-contained")
|
||||
appendLine("2. Or remove selfContained=true if this set is designed to compose with other sets")
|
||||
})
|
||||
errors.add(SelfContainedValidationError(context = moduleSet.name, missingDependencies = missingDeps))
|
||||
}
|
||||
}
|
||||
|
||||
return errors
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates that all products have resolvable module set dependencies.
|
||||
*
|
||||
*
|
||||
* This is Tier 2 validation that ensures products can actually load at runtime.
|
||||
* It validates that:
|
||||
* 1. All module sets referenced by a product exist and are resolvable
|
||||
* 2. All modules in those sets can have their dependencies satisfied within the product's composition
|
||||
* 3. No module references dependencies outside the product's available modules
|
||||
*
|
||||
* This catches the class of errors like:
|
||||
* "Plugin 'Java' has dependency on 'com.intellij.modules.vcs' which is not installed"
|
||||
*
|
||||
* @param allModuleSets All available module sets (community + ultimate)
|
||||
* @param productSpecs List of (productName, ProductModulesContentSpec) pairs
|
||||
* @param descriptorCache Cache for module descriptor information
|
||||
* @throws IllegalStateException if any product has unresolvable dependencies
|
||||
*
|
||||
* @return List of validation errors (empty if all products are valid)
|
||||
*/
|
||||
internal fun validateProductModuleSets(
|
||||
internal suspend fun validateProductModuleSets(
|
||||
allModuleSets: List<ModuleSet>,
|
||||
productSpecs: List<Pair<String, ProductModulesContentSpec?>>,
|
||||
descriptorCache: ModuleDescriptorCache,
|
||||
precomputedEmbeddedModules: Set<String>? = null,
|
||||
) {
|
||||
data class ProductError(
|
||||
val productName: String,
|
||||
val missingModules: Map<String, Set<String>>, // module -> set of dependencies it needs
|
||||
)
|
||||
|
||||
val productErrors = mutableListOf<ProductError>()
|
||||
pluginContentJobs: Map<String, Deferred<PluginContentInfo?>> = emptyMap(),
|
||||
): List<ValidationError> {
|
||||
val allPluginModules = collectAllPluginModules(pluginContentJobs)
|
||||
val moduleSetsByName = allModuleSets.associateBy { it.name }
|
||||
|
||||
val errors = mutableListOf<ValidationError>()
|
||||
|
||||
for ((productName, spec) in productSpecs) {
|
||||
// Skip products without specs
|
||||
if (spec == null) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Build index of modules available in this product
|
||||
val productIndex = buildProductModuleIndex(productName, spec, precomputedEmbeddedModules)
|
||||
|
||||
// Validate that all referenced module sets exist
|
||||
val missingModuleSets = productIndex.referencedModuleSets.filter { it !in moduleSetsByName }
|
||||
|
||||
val productIndex = buildProductModuleIndex(productName, spec, pluginContentJobs)
|
||||
|
||||
// Check for missing module sets
|
||||
val missingModuleSets = productIndex.referencedModuleSets.filterNot { it in moduleSetsByName }
|
||||
if (missingModuleSets.isNotEmpty()) {
|
||||
error(buildString {
|
||||
appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product '$productName' references non-existent module sets${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
for (setName in missingModuleSets.sorted()) {
|
||||
appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Module set '${AnsiColors.BOLD}$setName${AnsiColors.RESET}' does not exist")
|
||||
}
|
||||
appendLine()
|
||||
appendLine("${AnsiColors.BLUE}💡 Fix: Remove the reference or define the module set${AnsiColors.RESET}")
|
||||
})
|
||||
errors.add(MissingModuleSetsError(context = productName, missingModuleSets = missingModuleSets.toSet()))
|
||||
}
|
||||
|
||||
// Validate no duplicate content modules in product
|
||||
// This validation ALWAYS runs, even for products in allowUnresolvableProducts
|
||||
|
||||
// Check for duplicate content modules
|
||||
val allContentModules = mutableListOf<String>()
|
||||
for (moduleSetWithOverrides in spec.moduleSets) {
|
||||
val moduleSet = moduleSetsByName[moduleSetWithOverrides.moduleSet.name]
|
||||
@@ -198,222 +123,276 @@ internal fun validateProductModuleSets(
|
||||
for (module in spec.additionalModules) {
|
||||
allContentModules.add(module.name)
|
||||
}
|
||||
|
||||
|
||||
val duplicateModules = allContentModules.groupingBy { it }.eachCount().filter { it.value > 1 }
|
||||
if (duplicateModules.isNotEmpty()) {
|
||||
error(buildString {
|
||||
appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product '$productName' has duplicate content modules${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
appendLine("${AnsiColors.YELLOW}Duplicated modules (appearing ${AnsiColors.BOLD}${duplicateModules.values.max()}${AnsiColors.RESET}${AnsiColors.YELLOW} times):${AnsiColors.RESET}")
|
||||
for ((moduleName, count) in duplicateModules.entries.sortedBy { it.key }) {
|
||||
appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} ${AnsiColors.BOLD}$moduleName${AnsiColors.RESET} (appears $count times)")
|
||||
}
|
||||
appendLine()
|
||||
appendLine("${AnsiColors.BLUE}💡 This causes runtime error: \"Plugin has duplicated content modules declarations\"${AnsiColors.RESET}")
|
||||
appendLine("${AnsiColors.BLUE}Fix: Remove duplicate moduleSet() nesting or redundant module() calls${AnsiColors.RESET}")
|
||||
})
|
||||
errors.add(DuplicateModulesError(context = productName, duplicates = duplicateModules))
|
||||
}
|
||||
|
||||
// Validate module dependencies within product scope
|
||||
// Note: For non-embedded modules, we use a heuristic: if a dependency has an XML
|
||||
// descriptor but is not in the product's content modules, we assume it's a plugin
|
||||
// module and skip validation. This is because non-embedded content modules can
|
||||
// depend on plugins (bundled or not). This heuristic does not verify the plugin
|
||||
// is actually bundled with the product - that's a separate validation concern.
|
||||
val missingDependencies = mutableMapOf<String, MutableSet<String>>()
|
||||
|
||||
for (moduleName in productIndex.allModules) {
|
||||
val info = descriptorCache.getOrAnalyze(moduleName) ?: continue
|
||||
val reachableModules = productIndex.moduleToReachableModules[moduleName] ?: emptySet()
|
||||
val isEmbedded = moduleName in productIndex.embeddedModules
|
||||
// Check for missing dependencies
|
||||
val criticalModules = productIndex.moduleLoadings
|
||||
.filterValues { it == ModuleLoadingRuleValue.EMBEDDED || it == ModuleLoadingRuleValue.REQUIRED }
|
||||
.keys
|
||||
|
||||
// Check direct dependencies
|
||||
for (dependency in info.dependencies) {
|
||||
if (dependency in reachableModules || dependency in KNOWN_MISSING_DEPENDENCIES) {
|
||||
continue // OK - in content or known exception
|
||||
}
|
||||
val missingDeps = findMissingTransitiveDependencies(
|
||||
modules = productIndex.allModules,
|
||||
availableModules = productIndex.allModules,
|
||||
descriptorCache = descriptorCache,
|
||||
allowedMissing = spec.allowedMissingDependencies,
|
||||
crossPluginModules = allPluginModules,
|
||||
criticalModules = criticalModules,
|
||||
)
|
||||
|
||||
if (isEmbedded) {
|
||||
// Embedded modules: strict - must be in content
|
||||
missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dependency)
|
||||
} else {
|
||||
// Non-embedded modules: heuristic - if has descriptor, assume plugin module
|
||||
if (!descriptorCache.hasDescriptor(dependency)) {
|
||||
// No descriptor = truly missing
|
||||
missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dependency)
|
||||
}
|
||||
// else: has descriptor, assume it's a plugin module - OK
|
||||
}
|
||||
}
|
||||
|
||||
// Check transitive dependencies
|
||||
val visited = mutableSetOf(moduleName)
|
||||
val queue = ArrayDeque(info.dependencies)
|
||||
|
||||
while (queue.isNotEmpty()) {
|
||||
val dep = queue.removeFirst()
|
||||
if (dep in visited) continue
|
||||
visited.add(dep)
|
||||
|
||||
if (dep !in reachableModules && dep !in KNOWN_MISSING_DEPENDENCIES) {
|
||||
if (isEmbedded) {
|
||||
// Embedded modules: strict - must be in content
|
||||
missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dep)
|
||||
} else {
|
||||
// Non-embedded modules: heuristic - if has descriptor, assume plugin module
|
||||
if (!descriptorCache.hasDescriptor(dep)) {
|
||||
missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val depInfo = descriptorCache.getOrAnalyze(dep)
|
||||
if (depInfo != null) {
|
||||
queue.addAll(depInfo.dependencies)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (missingDependencies.isNotEmpty()) {
|
||||
productErrors.add(ProductError(productName, missingDependencies))
|
||||
if (missingDeps.isNotEmpty()) {
|
||||
errors.add(MissingDependenciesError(context = productName, missingModules = missingDeps, allModuleSets = allModuleSets))
|
||||
}
|
||||
}
|
||||
|
||||
// Report all errors
|
||||
if (productErrors.isNotEmpty()) {
|
||||
error(buildString {
|
||||
appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product-level validation failed: Unresolvable module dependencies${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
|
||||
for (productError in productErrors) {
|
||||
appendLine("${AnsiColors.BOLD}Product: ${productError.productName}${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
|
||||
// Group by missing dependency for clearer output
|
||||
val depToModules = mutableMapOf<String, MutableSet<String>>()
|
||||
for ((module, deps) in productError.missingModules) {
|
||||
for (dep in deps) {
|
||||
depToModules.getOrPut(dep) { mutableSetOf() }.add(module)
|
||||
}
|
||||
}
|
||||
|
||||
for ((missingDep, needingModules) in depToModules.entries.sortedByDescending { it.value.size }) {
|
||||
appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$missingDep'${AnsiColors.RESET}")
|
||||
appendLine(" Needed by: ${needingModules.sorted().take(5).joinToString(", ")}")
|
||||
if (needingModules.size > 5) {
|
||||
appendLine(" ... and ${needingModules.size - 5} more modules")
|
||||
}
|
||||
|
||||
// Suggest which module sets contain this dependency
|
||||
val containingSets = allModuleSets.filter { moduleSet ->
|
||||
ModuleSetTraversal.containsModule(moduleSet, missingDep)
|
||||
}.map { it.name }
|
||||
|
||||
if (containingSets.isNotEmpty()) {
|
||||
appendLine(" ${AnsiColors.BLUE}Suggestion:${AnsiColors.RESET} Add module set: ${containingSets.joinToString(" or ")}")
|
||||
}
|
||||
appendLine()
|
||||
}
|
||||
|
||||
appendLine()
|
||||
}
|
||||
|
||||
appendLine("${AnsiColors.BLUE}💡 This will cause runtime errors: \"Plugin X has dependency on Y which is not installed\"${AnsiColors.RESET}")
|
||||
appendLine()
|
||||
appendLine("${AnsiColors.BOLD}To fix:${AnsiColors.RESET}")
|
||||
appendLine("${AnsiColors.BLUE}1.${AnsiColors.RESET} Add the required module sets to the product's getProductContentDescriptor()")
|
||||
appendLine("${AnsiColors.BLUE}2.${AnsiColors.RESET} Or add individual modules via module()/embeddedModule()")
|
||||
appendLine("${AnsiColors.BLUE}3.${AnsiColors.RESET} Or add the product to allowUnresolvableProducts if this is intentional")
|
||||
})
|
||||
}
|
||||
|
||||
return errors
|
||||
}
|
||||
|
||||
// Helper functions
|
||||
// endregion
|
||||
|
||||
// region Helper Functions
|
||||
|
||||
private fun collectSelfContainedSets(allModuleSets: List<ModuleSet>): List<ModuleSet> {
|
||||
val result = mutableListOf<ModuleSet>()
|
||||
|
||||
fun visit(moduleSet: ModuleSet) {
|
||||
if (moduleSet.selfContained) {
|
||||
result.add(moduleSet)
|
||||
}
|
||||
for (nestedSet in moduleSet.nestedSets) {
|
||||
visit(nestedSet)
|
||||
}
|
||||
}
|
||||
|
||||
for (moduleSet in allModuleSets) {
|
||||
visit(moduleSet)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
private suspend fun collectAllPluginModules(pluginContentJobs: Map<String, Deferred<PluginContentInfo?>>): Set<String> {
|
||||
val result = mutableSetOf<String>()
|
||||
for ((_, job) in pluginContentJobs) {
|
||||
job.await()?.contentModules?.let { result.addAll(it) }
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds an index of all modules available in a specific product.
|
||||
* This is product-scoped, unlike the global ModuleSetIndex.
|
||||
* Finds missing transitive dependencies using BFS traversal.
|
||||
*
|
||||
* @param modules Modules to check dependencies for
|
||||
* @param availableModules Modules that are available (dependencies should be in this set)
|
||||
* @param descriptorCache Cache for module descriptor information
|
||||
* @param allowedMissing Dependencies explicitly allowed to be missing
|
||||
* @param crossPluginModules Modules from other plugins (valid for non-critical modules)
|
||||
* @param criticalModules Modules that cannot depend on cross-plugin modules
|
||||
* @return Map of missing dependency -> set of modules that need it
|
||||
*/
|
||||
internal fun findMissingTransitiveDependencies(
|
||||
modules: Set<String>,
|
||||
availableModules: Set<String>,
|
||||
descriptorCache: ModuleDescriptorCache,
|
||||
allowedMissing: Set<String> = emptySet(),
|
||||
crossPluginModules: Set<String> = emptySet(),
|
||||
criticalModules: Set<String> = emptySet(),
|
||||
): Map<String, Set<String>> {
|
||||
val missingDeps = mutableMapOf<String, MutableSet<String>>()
|
||||
|
||||
for (moduleName in modules) {
|
||||
val info = descriptorCache.getOrAnalyze(moduleName) ?: continue
|
||||
val isCritical = moduleName in criticalModules
|
||||
val visited = mutableSetOf(moduleName)
|
||||
val queue = ArrayDeque(info.dependencies)
|
||||
|
||||
while (queue.isNotEmpty()) {
|
||||
val dep = queue.removeFirst()
|
||||
if (dep in visited) {
|
||||
continue
|
||||
}
|
||||
visited.add(dep)
|
||||
|
||||
when {
|
||||
dep in availableModules -> {
|
||||
// Present - traverse into its deps
|
||||
descriptorCache.getOrAnalyze(dep)?.dependencies?.let { queue.addAll(it) }
|
||||
}
|
||||
dep in allowedMissing -> {
|
||||
// Explicitly allowed to be missing - skip
|
||||
}
|
||||
dep in crossPluginModules && !isCritical -> {
|
||||
// Valid cross-plugin optional dependency - skip
|
||||
}
|
||||
else -> {
|
||||
// Missing dependency
|
||||
missingDeps.getOrPut(dep) { mutableSetOf() }.add(moduleName)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return missingDeps
|
||||
}
|
||||
|
||||
private data class ProductModuleIndex(
|
||||
val productName: String,
|
||||
val allModules: Set<String>, // All modules available in this product
|
||||
val embeddedModules: Set<String>, // Modules with EMBEDDED loading
|
||||
val moduleToReachableModules: Map<String, Set<String>>, // Product-scoped reachability
|
||||
val referencedModuleSets: Set<String>, // Module set names used by product
|
||||
val allModules: Set<String>,
|
||||
val referencedModuleSets: Set<String>,
|
||||
val moduleLoadings: Map<String, ModuleLoadingRuleValue?>,
|
||||
)
|
||||
|
||||
private fun buildProductModuleIndex(
|
||||
private suspend fun buildProductModuleIndex(
|
||||
productName: String,
|
||||
spec: ProductModulesContentSpec,
|
||||
precomputedEmbeddedModules: Set<String>? = null,
|
||||
pluginContentJobs: Map<String, Deferred<PluginContentInfo?>> = emptyMap(),
|
||||
): ProductModuleIndex {
|
||||
val allModules = mutableSetOf<String>()
|
||||
val referencedModuleSets = mutableSetOf<String>()
|
||||
val moduleLoadings = mutableMapOf<String, ModuleLoadingRuleValue?>()
|
||||
|
||||
// Recursively collect all modules from a module set
|
||||
fun collectModulesFromSet(moduleSet: ModuleSet) {
|
||||
for (module in moduleSet.modules) {
|
||||
allModules.add(module.name)
|
||||
}
|
||||
for (nestedSet in moduleSet.nestedSets) {
|
||||
collectModulesFromSet(nestedSet)
|
||||
}
|
||||
}
|
||||
|
||||
// Process all module sets referenced by the product
|
||||
// Use moduleSetWithOverrides.moduleSet directly - it already contains the full ModuleSet
|
||||
for (moduleSetWithOverrides in spec.moduleSets) {
|
||||
val moduleSet = moduleSetWithOverrides.moduleSet
|
||||
referencedModuleSets.add(moduleSet.name)
|
||||
collectModulesFromSet(moduleSet)
|
||||
collectModulesWithLoadings(moduleSet, allModules, moduleLoadings)
|
||||
}
|
||||
|
||||
// Add additional individual modules
|
||||
for (module in spec.additionalModules) {
|
||||
allModules.add(module.name)
|
||||
moduleLoadings[module.name] = module.loading
|
||||
}
|
||||
|
||||
// Use precomputed embedded modules if available (optimization to avoid recomputing),
|
||||
// otherwise compute just for this product's modules
|
||||
val embeddedModules = if (precomputedEmbeddedModules != null) {
|
||||
// Filter to only include modules that are in this product
|
||||
allModules.filterTo(mutableSetOf()) { it in precomputedEmbeddedModules }
|
||||
for (pluginName in spec.bundledPlugins) {
|
||||
pluginContentJobs[pluginName]?.await()?.contentModules?.let { allModules.addAll(it) }
|
||||
}
|
||||
else {
|
||||
// Fallback: compute embedded modules for this product
|
||||
val result = mutableSetOf<String>()
|
||||
fun collectEmbeddedFromSet(moduleSet: ModuleSet) {
|
||||
for (module in moduleSet.modules) {
|
||||
if (module.loading == ModuleLoadingRuleValue.EMBEDDED) {
|
||||
result.add(module.name)
|
||||
}
|
||||
}
|
||||
for (nestedSet in moduleSet.nestedSets) {
|
||||
collectEmbeddedFromSet(nestedSet)
|
||||
}
|
||||
}
|
||||
for (moduleSetWithOverrides in spec.moduleSets) {
|
||||
collectEmbeddedFromSet(moduleSetWithOverrides.moduleSet)
|
||||
}
|
||||
for (module in spec.additionalModules) {
|
||||
if (module.loading == ModuleLoadingRuleValue.EMBEDDED) {
|
||||
result.add(module.name)
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
// Build product-scoped reachability: each module can see all other modules in the product
|
||||
// This is different from module set validation where modules can only see within their set
|
||||
val moduleToReachableModules = allModules.associateWith { allModules }
|
||||
|
||||
return ProductModuleIndex(
|
||||
productName = productName,
|
||||
allModules = allModules,
|
||||
embeddedModules = embeddedModules,
|
||||
moduleToReachableModules = moduleToReachableModules,
|
||||
referencedModuleSets = referencedModuleSets,
|
||||
moduleLoadings = moduleLoadings,
|
||||
)
|
||||
}
|
||||
|
||||
private fun collectModulesWithLoadings(
|
||||
moduleSet: ModuleSet,
|
||||
modules: MutableSet<String>,
|
||||
loadings: MutableMap<String, ModuleLoadingRuleValue?>,
|
||||
) {
|
||||
for (module in moduleSet.modules) {
|
||||
modules.add(module.name)
|
||||
loadings[module.name] = module.loading
|
||||
}
|
||||
for (nestedSet in moduleSet.nestedSets) {
|
||||
collectModulesWithLoadings(nestedSet, modules, loadings)
|
||||
}
|
||||
}
|
||||
|
||||
// endregion
|
||||
|
||||
// region Formatting
|
||||
|
||||
internal fun formatValidationErrors(errors: List<ValidationError>): String {
|
||||
if (errors.isEmpty()) {
|
||||
return ""
|
||||
}
|
||||
|
||||
return buildString {
|
||||
for (error in errors) {
|
||||
when (error) {
|
||||
is SelfContainedValidationError -> formatSelfContainedError(this, error)
|
||||
is MissingModuleSetsError -> formatMissingModuleSetsError(this, error)
|
||||
is DuplicateModulesError -> formatDuplicateModulesError(this, error)
|
||||
is MissingDependenciesError -> formatMissingDependenciesError(this, error)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun formatSelfContainedError(sb: StringBuilder, error: SelfContainedValidationError) {
|
||||
sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Module set '${error.context}' is marked selfContained but has unresolvable dependencies${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
|
||||
for ((dep, needingModules) in error.missingDependencies.entries.sortedByDescending { it.value.size }) {
|
||||
sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$dep'${AnsiColors.RESET}")
|
||||
sb.appendLine(" Needed by: ${needingModules.sorted().joinToString(", ")}")
|
||||
}
|
||||
|
||||
sb.appendLine()
|
||||
sb.appendLine("${AnsiColors.YELLOW}To fix:${AnsiColors.RESET}")
|
||||
sb.appendLine("1. Add the missing modules/sets to '${error.context}' to make it truly self-contained")
|
||||
sb.appendLine("2. Or remove selfContained=true if this set is designed to compose with other sets")
|
||||
sb.appendLine()
|
||||
}
|
||||
|
||||
private fun formatMissingModuleSetsError(sb: StringBuilder, error: MissingModuleSetsError) {
|
||||
sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product '${error.context}' references non-existent module sets${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
for (setName in error.missingModuleSets.sorted()) {
|
||||
sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Module set '${AnsiColors.BOLD}$setName${AnsiColors.RESET}' does not exist")
|
||||
}
|
||||
sb.appendLine()
|
||||
sb.appendLine("${AnsiColors.BLUE}Fix: Remove the reference or define the module set${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
}
|
||||
|
||||
private fun formatDuplicateModulesError(sb: StringBuilder, error: DuplicateModulesError) {
|
||||
sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product '${error.context}' has duplicate content modules${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
sb.appendLine("${AnsiColors.YELLOW}Duplicated modules (appearing ${AnsiColors.BOLD}${error.duplicates.values.max()}${AnsiColors.RESET}${AnsiColors.YELLOW} times):${AnsiColors.RESET}")
|
||||
for ((moduleName, count) in error.duplicates.entries.sortedBy { it.key }) {
|
||||
sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} ${AnsiColors.BOLD}$moduleName${AnsiColors.RESET} (appears $count times)")
|
||||
}
|
||||
sb.appendLine()
|
||||
sb.appendLine("${AnsiColors.BLUE}This causes runtime error: \"Plugin has duplicated content modules declarations\"${AnsiColors.RESET}")
|
||||
sb.appendLine("${AnsiColors.BLUE}Fix: Remove duplicate moduleSet() nesting or redundant module() calls${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
}
|
||||
|
||||
private fun formatMissingDependenciesError(sb: StringBuilder, error: MissingDependenciesError) {
|
||||
sb.appendLine("${AnsiColors.BOLD}Product: ${error.context}${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
|
||||
// Group by missing dependency for clearer output
|
||||
val depToModules = mutableMapOf<String, MutableSet<String>>()
|
||||
for ((module, deps) in error.missingModules) {
|
||||
for (dep in deps) {
|
||||
depToModules.getOrPut(dep) { mutableSetOf() }.add(module)
|
||||
}
|
||||
}
|
||||
|
||||
for ((missingDep, needingModules) in depToModules.entries.sortedByDescending { it.value.size }) {
|
||||
sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$missingDep'${AnsiColors.RESET}")
|
||||
sb.appendLine(" Needed by: ${needingModules.sorted().take(5).joinToString(", ")}")
|
||||
if (needingModules.size > 5) {
|
||||
sb.appendLine(" ... and ${needingModules.size - 5} more modules")
|
||||
}
|
||||
|
||||
val containingSets = error.allModuleSets
|
||||
.filter { ModuleSetTraversal.containsModule(it, missingDep) }
|
||||
.map { it.name }
|
||||
|
||||
if (containingSets.isNotEmpty()) {
|
||||
sb.appendLine(" ${AnsiColors.BLUE}Suggestion:${AnsiColors.RESET} Add module set: ${containingSets.joinToString(" or ")}")
|
||||
}
|
||||
sb.appendLine()
|
||||
}
|
||||
}
|
||||
|
||||
internal fun formatProductDependencyErrorsHeader(sb: StringBuilder) {
|
||||
sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product-level validation failed: Unresolvable module dependencies${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
}
|
||||
|
||||
internal fun formatProductDependencyErrorsFooter(sb: StringBuilder) {
|
||||
sb.appendLine("${AnsiColors.BLUE}This will cause runtime errors: \"Plugin X has dependency on Y which is not installed\"${AnsiColors.RESET}")
|
||||
sb.appendLine()
|
||||
sb.appendLine("${AnsiColors.BOLD}To fix:${AnsiColors.RESET}")
|
||||
sb.appendLine("${AnsiColors.BLUE}1.${AnsiColors.RESET} Add the required module sets to the product's getProductContentDescriptor()")
|
||||
sb.appendLine("${AnsiColors.BLUE}2.${AnsiColors.RESET} Or add individual modules via module()/embeddedModule()")
|
||||
sb.appendLine("${AnsiColors.BLUE}3.${AnsiColors.RESET} Or add the product to allowUnresolvableProducts if this is intentional")
|
||||
}
|
||||
|
||||
// endregion
|
||||
|
||||
@@ -4,12 +4,16 @@
|
||||
package org.jetbrains.intellij.build.productLayout
|
||||
|
||||
import com.intellij.platform.plugins.parser.impl.elements.ModuleLoadingRuleValue
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.async
|
||||
import kotlinx.coroutines.awaitAll
|
||||
import kotlinx.coroutines.coroutineScope
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.serialization.SerialName
|
||||
import kotlinx.serialization.Serializable
|
||||
import org.jetbrains.intellij.build.ModuleOutputProvider
|
||||
import org.jetbrains.intellij.build.PLUGIN_XML_RELATIVE_PATH
|
||||
import org.jetbrains.intellij.build.findFileInModuleSources
|
||||
import org.jetbrains.jps.model.java.JavaSourceRootType
|
||||
import org.jetbrains.jps.model.java.JpsJavaExtensionService
|
||||
import java.nio.file.Files
|
||||
@@ -33,7 +37,7 @@ data class ProductConfigurationRegistry(@JvmField val products: Map<String, Prod
|
||||
data class ProductConfiguration(
|
||||
@JvmField val modules: List<String>,
|
||||
@JvmField @SerialName("class") val className: String,
|
||||
@JvmField val pluginXmlPath: String? = null
|
||||
@JvmField val pluginXmlPath: String? = null,
|
||||
)
|
||||
|
||||
/**
|
||||
@@ -76,7 +80,7 @@ fun findProductPropertiesSourceFile(
|
||||
buildModules: List<String>,
|
||||
productPropertiesClass: Class<*>,
|
||||
moduleOutputProvider: ModuleOutputProvider,
|
||||
projectRoot: Path
|
||||
projectRoot: Path,
|
||||
): String {
|
||||
val className = productPropertiesClass.name
|
||||
|
||||
@@ -105,17 +109,6 @@ fun findProductPropertiesSourceFile(
|
||||
throw IllegalStateException("Cannot find source file for $productPropertiesClass (searched for $relativePath in modules: $buildModules)")
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts product validation data from discovered products.
|
||||
* Returns list of (productName, ProductModulesContentSpec) pairs for validation.
|
||||
*
|
||||
* @param discoveredProducts List of discovered products
|
||||
* @return List of product name and spec pairs for validation
|
||||
*/
|
||||
fun extractProductsForValidation(discoveredProducts: List<DiscoveredProduct>): List<Pair<String, ProductModulesContentSpec?>> {
|
||||
return discoveredProducts.map { it.name to it.spec }
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates product XMLs for all products using programmatic content.
|
||||
* Takes discovered products and test product specs, then generates complete plugin.xml files.
|
||||
@@ -166,13 +159,13 @@ internal suspend fun generateAllProductXmlFiles(
|
||||
val spec = discovered.spec ?: return@async null
|
||||
|
||||
val pluginXmlPath = projectRoot.resolve(pluginXmlRelativePath)
|
||||
|
||||
|
||||
// Extract ProductProperties class name (works with both ProductProperties and null)
|
||||
val productPropertiesClass = when (val props = discovered.properties) {
|
||||
null -> "test-product"
|
||||
else -> props.javaClass.name
|
||||
}
|
||||
|
||||
|
||||
generateProductXml(
|
||||
pluginXmlPath = pluginXmlPath,
|
||||
spec = spec,
|
||||
@@ -231,7 +224,7 @@ private fun aggregateAndCleanupOrphanedFiles(moduleSetResults: List<ModuleSetGen
|
||||
aggregatedTrackingMap.computeIfAbsent(dir) { mutableSetOf() }.addAll(files)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
val deletedFiles = cleanupOrphanedModuleSetFiles(aggregatedTrackingMap)
|
||||
if (deletedFiles.isNotEmpty()) {
|
||||
println("\nDeleted ${deletedFiles.size} orphaned files")
|
||||
@@ -257,7 +250,7 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig)
|
||||
|
||||
// Discover all module sets and validate products
|
||||
val allModuleSets = discoverAllModuleSets(config.moduleSetSources)
|
||||
val products = extractProductsForValidation(config.discoveredProducts)
|
||||
val products = config.discoveredProducts.map { it.name to it.spec }
|
||||
validateNoRedundantModuleSets(allModuleSets = allModuleSets, productSpecs = products)
|
||||
|
||||
// Execute all generation operations in parallel
|
||||
@@ -280,6 +273,16 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig)
|
||||
}
|
||||
}
|
||||
|
||||
// Collect all bundled plugins and launch content extraction jobs ONCE
|
||||
// multiple consumers can await these Deferred values (validation + plugin dep gen)
|
||||
val allBundledPlugins = (config.discoveredProducts.asSequence().mapNotNull { it.spec?.bundledPlugins }.flatten() + config.additionalPlugins)
|
||||
.distinct()
|
||||
.toList()
|
||||
|
||||
val pluginContentJobs = allBundledPlugins.associateWith { pluginName ->
|
||||
async { extractPluginContent(pluginName, config.moduleOutputProvider) }
|
||||
}
|
||||
|
||||
// TIER 2: Parallel dependency and product generation (can run concurrently with TIER 1)
|
||||
val dependencyJob = async {
|
||||
val moduleSetsByLabel = config.moduleSetSources.mapValues { (_, source) ->
|
||||
@@ -293,21 +296,12 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig)
|
||||
coreModuleSets = moduleSetsByLabel.get("core") ?: emptyList(),
|
||||
moduleOutputProvider = config.moduleOutputProvider,
|
||||
productSpecs = products,
|
||||
embeddedModulesDeferred = embeddedModulesDeferred,
|
||||
pluginContentJobs = pluginContentJobs,
|
||||
)
|
||||
}
|
||||
|
||||
// TIER 3: Plugin dependency generation for bundled plugins
|
||||
val pluginDependencyJob = async {
|
||||
val allBundledPlugins = (
|
||||
config.discoveredProducts
|
||||
.asSequence()
|
||||
.mapNotNull { it.spec?.bundledPlugins }
|
||||
.flatten() + config.additionalPlugins
|
||||
)
|
||||
.distinct()
|
||||
.toList()
|
||||
|
||||
if (allBundledPlugins.isEmpty()) {
|
||||
null
|
||||
}
|
||||
@@ -322,12 +316,7 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig)
|
||||
false
|
||||
}
|
||||
}
|
||||
generatePluginDependencies(
|
||||
plugins = allBundledPlugins,
|
||||
moduleOutputProvider = config.moduleOutputProvider,
|
||||
descriptorCache = cache,
|
||||
dependencyFilter = dependencyFilter,
|
||||
)
|
||||
generatePluginDependencies(plugins = allBundledPlugins, pluginContentJobs = pluginContentJobs, descriptorCache = cache, dependencyFilter = dependencyFilter)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -360,6 +349,38 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig)
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of extracting plugin content from plugin.xml.
|
||||
* Contains everything needed for both validation and dependency generation.
|
||||
*/
|
||||
data class PluginContentInfo(
|
||||
@JvmField val pluginXmlPath: Path,
|
||||
@JvmField val pluginXmlContent: String,
|
||||
@JvmField val contentModules: Set<String>,
|
||||
/** Lazy JPS production dependencies - only called by plugin dep gen, not validation */
|
||||
@JvmField val jpsDependencies: () -> List<String>,
|
||||
)
|
||||
|
||||
/**
|
||||
* Extracts content modules from a plugin's plugin.xml.
|
||||
* Returns null if plugin.xml not found or has module references with '/'.
|
||||
*/
|
||||
private suspend fun extractPluginContent(
|
||||
pluginName: String,
|
||||
moduleOutputProvider: ModuleOutputProvider,
|
||||
): PluginContentInfo? {
|
||||
val jpsModule = moduleOutputProvider.findModule(pluginName) ?: return null
|
||||
val pluginXmlPath = findFileInModuleSources(module = jpsModule, relativePath = PLUGIN_XML_RELATIVE_PATH, onlyProductionSources = true) ?: return null
|
||||
val content = withContext(Dispatchers.IO) { Files.readString(pluginXmlPath) }
|
||||
val contentModules = extractContentModulesFromText(content) ?: return null
|
||||
return PluginContentInfo(
|
||||
pluginXmlPath = pluginXmlPath,
|
||||
pluginXmlContent = content,
|
||||
contentModules = contentModules,
|
||||
jpsDependencies = { jpsModule.getProductionModuleDependencies(withTests = false).map { it.moduleReference.moduleName }.toList() },
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects all embedded modules from all product specs.
|
||||
* Used to filter out embedded platform modules from plugin dependencies.
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
# Plugin Model Validation
|
||||
|
||||
This document describes the validation system that ensures module dependencies are resolvable at build time, preventing runtime errors.
|
||||
|
||||
## Overview
|
||||
|
||||
### Why Validation Exists
|
||||
|
||||
Without validation, dependency errors only surface at runtime:
|
||||
```
|
||||
Plugin 'Java' has dependency on 'com.intellij.modules.vcs' which is not installed
|
||||
```
|
||||
|
||||
The validation system catches these errors during `Generate Product Layouts`, providing:
|
||||
- Early detection of missing dependencies
|
||||
- Clear error messages with suggested fixes
|
||||
- Module set hierarchy awareness
|
||||
- Cross-plugin dependency support
|
||||
|
||||
### When Validation Runs
|
||||
|
||||
Validation runs automatically when you execute:
|
||||
- **IDE**: Run configuration "Generate Product Layouts"
|
||||
- **CLI**: `UltimateModuleSets.main()` or `CommunityModuleSets.main()`
|
||||
- **Bazel**: `bazel run //platform/buildScripts:product-model-tool`
|
||||
|
||||
## Two-Tier Validation System
|
||||
|
||||
The build system uses a **two-tier validation approach** to ensure module dependencies are resolvable while avoiding false positives.
|
||||
|
||||
### Tier 1: Product-Level Validation
|
||||
|
||||
**What it validates**: All products with their complete module composition
|
||||
|
||||
**How it works**:
|
||||
- Collects all modules from a product's complete module set hierarchy
|
||||
- Validates that each module's dependencies are available within that product context
|
||||
- Includes bundled plugin content modules in available modules
|
||||
- Reports errors per-product with clear affected modules
|
||||
|
||||
**Why this tier exists**: Products are the actual deployment units. Most module sets (like `debugger()`, `vcs()`, `xml()`) are **composable building blocks** designed to work together. Validating them in isolation would produce false positives because they intentionally depend on modules from other sets.
|
||||
|
||||
**Example**:
|
||||
- Product `GoLand` uses: `ide.ultimate`, `ssh`, `rd.common`
|
||||
- Validation checks: Can all modules in `ide.ultimate` + `ssh` + `rd.common` resolve their dependencies within this combined set?
|
||||
|
||||
### Tier 2: Self-Contained Module Set Validation
|
||||
|
||||
**What it validates**: Module sets marked with `selfContained = true`
|
||||
|
||||
**How it works**:
|
||||
- Validates the module set in isolation without considering other module sets
|
||||
- Flattens the entire hierarchy (nested sets all see each other)
|
||||
- Ensures all dependencies are resolvable within the set itself
|
||||
- Reports errors specific to that module set
|
||||
|
||||
**Why this tier exists**: Some module sets are designed to be **standalone/self-contained** and used directly by products without composition with other sets. These sets must have all their dependencies available internally.
|
||||
|
||||
**When to use `selfContained = true`**:
|
||||
- Module set is used directly by products as the primary/only module set
|
||||
- Module set represents a complete, independent runtime environment
|
||||
- You want to enforce that the set doesn't leak dependencies on external sets
|
||||
|
||||
**Example**: `core.platform` is self-contained because CodeServer uses it alone.
|
||||
|
||||
```kotlin
|
||||
fun corePlatform(): ModuleSet = moduleSet(
|
||||
name = "core.platform",
|
||||
selfContained = true, // Must be resolvable in isolation
|
||||
outputModule = "intellij.platform.ide.core"
|
||||
) {
|
||||
moduleSet(librariesPlatform())
|
||||
moduleSet(rpcMinimal())
|
||||
embeddedModule("intellij.platform.core", includeDependencies = true)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-Plugin Dependency Validation
|
||||
|
||||
### The Problem
|
||||
|
||||
Bundled plugins can have content modules that optionally depend on modules from non-bundled plugins. For example:
|
||||
|
||||
- `intellij.fullLine.cpp` (bundled in IDEA Ultimate) depends on `intellij.c.core`
|
||||
- `intellij.c.core` is part of the C/C++ plugin (`intellij.c`), which is **not bundled**
|
||||
|
||||
Without special handling, this triggers a validation error because `intellij.c.core` is not in the product's module sets.
|
||||
|
||||
### Loading Attribute Semantics
|
||||
|
||||
The solution depends on the content module's **loading attribute**:
|
||||
|
||||
| Loading Value | Meaning | Cross-Plugin Deps Allowed? |
|
||||
|---------------|---------|---------------------------|
|
||||
| `embedded` | Core module, loaded into main classloader | **No** - must have all deps in product |
|
||||
| `required` | Required at startup, loaded early | **No** - must have all deps in product |
|
||||
| `optional` | Loaded on demand if deps available | **Yes** - can depend on non-bundled plugins |
|
||||
| `on_demand` | Loaded only when explicitly requested | **Yes** - can depend on non-bundled plugins |
|
||||
| (unspecified) | Default loading behavior | **Yes** - can depend on non-bundled plugins |
|
||||
|
||||
**Key insight**: Critical modules (`embedded`/`required`) cannot depend on non-bundled plugins because those plugins may not be installed. Non-critical modules can safely have cross-plugin dependencies since they won't prevent IDE startup if the dependency is missing.
|
||||
|
||||
### Configuration
|
||||
|
||||
Non-bundled plugins for validation are configured in `UltimateModuleSets.kt`:
|
||||
|
||||
```kotlin
|
||||
/**
|
||||
* Non-bundled plugins whose content modules should be available for cross-plugin
|
||||
* dependency validation. These are plugins that bundled plugins may optionally depend on.
|
||||
*
|
||||
* When a bundled plugin's content module (with non-critical loading like optional/on_demand)
|
||||
* depends on a module from these plugins, the dependency is considered valid.
|
||||
*/
|
||||
val NON_BUNDLED_PLUGINS_FOR_VALIDATION: List<String> = listOf(
|
||||
"intellij.c", // C/C++ support - needed by intellij.fullLine.cpp
|
||||
)
|
||||
```
|
||||
|
||||
### Adding a New Non-Bundled Plugin
|
||||
|
||||
When you encounter a validation error for a cross-plugin dependency:
|
||||
|
||||
1. **Verify the source module is NOT critical** (not `embedded` or `required`)
|
||||
2. **Find the plugin module name** (not the plugin ID):
|
||||
```bash
|
||||
# Find the .iml file for the plugin
|
||||
find . -name "*.iml" | xargs grep -l "plugin-descriptor-xml"
|
||||
```
|
||||
3. **Add to the list**:
|
||||
```kotlin
|
||||
val NON_BUNDLED_PLUGINS_FOR_VALIDATION: List<String> = listOf(
|
||||
"intellij.c",
|
||||
"intellij.new.plugin", // Description of why it's needed
|
||||
)
|
||||
```
|
||||
|
||||
### How It Works Internally
|
||||
|
||||
1. **Plugin content extraction**: The generator uses `extractPluginContent()` to read each plugin's content modules from its `plugin.xml`
|
||||
|
||||
2. **All plugin modules collection**: At validation start, all plugin content modules (bundled + non-bundled from the config) are collected:
|
||||
```kotlin
|
||||
val allPluginModules = mutableSetOf<String>()
|
||||
for ((_, job) in pluginContentJobs) {
|
||||
val info = job.await()
|
||||
if (info != null) {
|
||||
allPluginModules.addAll(info.contentModules)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **Validation with loading check**: For each module's dependencies:
|
||||
```kotlin
|
||||
val loading = productIndex.moduleLoadings[moduleName]
|
||||
val isCritical = loading == ModuleLoadingRuleValue.EMBEDDED ||
|
||||
loading == ModuleLoadingRuleValue.REQUIRED
|
||||
|
||||
if (dep in reachableModules) {
|
||||
// Present in product - valid
|
||||
} else if (dep in allowedMissing) {
|
||||
// Explicitly allowed - skip
|
||||
} else if (dep in allPluginModules && !isCritical) {
|
||||
// Exists in another plugin AND source is NOT critical - valid
|
||||
} else {
|
||||
// Missing dependency - report error
|
||||
}
|
||||
```
|
||||
|
||||
## allowMissingDependencies
|
||||
|
||||
### What It Is
|
||||
|
||||
Products can specify modules that are allowed to be missing from validation:
|
||||
|
||||
```kotlin
|
||||
override fun getProductContentModules(): ProductModulesContentSpec = productModules {
|
||||
moduleSet(CommunityModuleSets.essential())
|
||||
// ...
|
||||
|
||||
allowMissingDependencies = setOf(
|
||||
"intellij.some.optional.module"
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### When to Use
|
||||
|
||||
Use `allowMissingDependencies` sparingly for:
|
||||
- **Modules provided by bundled plugins** that aren't declared as content modules
|
||||
- **Temporary allowances** during migration (with a TODO to fix)
|
||||
- **Platform quirks** where a dependency exists at runtime but not in module model
|
||||
|
||||
### When NOT to Use
|
||||
|
||||
Do NOT use for:
|
||||
- **Cross-plugin dependencies** - use `NON_BUNDLED_PLUGINS_FOR_VALIDATION` instead
|
||||
- **Avoiding validation errors** - fix the root cause
|
||||
- **Optional features** - use proper loading attributes instead
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Document each entry**: Add a comment explaining why it's needed
|
||||
2. **Review periodically**: Remove entries that are no longer needed
|
||||
3. **Prefer proper fixes**: `allowMissingDependencies` is a workaround, not a solution
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Errors
|
||||
|
||||
#### 1. Missing Dependency in Product
|
||||
|
||||
```
|
||||
❌ Product-level validation failed: Unresolvable module dependencies
|
||||
|
||||
Product: GoLand
|
||||
|
||||
✗ Missing: 'intellij.platform.polySymbols'
|
||||
Needed by: intellij.platform.vcs.impl
|
||||
Suggestion: Add module set: symbols
|
||||
```
|
||||
|
||||
**Cause**: A module in the product depends on another module not available in the product's composition.
|
||||
|
||||
**Fix options**:
|
||||
1. Add the suggested module set to the product
|
||||
2. Add the individual module via `module()` or `embeddedModule()`
|
||||
3. If it's a cross-plugin dependency with non-critical loading, add the plugin to `NON_BUNDLED_PLUGINS_FOR_VALIDATION`
|
||||
|
||||
#### 2. Self-Contained Set with Unresolvable Dependencies
|
||||
|
||||
```
|
||||
❌ Module set 'core.platform' is marked selfContained but has unresolvable dependencies
|
||||
|
||||
✗ Missing: 'fleet.kernel'
|
||||
Needed by: intellij.platform.kernel
|
||||
|
||||
💡 To fix:
|
||||
1. Add the missing modules/sets to 'core.platform' to make it truly self-contained
|
||||
2. Or remove selfContained=true if this set is designed to compose with other sets
|
||||
```
|
||||
|
||||
**Cause**: A self-contained module set doesn't have all its dependencies internally.
|
||||
|
||||
**Fix options**:
|
||||
1. Add the missing module/module set
|
||||
2. Remove `selfContained = true` if the set is meant to be composed with others
|
||||
|
||||
#### 3. Duplicate Content Modules
|
||||
|
||||
```
|
||||
❌ Product 'GoLand' has duplicate content modules
|
||||
|
||||
Duplicated modules (appearing 2 times):
|
||||
✗ intellij.platform.vcs.impl (appears 2 times)
|
||||
|
||||
💡 This causes runtime error: "Plugin has duplicated content modules declarations"
|
||||
Fix: Remove duplicate moduleSet() nesting or redundant module() calls
|
||||
```
|
||||
|
||||
**Cause**: The same module appears twice in the product's content.
|
||||
|
||||
**Fix**: Remove the duplicate - either from a module set or from direct `module()` calls.
|
||||
|
||||
### Investigation Tools
|
||||
|
||||
Use the Plugin Model Analyzer MCP for investigation:
|
||||
|
||||
```kotlin
|
||||
// Find dependency path between modules
|
||||
mcp__PluginModelAnalyzer__find_dependency_path(
|
||||
fromModule = "intellij.platform.vcs.impl",
|
||||
toModule = "intellij.c.core"
|
||||
)
|
||||
|
||||
// Check which module sets contain a dependency
|
||||
mcp__PluginModelAnalyzer__suggest_module_set_for_modules(
|
||||
moduleNames = ["intellij.c.core"]
|
||||
)
|
||||
|
||||
// Check module reachability within a module set
|
||||
mcp__PluginModelAnalyzer__check_module_reachability(
|
||||
moduleName = "intellij.platform.kernel",
|
||||
moduleSetName = "core.platform"
|
||||
)
|
||||
|
||||
// Get module info including which products/sets include it
|
||||
mcp__PluginModelAnalyzer__get_module_info(
|
||||
moduleName = "intellij.fullLine.cpp"
|
||||
)
|
||||
```
|
||||
|
||||
### Investigation Strategy
|
||||
|
||||
When you encounter a validation error:
|
||||
|
||||
1. **Identify the dependency chain**: Use `find_dependency_path` to understand why the dependency is needed
|
||||
|
||||
2. **Check if cross-plugin dependency**: Is the missing module in a non-bundled plugin? If so, check if the source module's loading allows cross-plugin deps.
|
||||
|
||||
3. **Find the right fix level**:
|
||||
- Missing module set in hierarchy → add the nested set
|
||||
- Product missing required set → add to product
|
||||
- Cross-plugin dep with non-critical loading → add to `NON_BUNDLED_PLUGINS_FOR_VALIDATION`
|
||||
- Truly missing infrastructure → add to appropriate module set
|
||||
|
||||
4. **Verify the fix**: Run "Generate Product Layouts" again
|
||||
|
||||
## See Also
|
||||
|
||||
- [Module Sets Documentation](module-sets.md) - How module sets work
|
||||
- [Programmatic Content](programmatic-content.md) - Product content descriptors
|
||||
- `DependencyValidation.kt` - Validation implementation
|
||||
- `UltimateModuleSets.kt` - Non-bundled plugins configuration
|
||||
@@ -32,6 +32,18 @@ fun isCommunityModule(module: JpsModule, context: BuildContext): Boolean {
|
||||
}
|
||||
}
|
||||
|
||||
val knownMissingModuleDependencies: List<String> = listOf(
|
||||
// todo not included into any plugin - investigate why and fix
|
||||
"intellij.javaee.jpa",
|
||||
"intellij.rider.plugins.fsharp",
|
||||
// conditional xi-include
|
||||
"kotlin.base.scripting.k1",
|
||||
// todo special module (make it not special)
|
||||
"intellij.platform.commercial.verifier",
|
||||
// included using `withModule`
|
||||
"intellij.python.frontend"
|
||||
)
|
||||
|
||||
/**
|
||||
* Describes a distribution of an IntelliJ-based IDE hosted in the IntelliJ repository.
|
||||
*/
|
||||
|
||||
+8
-5
@@ -1,9 +1,12 @@
|
||||
<idea-plugin>
|
||||
<dependencies>
|
||||
<module name="intellij.platform.compose"/>
|
||||
|
||||
<module name="intellij.platform.jewel.markdown.core"/>
|
||||
<module name="intellij.platform.jewel.markdown.ideLafBridgeStyling"/>
|
||||
<module name="intellij.platform.jewel.markdown.extensions.images"/>
|
||||
<module name="intellij.platform.compose"/>
|
||||
<module name="intellij.platform.jewel.markdown.core"/>
|
||||
<module name="intellij.platform.jewel.markdown.ideLafBridgeStyling"/>
|
||||
<module name="intellij.platform.jewel.markdown.extensions.images"/>
|
||||
<!-- editor-fold desc="Generated dependencies - do not edit manually" -->
|
||||
<module name="intellij.libraries.compose.foundation.desktop"/>
|
||||
<module name="intellij.libraries.skiko"/>
|
||||
<!-- end editor-fold -->
|
||||
</dependencies>
|
||||
</idea-plugin>
|
||||
|
||||
@@ -7,6 +7,9 @@
|
||||
<module name="intellij.properties.backend"/>
|
||||
<module name="intellij.properties.backend.psi"/>
|
||||
<module name="intellij.libraries.xerces"/>
|
||||
<!-- editor-fold desc="Generated dependencies - do not edit manually" -->
|
||||
<module name="intellij.java.aetherDependencyResolver"/>
|
||||
<!-- end editor-fold -->
|
||||
</dependencies>
|
||||
|
||||
<resource-bundle>messages.DevKitBundle</resource-bundle>
|
||||
|
||||
+3
@@ -4,6 +4,9 @@
|
||||
<plugin id="com.intellij.java"/>
|
||||
<plugin id="org.jetbrains.kotlin"/>
|
||||
<module name="intellij.devkit.core"/>
|
||||
<!-- editor-fold desc="Generated dependencies - do not edit manually" -->
|
||||
<module name="intellij.java.aetherDependencyResolver"/>
|
||||
<!-- end editor-fold -->
|
||||
</dependencies>
|
||||
|
||||
<resource-bundle>messages.DevKitWorkspaceModelBundle</resource-bundle>
|
||||
|
||||
@@ -71,6 +71,9 @@ open class PyCharmCommunityProperties(protected val communityHome: Path) : PyCha
|
||||
// Static includes
|
||||
deprecatedInclude("intellij.platform.extended.community.impl", "META-INF/community-extensions.xml", ultimateOnly = true)
|
||||
deprecatedInclude("intellij.pycharm.community", "META-INF/pycharm-core-customization.xml")
|
||||
|
||||
allowMissingDependencies(knownMissingModuleDependencies)
|
||||
bundledPlugins(productLayout.bundledPluginModules.toList())
|
||||
}
|
||||
|
||||
override suspend fun copyAdditionalFiles(targetDir: Path, context: BuildContext) {
|
||||
|
||||
Reference in New Issue
Block a user