diff --git a/platform/build-scripts/product-dsl/module-sets.md b/platform/build-scripts/product-dsl/module-sets.md index d55d3cd31551..4345c96dd495 100644 --- a/platform/build-scripts/product-dsl/module-sets.md +++ b/platform/build-scripts/product-dsl/module-sets.md @@ -169,182 +169,9 @@ But does NOT duplicate these content modules (already in `fleetMinimal()`): ## Module Set Validation -### Two-Tier Validation System - -The build system uses a **two-tier validation approach** to ensure module dependencies are resolvable while avoiding false positives: - -#### Tier 1: Product-Level Validation - -**What it validates**: All products with their complete module composition - -**How it works**: -- Collects all modules from a product's complete module set hierarchy -- Validates that each module's dependencies are available within that product context -- Reports errors per-product with clear affected modules - -**Why this tier exists**: Products are the actual deployment units. Most module sets (like `debugger()`, `vcs()`, `xml()`) are **composable building blocks** designed to work together. Validating them in isolation would produce false positives because they intentionally depend on modules from other sets (e.g., `debugger()` depends on `intellij.platform.core` from `essential()`). - -**Example**: -- Product `GoLand` uses: `ide.ultimate`, `ssh`, `rd.common` -- Validation checks: Can all modules in `ide.ultimate` + `ssh` + `rd.common` resolve their dependencies within this combined set? - -#### Tier 2: Self-Contained Validation - -**What it validates**: Module sets marked with `selfContained = true` - -**How it works**: -- Validates the module set in isolation without considering other module sets -- Ensures all dependencies are resolvable within the set itself -- Reports errors specific to that module set - -**Why this tier exists**: Some module sets are designed to be **standalone/self-contained** and used directly by products without composition with other sets. These sets must have all their dependencies available internally. - -**When to use `selfContained = true`**: -- Module set is used directly by products as the primary/only module set -- Module set represents a complete, independent runtime environment -- You want to enforce that the set doesn't leak dependencies on external sets - -**When NOT to use `selfContained = true`**: -- Module set is designed to be composed with other sets (debugger, vcs, xml, ssh) -- Module set intentionally depends on modules from base sets like `essential()` -- Module set is a specialized feature addition to a larger base - -### Example: core.platform (Self-Contained) - -```kotlin -/** - * Core platform modules without IDE or language support. - * Used by: CodeServer (analysis tool) - */ -fun corePlatform(): ModuleSet = moduleSet( - name = "core.platform", - selfContained = true, // ✅ Must be resolvable in isolation - outputModule = "intellij.platform.ide.core" -) { - moduleSet(librariesPlatform()) - moduleSet(rpcMinimal()) // Provides kernel + fleet deps - - embeddedModule("intellij.platform.core", includeDependencies = true) - embeddedModule("intellij.platform.ide.core", includeDependencies = true) - // ... -} -``` - -**Why selfContained**: CodeServer uses `core.platform` alone without other module sets. It must contain everything needed for the platform runtime. - -### Example: debugger() (Composable, Not Self-Contained) - -```kotlin -/** - * Debugger platform modules. - * Used by: All IDE products via essential() - */ -fun debugger(): ModuleSet = moduleSet( - name = "debugger" - // ❌ NOT selfContained - designed to compose with essential() -) { - module("intellij.platform.debugger.impl.backend") - embeddedModule("intellij.platform.debugger") - // Depends on intellij.platform.core from essential() ✅ OK -} -``` - -**Why NOT selfContained**: Debugger is always used together with `essential()` which provides core platform modules. Validating debugger in isolation would fail because it depends on `intellij.platform.core`, but that's correct by design. - -### Validation Error Examples - -#### Product-Level Error - -``` -❌ Unresolvable dependencies in products - - Product: GoLand - ✗ Missing: 'intellij.platform.polySymbols' - Needed by: intellij.platform.vcs.impl - Chain: intellij.platform.vcs.impl → intellij.platform.polySymbols -``` - -**What this means**: The product `GoLand` is missing a module that one of its included modules depends on. Fix by adding the missing module or a module set that contains it. - -#### Self-Contained Error - -``` -❌ Module set 'core.platform' is marked selfContained but has unresolvable dependencies - - ✗ Missing: 'fleet.kernel' - Needed by: intellij.platform.kernel - Suggestion: Include fleet() or add fleet.kernel directly -``` - -**What this means**: A self-contained module set is missing a dependency. Fix by adding the missing module/set to make it truly self-contained. - -### Troubleshooting Dependency Errors - -When you encounter a dependency validation error, **resist the temptation to simply add the missing module to the failing module set**. This creates technical debt and breaks the isolation principle. - -#### ❌ Wrong Approach - -```kotlin -// BAD: Adding a library directly to debugger module set -fun debugger(): ModuleSet = moduleSet("debugger") { - module("intellij.platform.debugger.impl") - embeddedModule("intellij.libraries.kotlinx.serialization.core") // ❌ Wrong! -} -``` - -**Why it's wrong**: The debugger module set should contain only debugger-related modules. Adding serialization libraries pollutes its domain and creates maintenance burden. - -#### ✅ Correct Approach - -1. **Identify the dependency chain**: Use PMA MCP to trace dependencies - ``` - mcp__PluginModelAnalyzer__find_dependency_path( - fromModule="intellij.platform.debugger.impl", - toModule="intellij.libraries.kotlinx.serialization.core" - ) - ``` - -2. **Find which module set provides the dependency**: Check existing module sets - ``` - mcp__PluginModelAnalyzer__suggest_module_set_for_modules( - moduleNames=["intellij.libraries.kotlinx.serialization.core"] - ) - ``` - -3. **Analyze the product's module set hierarchy**: Trace what the product includes - - Does `essentialMinimal()` → `coreLang()` → `coreIde()` → `corePlatform()` → `librariesPlatform()` include it? - - Is there a missing link in the chain? - -4. **Fix at the appropriate level**: - - If a base module set is missing a nested set → add the nested set - - If a product is missing a required module set → add to product - - If the module truly belongs in the domain → then add it directly (rare) - -#### Example: Serialization Library Missing - -**Error**: `debugger` module set missing `intellij.libraries.kotlinx.serialization.core` - -**Investigation**: -- `kotlinx.serialization.core` is in `librariesPlatform()` module set -- `librariesPlatform()` is nested in `corePlatform()` -- `corePlatform()` is nested in `coreIde()` → `coreLang()` → `essentialMinimal()` -- Product uses `essentialMinimal()` ✅ so it should have the library - -**Root cause options**: -1. Product doesn't include `essentialMinimal()` - add it -2. The chain is broken somewhere - fix the nesting -3. Validation logic has a bug - investigate the validator - -**Key insight**: The fix is rarely "add module X to module set Y". It's usually "ensure the module set hierarchy correctly provides module X through proper nesting". - -### Best Practices - -1. **Start without `selfContained`**: Most module sets should NOT be self-contained. Only flag sets that truly need isolation. - -2. **Keep self-contained sets minimal**: If a set needs `selfContained = true`, try to minimize dependencies to keep it lightweight. - -3. **Document why self-contained**: When using `selfContained = true`, add a KDoc comment explaining why this set needs isolation. - -4. **Use product-level validation as guide**: If validation suggests adding many dependencies to a self-contained set, consider whether it should really be self-contained. - -5. **Test with real products**: After changes, run "Generate Product Layouts" to validate both tiers work correctly. \ No newline at end of file +See [Validation Documentation](validation.md) for comprehensive coverage of: +- Two-tier validation system (product-level and self-contained) +- Cross-plugin dependency validation +- Loading attribute semantics (`embedded`/`required` vs `optional`/`on_demand`) +- `allowMissingDependencies` usage +- Troubleshooting validation errors \ No newline at end of file diff --git a/platform/build-scripts/product-dsl/programmatic-content.md b/platform/build-scripts/product-dsl/programmatic-content.md index f03651fc19bc..c22f877a47e9 100644 --- a/platform/build-scripts/product-dsl/programmatic-content.md +++ b/platform/build-scripts/product-dsl/programmatic-content.md @@ -767,7 +767,8 @@ Before committing changes: ## See Also -- [Module Sets Documentation](module-sets.md) +- [Module Sets Documentation](module-sets.md) - How module sets work and composition +- [Validation Documentation](validation.md) - Dependency validation and troubleshooting - `ProductModulesContentSpec` class documentation - `ModuleSet` and `ContentModule` classes - Example: `GatewayProperties.getProductContentModules()` \ No newline at end of file diff --git a/platform/build-scripts/product-dsl/src/ModuleDescriptorDependencyGenerator.kt b/platform/build-scripts/product-dsl/src/ModuleDescriptorDependencyGenerator.kt index 629dba9fcdeb..bc932c9cdedb 100644 --- a/platform/build-scripts/product-dsl/src/ModuleDescriptorDependencyGenerator.kt +++ b/platform/build-scripts/product-dsl/src/ModuleDescriptorDependencyGenerator.kt @@ -9,6 +9,11 @@ import kotlinx.coroutines.awaitAll import kotlinx.coroutines.coroutineScope import org.jetbrains.intellij.build.ModuleOutputProvider import org.jetbrains.intellij.build.findFileInModuleSources +import org.jetbrains.intellij.build.productLayout.analysis.MissingDependenciesError +import org.jetbrains.intellij.build.productLayout.analysis.ValidationError +import org.jetbrains.intellij.build.productLayout.analysis.formatProductDependencyErrorsFooter +import org.jetbrains.intellij.build.productLayout.analysis.formatProductDependencyErrorsHeader +import org.jetbrains.intellij.build.productLayout.analysis.formatValidationErrors import org.jetbrains.intellij.build.productLayout.analysis.validateProductModuleSets import org.jetbrains.intellij.build.productLayout.analysis.validateSelfContainedModuleSets import java.nio.file.Files @@ -45,10 +50,9 @@ internal suspend fun generateModuleDescriptorDependencies( coreModuleSets: List = emptyList(), moduleOutputProvider: ModuleOutputProvider, productSpecs: List> = emptyList(), - embeddedModulesDeferred: Deferred>? = null, + pluginContentJobs: Map> = emptyMap(), ): DependencyGenerationResult = coroutineScope { val allModuleSets = communityModuleSets + coreModuleSets + ultimateModuleSets - val embeddedModules = embeddedModulesDeferred?.await() ?: emptySet() val modulesToProcess = collectModulesToProcess(allModuleSets) if (modulesToProcess.isEmpty()) { return@coroutineScope DependencyGenerationResult(emptyList()) @@ -56,18 +60,33 @@ internal suspend fun generateModuleDescriptorDependencies( val cache = ModuleDescriptorCache(moduleOutputProvider) + // Collect all validation errors + val errors = mutableListOf() + // Validate self-contained module sets in isolation - // Module sets marked with selfContained=true must be resolvable without other sets - validateSelfContainedModuleSets(allModuleSets, cache) + errors.addAll(validateSelfContainedModuleSets(allModuleSets, cache)) // Tier 2: Validate product-level dependencies - // This ensures all products can load without missing dependency errors - validateProductModuleSets( + errors.addAll(validateProductModuleSets( allModuleSets = allModuleSets, productSpecs = productSpecs, descriptorCache = cache, - precomputedEmbeddedModules = embeddedModules, - ) + pluginContentJobs = pluginContentJobs, + )) + + // Report all errors at once + if (errors.isNotEmpty()) { + val hasMissingDependencies = errors.any { it is MissingDependenciesError } + error(buildString { + if (hasMissingDependencies) { + formatProductDependencyErrorsHeader(this) + } + append(formatValidationErrors(errors)) + if (hasMissingDependencies) { + formatProductDependencyErrorsFooter(this) + } + }) + } // Write XML files in parallel val results = modulesToProcess.map { moduleName -> diff --git a/platform/build-scripts/product-dsl/src/PluginDependencyGenerator.kt b/platform/build-scripts/product-dsl/src/PluginDependencyGenerator.kt index e47f95486cf8..16bf00fbfd3a 100644 --- a/platform/build-scripts/product-dsl/src/PluginDependencyGenerator.kt +++ b/platform/build-scripts/product-dsl/src/PluginDependencyGenerator.kt @@ -1,12 +1,12 @@ // Copyright 2000-2025 JetBrains s.r.o. and contributors. Use of this source code is governed by the Apache 2.0 license. +@file:Suppress("ReplaceGetOrSet") + package org.jetbrains.intellij.build.productLayout +import kotlinx.coroutines.Deferred import kotlinx.coroutines.async import kotlinx.coroutines.awaitAll import kotlinx.coroutines.coroutineScope -import org.jetbrains.intellij.build.ModuleOutputProvider -import org.jetbrains.intellij.build.PLUGIN_XML_RELATIVE_PATH -import org.jetbrains.intellij.build.findFileInModuleSources import java.nio.file.Files /** @@ -14,13 +14,16 @@ import java.nio.file.Files * Uses same logic as ModuleDescriptorDependencyGenerator but for plugin.xml files. * * For each bundled plugin module: - * 1. Finds META-INF/plugin.xml in module sources - * 2. Gets JPS production dependencies that have XML descriptors (content modules) + * 1. Uses pre-extracted content from shared jobs (path, content, JPS deps - avoids duplicate lookups) + * 2. Filters JPS production dependencies to those with XML descriptors * 3. Updates the `` section with generated `` entries + * + * @param pluginContentJobs Pre-launched async jobs containing all plugin info. + * Multiple consumers can await the same Deferred - extraction runs only once per plugin. */ internal suspend fun generatePluginDependencies( plugins: List, - moduleOutputProvider: ModuleOutputProvider, + pluginContentJobs: Map>, descriptorCache: ModuleDescriptorCache, dependencyFilter: (String) -> Boolean, ): PluginDependencyGenerationResult = coroutineScope { @@ -32,7 +35,7 @@ internal suspend fun generatePluginDependencies( async { generatePluginDependency( pluginModuleName = pluginModuleName, - moduleOutputProvider = moduleOutputProvider, + pluginContentJobs = pluginContentJobs, descriptorCache = descriptorCache, dependencyFilter = dependencyFilter, ) @@ -45,49 +48,38 @@ internal suspend fun generatePluginDependencies( /** * Generates dependencies for a single plugin module. * - * @return PluginDependencyFileResult or null if plugin.xml not found + * @param pluginContentJobs Pre-launched async jobs containing all plugin info. + * @return PluginDependencyFileResult or null if plugin.xml not found or has module refs with '/' */ -private fun generatePluginDependency( +private suspend fun generatePluginDependency( pluginModuleName: String, - moduleOutputProvider: ModuleOutputProvider, + pluginContentJobs: Map>, descriptorCache: ModuleDescriptorCache, dependencyFilter: (String) -> Boolean, ): PluginDependencyFileResult? { - val jpsModule = moduleOutputProvider.findModule(pluginModuleName) ?: return null + // All data from shared jobs - NO additional lookups needed + val info = pluginContentJobs.get(pluginModuleName)?.await() ?: return null - val pluginXmlPath = findFileInModuleSources(module = jpsModule, relativePath = PLUGIN_XML_RELATIVE_PATH, onlyProductionSources = true) ?: return null - - // Read file once and extract content modules (these should be excluded from dependencies) - val pluginXmlContent = Files.readString(pluginXmlPath) - // null means plugin has content module references (modules with '/'), skip it - val contentModules = extractContentModulesFromText(pluginXmlContent) ?: return null - - // Get JPS dependencies that have XML descriptors (content modules) - // Skip if: - // 1. Module is in section of this plugin.xml - // 2. Module is filtered out by dependencyFilter - // 3. Module doesn't have a descriptor - val deps = mutableListOf() - for (dep in jpsModule.getProductionModuleDependencies(withTests = false)) { - val depName = dep.moduleReference.moduleName - if (depName in contentModules) { - continue + // Filter JPS dependencies: exclude content modules, apply filter, require descriptor + val dependencies = info.jpsDependencies() + .filter { depName -> + depName !in info.contentModules && + dependencyFilter(depName) && + descriptorCache.hasDescriptor(depName) } - if (!dependencyFilter(depName)) { - continue - } - if (!descriptorCache.hasDescriptor(depName)) { - continue - } - deps.add(depName) - } + .distinct() + .sorted() - val dependencies = deps.distinct().sorted() - val status = updateXmlDependencies(path = pluginXmlPath, content = pluginXmlContent, moduleDependencies = dependencies, preserveExistingModule = { !dependencyFilter(it) }) + val status = updateXmlDependencies( + path = info.pluginXmlPath, + content = info.pluginXmlContent, + moduleDependencies = dependencies, + preserveExistingModule = { !dependencyFilter(it) }, + ) // Also process content modules - generate dependencies for their module descriptors val contentModuleResults = mutableListOf() - for (contentModuleName in contentModules) { + for (contentModuleName in info.contentModules) { val result = generateContentModuleDependencies(contentModuleName = contentModuleName, descriptorCache = descriptorCache, dependencyFilter = dependencyFilter) if (result != null) { contentModuleResults.add(result) @@ -96,7 +88,7 @@ private fun generatePluginDependency( return PluginDependencyFileResult( pluginModuleName = pluginModuleName, - pluginXmlPath = pluginXmlPath, + pluginXmlPath = info.pluginXmlPath, status = status, dependencyCount = dependencies.size, contentModuleResults = contentModuleResults, diff --git a/platform/build-scripts/product-dsl/src/ProductModulesContentSpec.kt b/platform/build-scripts/product-dsl/src/ProductModulesContentSpec.kt index 7697c7ebc2d5..bd0209d8a274 100644 --- a/platform/build-scripts/product-dsl/src/ProductModulesContentSpec.kt +++ b/platform/build-scripts/product-dsl/src/ProductModulesContentSpec.kt @@ -141,6 +141,13 @@ class ProductModulesContentSpec( */ @JvmField val bundledPlugins: List = emptyList(), + /** + * Modules that are allowed to be missing during validation. + * These are typically provided by plugin layouts rather than module sets. + * Example: CIDR modules for CLion/AppCode that come from plugin bundles. + */ + @JvmField val allowedMissingDependencies: Set = emptySet(), + /** * Composition graph tracking how this spec was assembled. * Records all include(), moduleSet(), and other composition operations. @@ -166,6 +173,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { private val moduleSets = mutableListOf() private val additionalModules = mutableListOf() private val bundledPlugins = mutableListOf() + private val allowedMissingDeps = LinkedHashSet() // Composition tracking private val compositionGraph = mutableListOf() @@ -230,6 +238,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { xmlIncludes.addAll(spec.deprecatedXmlIncludes) moduleSets.addAll(spec.moduleSets) additionalModules.addAll(spec.additionalModules) + allowedMissingDeps.addAll(spec.allowedMissingDependencies) // Also preserve the nested spec's composition graph for deep analysis compositionGraph.addAll(spec.compositionGraph) @@ -310,7 +319,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { type = CompositionType.DIRECT_MODULE, reference = name, path = pathStack.toList(), - sourceLocation = null + sourceLocation = null, )) } @@ -323,7 +332,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { type = CompositionType.DIRECT_MODULE, reference = name, path = pathStack.toList(), - sourceLocation = null + sourceLocation = null, )) } @@ -351,6 +360,16 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { bundledPlugins.addAll(pluginModules) } + /** + * Allow specific modules to be missing during validation. + * Use for modules provided by plugin layouts rather than module sets. + * + * @param modules Module names that are allowed to be missing + */ + fun allowMissingDependencies(vararg modules: String) { + allowedMissingDeps.addAll(modules) + } + @PublishedApi internal fun build(): ProductModulesContentSpec { return ProductModulesContentSpec( @@ -360,6 +379,7 @@ class ProductModulesContentSpecBuilder @PublishedApi internal constructor() { moduleSets = java.util.List.copyOf(moduleSets), additionalModules = java.util.List.copyOf(additionalModules), bundledPlugins = java.util.List.copyOf(bundledPlugins), + allowedMissingDependencies = java.util.Set.copyOf(allowedMissingDeps), compositionGraph = java.util.List.copyOf(compositionGraph), metadata = metadata, ) diff --git a/platform/build-scripts/product-dsl/src/analysis/DependencyValidation.kt b/platform/build-scripts/product-dsl/src/analysis/DependencyValidation.kt index ad06fd2782ca..e836f362bc3e 100644 --- a/platform/build-scripts/product-dsl/src/analysis/DependencyValidation.kt +++ b/platform/build-scripts/product-dsl/src/analysis/DependencyValidation.kt @@ -2,192 +2,117 @@ package org.jetbrains.intellij.build.productLayout.analysis import com.intellij.platform.plugins.parser.impl.elements.ModuleLoadingRuleValue +import kotlinx.coroutines.Deferred import org.jetbrains.intellij.build.productLayout.AnsiColors import org.jetbrains.intellij.build.productLayout.ModuleDescriptorCache import org.jetbrains.intellij.build.productLayout.ModuleSet +import org.jetbrains.intellij.build.productLayout.PluginContentInfo import org.jetbrains.intellij.build.productLayout.ProductModulesContentSpec -/** - * Known missing dependencies that are temporarily allowed. - * These are typically provided by plugin layouts rather than module sets. - * TODO: Move these to proper module sets or fix product definitions - */ -internal val KNOWN_MISSING_DEPENDENCIES = setOf( - "intellij.cidr.core", - "intellij.cidr.util.execution", - "intellij.cidr.debugger.core", - "intellij.cidr.debugger.backend", - "intellij.cidr.runner" -) +// region Validation Result Types + +internal sealed interface ValidationError { + val context: String +} + +internal data class SelfContainedValidationError( + override val context: String, + val missingDependencies: Map>, +) : ValidationError + +internal data class MissingModuleSetsError( + override val context: String, + val missingModuleSets: Set, +) : ValidationError + +internal data class DuplicateModulesError( + override val context: String, + val duplicates: Map, +) : ValidationError + +internal data class MissingDependenciesError( + override val context: String, + val missingModules: Map>, + val allModuleSets: List, +) : ValidationError + +// endregion + +// region Validation Functions /** * Validates module sets marked with selfContained=true in isolation. - * + * * Self-contained module sets must be resolvable without other module sets. * This ensures they have all their dependencies available internally. - * + * * Example: core.platform is self-contained because CodeServer uses it alone * without other module sets, so it must contain everything needed. - * - * Only validates self-contained sets to avoid false positives from composable - * module sets like debugger(), vcs(), xml() that depend on modules from other sets. - * - * **Key difference from normal validation**: For self-contained sets, ALL modules - * in the entire hierarchy (including nested sets) can see ALL other modules. - * This matches runtime behavior where order doesn't matter (topological sort). + * + * @return List of validation errors (empty if all self-contained sets are valid) */ internal fun validateSelfContainedModuleSets( allModuleSets: List, - descriptorCache: ModuleDescriptorCache -) { - // Collect all self-contained module sets (recursively check nested sets too) - val selfContainedSets = mutableListOf() - fun collectSelfContained(moduleSet: ModuleSet) { - if (moduleSet.selfContained) { - selfContainedSets.add(moduleSet) - } - for (nestedSet in moduleSet.nestedSets) { - collectSelfContained(nestedSet) - } - } - for (moduleSet in allModuleSets) { - collectSelfContained(moduleSet) - } - + descriptorCache: ModuleDescriptorCache, +): List { + val selfContainedSets = collectSelfContainedSets(allModuleSets) if (selfContainedSets.isEmpty()) { - return + return emptyList() } - - // Validate each self-contained set in isolation + + val errors = mutableListOf() + for (moduleSet in selfContainedSets) { - // Collect ALL modules from the entire hierarchy (flatten nested sets) - val allModulesInSet = mutableSetOf() - fun collectAllModules(ms: ModuleSet) { - for (module in ms.modules) { - allModulesInSet.add(module.name) - } - for (nestedSet in ms.nestedSets) { - collectAllModules(nestedSet) - } - } - collectAllModules(moduleSet) - - // Collect modules with descriptors - val modulesWithDescriptors = allModulesInSet.mapNotNull { moduleName -> - descriptorCache.getOrAnalyze(moduleName)?.let { moduleName to it } - } - - if (modulesWithDescriptors.isEmpty()) { - continue - } - - // Validate dependencies: each module must be able to reach its dependencies - // within the flattened set (all modules can see all other modules) - val missingDeps = mutableMapOf>() - - for ((moduleName, info) in modulesWithDescriptors) { - // Check direct and transitive dependencies - val visited = mutableSetOf(moduleName) - val queue = ArrayDeque(info.dependencies.map { it to listOf(moduleName) }) - - while (queue.isNotEmpty()) { - val (dep, chain) = queue.removeFirst() - - if (dep in visited) { - continue - } - visited.add(dep) - - // Check if dependency is in the flattened set - if (dep !in allModulesInSet) { - missingDeps.getOrPut(dep) { mutableSetOf() }.add(chain.first()) - } - - // Add transitive dependencies - val depInfo = descriptorCache.getOrAnalyze(dep) - if (depInfo != null) { - for (transitiveDep in depInfo.dependencies) { - queue.add(transitiveDep to (chain + dep)) - } - } - } - } - + val allModulesInSet = ModuleSetTraversal.collectAllModuleNames(moduleSet) + val missingDeps = findMissingTransitiveDependencies( + modules = allModulesInSet, + availableModules = allModulesInSet, + descriptorCache = descriptorCache, + ) + if (missingDeps.isNotEmpty()) { - error(buildString { - appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Module set '${moduleSet.name}' is marked selfContained but has unresolvable dependencies${AnsiColors.RESET}") - appendLine() - - for ((dep, needingModules) in missingDeps.entries.sortedByDescending { it.value.size }) { - appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$dep'${AnsiColors.RESET}") - appendLine(" Needed by: ${needingModules.sorted().joinToString(", ")}") - } - - appendLine() - appendLine("${AnsiColors.YELLOW}💡 To fix:${AnsiColors.RESET}") - appendLine("1. Add the missing modules/sets to '${moduleSet.name}' to make it truly self-contained") - appendLine("2. Or remove selfContained=true if this set is designed to compose with other sets") - }) + errors.add(SelfContainedValidationError(context = moduleSet.name, missingDependencies = missingDeps)) } } + + return errors } /** * Validates that all products have resolvable module set dependencies. - * + * * This is Tier 2 validation that ensures products can actually load at runtime. * It validates that: * 1. All module sets referenced by a product exist and are resolvable * 2. All modules in those sets can have their dependencies satisfied within the product's composition * 3. No module references dependencies outside the product's available modules - * - * This catches the class of errors like: - * "Plugin 'Java' has dependency on 'com.intellij.modules.vcs' which is not installed" - * - * @param allModuleSets All available module sets (community + ultimate) - * @param productSpecs List of (productName, ProductModulesContentSpec) pairs - * @param descriptorCache Cache for module descriptor information - * @throws IllegalStateException if any product has unresolvable dependencies + * + * @return List of validation errors (empty if all products are valid) */ -internal fun validateProductModuleSets( +internal suspend fun validateProductModuleSets( allModuleSets: List, productSpecs: List>, descriptorCache: ModuleDescriptorCache, - precomputedEmbeddedModules: Set? = null, -) { - data class ProductError( - val productName: String, - val missingModules: Map>, // module -> set of dependencies it needs - ) - - val productErrors = mutableListOf() + pluginContentJobs: Map> = emptyMap(), +): List { + val allPluginModules = collectAllPluginModules(pluginContentJobs) val moduleSetsByName = allModuleSets.associateBy { it.name } - + val errors = mutableListOf() + for ((productName, spec) in productSpecs) { - // Skip products without specs if (spec == null) { continue } - - // Build index of modules available in this product - val productIndex = buildProductModuleIndex(productName, spec, precomputedEmbeddedModules) - - // Validate that all referenced module sets exist - val missingModuleSets = productIndex.referencedModuleSets.filter { it !in moduleSetsByName } + + val productIndex = buildProductModuleIndex(productName, spec, pluginContentJobs) + + // Check for missing module sets + val missingModuleSets = productIndex.referencedModuleSets.filterNot { it in moduleSetsByName } if (missingModuleSets.isNotEmpty()) { - error(buildString { - appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product '$productName' references non-existent module sets${AnsiColors.RESET}") - appendLine() - for (setName in missingModuleSets.sorted()) { - appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Module set '${AnsiColors.BOLD}$setName${AnsiColors.RESET}' does not exist") - } - appendLine() - appendLine("${AnsiColors.BLUE}💡 Fix: Remove the reference or define the module set${AnsiColors.RESET}") - }) + errors.add(MissingModuleSetsError(context = productName, missingModuleSets = missingModuleSets.toSet())) } - - // Validate no duplicate content modules in product - // This validation ALWAYS runs, even for products in allowUnresolvableProducts + + // Check for duplicate content modules val allContentModules = mutableListOf() for (moduleSetWithOverrides in spec.moduleSets) { val moduleSet = moduleSetsByName[moduleSetWithOverrides.moduleSet.name] @@ -198,222 +123,276 @@ internal fun validateProductModuleSets( for (module in spec.additionalModules) { allContentModules.add(module.name) } - + val duplicateModules = allContentModules.groupingBy { it }.eachCount().filter { it.value > 1 } if (duplicateModules.isNotEmpty()) { - error(buildString { - appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product '$productName' has duplicate content modules${AnsiColors.RESET}") - appendLine() - appendLine("${AnsiColors.YELLOW}Duplicated modules (appearing ${AnsiColors.BOLD}${duplicateModules.values.max()}${AnsiColors.RESET}${AnsiColors.YELLOW} times):${AnsiColors.RESET}") - for ((moduleName, count) in duplicateModules.entries.sortedBy { it.key }) { - appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} ${AnsiColors.BOLD}$moduleName${AnsiColors.RESET} (appears $count times)") - } - appendLine() - appendLine("${AnsiColors.BLUE}💡 This causes runtime error: \"Plugin has duplicated content modules declarations\"${AnsiColors.RESET}") - appendLine("${AnsiColors.BLUE}Fix: Remove duplicate moduleSet() nesting or redundant module() calls${AnsiColors.RESET}") - }) + errors.add(DuplicateModulesError(context = productName, duplicates = duplicateModules)) } - - // Validate module dependencies within product scope - // Note: For non-embedded modules, we use a heuristic: if a dependency has an XML - // descriptor but is not in the product's content modules, we assume it's a plugin - // module and skip validation. This is because non-embedded content modules can - // depend on plugins (bundled or not). This heuristic does not verify the plugin - // is actually bundled with the product - that's a separate validation concern. - val missingDependencies = mutableMapOf>() - for (moduleName in productIndex.allModules) { - val info = descriptorCache.getOrAnalyze(moduleName) ?: continue - val reachableModules = productIndex.moduleToReachableModules[moduleName] ?: emptySet() - val isEmbedded = moduleName in productIndex.embeddedModules + // Check for missing dependencies + val criticalModules = productIndex.moduleLoadings + .filterValues { it == ModuleLoadingRuleValue.EMBEDDED || it == ModuleLoadingRuleValue.REQUIRED } + .keys - // Check direct dependencies - for (dependency in info.dependencies) { - if (dependency in reachableModules || dependency in KNOWN_MISSING_DEPENDENCIES) { - continue // OK - in content or known exception - } + val missingDeps = findMissingTransitiveDependencies( + modules = productIndex.allModules, + availableModules = productIndex.allModules, + descriptorCache = descriptorCache, + allowedMissing = spec.allowedMissingDependencies, + crossPluginModules = allPluginModules, + criticalModules = criticalModules, + ) - if (isEmbedded) { - // Embedded modules: strict - must be in content - missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dependency) - } else { - // Non-embedded modules: heuristic - if has descriptor, assume plugin module - if (!descriptorCache.hasDescriptor(dependency)) { - // No descriptor = truly missing - missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dependency) - } - // else: has descriptor, assume it's a plugin module - OK - } - } - - // Check transitive dependencies - val visited = mutableSetOf(moduleName) - val queue = ArrayDeque(info.dependencies) - - while (queue.isNotEmpty()) { - val dep = queue.removeFirst() - if (dep in visited) continue - visited.add(dep) - - if (dep !in reachableModules && dep !in KNOWN_MISSING_DEPENDENCIES) { - if (isEmbedded) { - // Embedded modules: strict - must be in content - missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dep) - } else { - // Non-embedded modules: heuristic - if has descriptor, assume plugin module - if (!descriptorCache.hasDescriptor(dep)) { - missingDependencies.getOrPut(moduleName) { mutableSetOf() }.add(dep) - } - } - } - - val depInfo = descriptorCache.getOrAnalyze(dep) - if (depInfo != null) { - queue.addAll(depInfo.dependencies) - } - } - } - - if (missingDependencies.isNotEmpty()) { - productErrors.add(ProductError(productName, missingDependencies)) + if (missingDeps.isNotEmpty()) { + errors.add(MissingDependenciesError(context = productName, missingModules = missingDeps, allModuleSets = allModuleSets)) } } - - // Report all errors - if (productErrors.isNotEmpty()) { - error(buildString { - appendLine("${AnsiColors.RED}${AnsiColors.BOLD}❌ Product-level validation failed: Unresolvable module dependencies${AnsiColors.RESET}") - appendLine() - - for (productError in productErrors) { - appendLine("${AnsiColors.BOLD}Product: ${productError.productName}${AnsiColors.RESET}") - appendLine() - - // Group by missing dependency for clearer output - val depToModules = mutableMapOf>() - for ((module, deps) in productError.missingModules) { - for (dep in deps) { - depToModules.getOrPut(dep) { mutableSetOf() }.add(module) - } - } - - for ((missingDep, needingModules) in depToModules.entries.sortedByDescending { it.value.size }) { - appendLine(" ${AnsiColors.RED}✗${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$missingDep'${AnsiColors.RESET}") - appendLine(" Needed by: ${needingModules.sorted().take(5).joinToString(", ")}") - if (needingModules.size > 5) { - appendLine(" ... and ${needingModules.size - 5} more modules") - } - - // Suggest which module sets contain this dependency - val containingSets = allModuleSets.filter { moduleSet -> - ModuleSetTraversal.containsModule(moduleSet, missingDep) - }.map { it.name } - - if (containingSets.isNotEmpty()) { - appendLine(" ${AnsiColors.BLUE}Suggestion:${AnsiColors.RESET} Add module set: ${containingSets.joinToString(" or ")}") - } - appendLine() - } - - appendLine() - } - - appendLine("${AnsiColors.BLUE}💡 This will cause runtime errors: \"Plugin X has dependency on Y which is not installed\"${AnsiColors.RESET}") - appendLine() - appendLine("${AnsiColors.BOLD}To fix:${AnsiColors.RESET}") - appendLine("${AnsiColors.BLUE}1.${AnsiColors.RESET} Add the required module sets to the product's getProductContentDescriptor()") - appendLine("${AnsiColors.BLUE}2.${AnsiColors.RESET} Or add individual modules via module()/embeddedModule()") - appendLine("${AnsiColors.BLUE}3.${AnsiColors.RESET} Or add the product to allowUnresolvableProducts if this is intentional") - }) - } + + return errors } -// Helper functions +// endregion + +// region Helper Functions + +private fun collectSelfContainedSets(allModuleSets: List): List { + val result = mutableListOf() + + fun visit(moduleSet: ModuleSet) { + if (moduleSet.selfContained) { + result.add(moduleSet) + } + for (nestedSet in moduleSet.nestedSets) { + visit(nestedSet) + } + } + + for (moduleSet in allModuleSets) { + visit(moduleSet) + } + return result +} + +private suspend fun collectAllPluginModules(pluginContentJobs: Map>): Set { + val result = mutableSetOf() + for ((_, job) in pluginContentJobs) { + job.await()?.contentModules?.let { result.addAll(it) } + } + return result +} /** - * Builds an index of all modules available in a specific product. - * This is product-scoped, unlike the global ModuleSetIndex. + * Finds missing transitive dependencies using BFS traversal. + * + * @param modules Modules to check dependencies for + * @param availableModules Modules that are available (dependencies should be in this set) + * @param descriptorCache Cache for module descriptor information + * @param allowedMissing Dependencies explicitly allowed to be missing + * @param crossPluginModules Modules from other plugins (valid for non-critical modules) + * @param criticalModules Modules that cannot depend on cross-plugin modules + * @return Map of missing dependency -> set of modules that need it */ +internal fun findMissingTransitiveDependencies( + modules: Set, + availableModules: Set, + descriptorCache: ModuleDescriptorCache, + allowedMissing: Set = emptySet(), + crossPluginModules: Set = emptySet(), + criticalModules: Set = emptySet(), +): Map> { + val missingDeps = mutableMapOf>() + + for (moduleName in modules) { + val info = descriptorCache.getOrAnalyze(moduleName) ?: continue + val isCritical = moduleName in criticalModules + val visited = mutableSetOf(moduleName) + val queue = ArrayDeque(info.dependencies) + + while (queue.isNotEmpty()) { + val dep = queue.removeFirst() + if (dep in visited) { + continue + } + visited.add(dep) + + when { + dep in availableModules -> { + // Present - traverse into its deps + descriptorCache.getOrAnalyze(dep)?.dependencies?.let { queue.addAll(it) } + } + dep in allowedMissing -> { + // Explicitly allowed to be missing - skip + } + dep in crossPluginModules && !isCritical -> { + // Valid cross-plugin optional dependency - skip + } + else -> { + // Missing dependency + missingDeps.getOrPut(dep) { mutableSetOf() }.add(moduleName) + } + } + } + } + + return missingDeps +} + private data class ProductModuleIndex( val productName: String, - val allModules: Set, // All modules available in this product - val embeddedModules: Set, // Modules with EMBEDDED loading - val moduleToReachableModules: Map>, // Product-scoped reachability - val referencedModuleSets: Set, // Module set names used by product + val allModules: Set, + val referencedModuleSets: Set, + val moduleLoadings: Map, ) -private fun buildProductModuleIndex( +private suspend fun buildProductModuleIndex( productName: String, spec: ProductModulesContentSpec, - precomputedEmbeddedModules: Set? = null, + pluginContentJobs: Map> = emptyMap(), ): ProductModuleIndex { val allModules = mutableSetOf() val referencedModuleSets = mutableSetOf() + val moduleLoadings = mutableMapOf() - // Recursively collect all modules from a module set - fun collectModulesFromSet(moduleSet: ModuleSet) { - for (module in moduleSet.modules) { - allModules.add(module.name) - } - for (nestedSet in moduleSet.nestedSets) { - collectModulesFromSet(nestedSet) - } - } - - // Process all module sets referenced by the product - // Use moduleSetWithOverrides.moduleSet directly - it already contains the full ModuleSet for (moduleSetWithOverrides in spec.moduleSets) { val moduleSet = moduleSetWithOverrides.moduleSet referencedModuleSets.add(moduleSet.name) - collectModulesFromSet(moduleSet) + collectModulesWithLoadings(moduleSet, allModules, moduleLoadings) } - // Add additional individual modules for (module in spec.additionalModules) { allModules.add(module.name) + moduleLoadings[module.name] = module.loading } - // Use precomputed embedded modules if available (optimization to avoid recomputing), - // otherwise compute just for this product's modules - val embeddedModules = if (precomputedEmbeddedModules != null) { - // Filter to only include modules that are in this product - allModules.filterTo(mutableSetOf()) { it in precomputedEmbeddedModules } + for (pluginName in spec.bundledPlugins) { + pluginContentJobs[pluginName]?.await()?.contentModules?.let { allModules.addAll(it) } } - else { - // Fallback: compute embedded modules for this product - val result = mutableSetOf() - fun collectEmbeddedFromSet(moduleSet: ModuleSet) { - for (module in moduleSet.modules) { - if (module.loading == ModuleLoadingRuleValue.EMBEDDED) { - result.add(module.name) - } - } - for (nestedSet in moduleSet.nestedSets) { - collectEmbeddedFromSet(nestedSet) - } - } - for (moduleSetWithOverrides in spec.moduleSets) { - collectEmbeddedFromSet(moduleSetWithOverrides.moduleSet) - } - for (module in spec.additionalModules) { - if (module.loading == ModuleLoadingRuleValue.EMBEDDED) { - result.add(module.name) - } - } - result - } - - // Build product-scoped reachability: each module can see all other modules in the product - // This is different from module set validation where modules can only see within their set - val moduleToReachableModules = allModules.associateWith { allModules } return ProductModuleIndex( productName = productName, allModules = allModules, - embeddedModules = embeddedModules, - moduleToReachableModules = moduleToReachableModules, referencedModuleSets = referencedModuleSets, + moduleLoadings = moduleLoadings, ) } +private fun collectModulesWithLoadings( + moduleSet: ModuleSet, + modules: MutableSet, + loadings: MutableMap, +) { + for (module in moduleSet.modules) { + modules.add(module.name) + loadings[module.name] = module.loading + } + for (nestedSet in moduleSet.nestedSets) { + collectModulesWithLoadings(nestedSet, modules, loadings) + } +} +// endregion + +// region Formatting + +internal fun formatValidationErrors(errors: List): String { + if (errors.isEmpty()) { + return "" + } + + return buildString { + for (error in errors) { + when (error) { + is SelfContainedValidationError -> formatSelfContainedError(this, error) + is MissingModuleSetsError -> formatMissingModuleSetsError(this, error) + is DuplicateModulesError -> formatDuplicateModulesError(this, error) + is MissingDependenciesError -> formatMissingDependenciesError(this, error) + } + } + } +} + +private fun formatSelfContainedError(sb: StringBuilder, error: SelfContainedValidationError) { + sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Module set '${error.context}' is marked selfContained but has unresolvable dependencies${AnsiColors.RESET}") + sb.appendLine() + + for ((dep, needingModules) in error.missingDependencies.entries.sortedByDescending { it.value.size }) { + sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$dep'${AnsiColors.RESET}") + sb.appendLine(" Needed by: ${needingModules.sorted().joinToString(", ")}") + } + + sb.appendLine() + sb.appendLine("${AnsiColors.YELLOW}To fix:${AnsiColors.RESET}") + sb.appendLine("1. Add the missing modules/sets to '${error.context}' to make it truly self-contained") + sb.appendLine("2. Or remove selfContained=true if this set is designed to compose with other sets") + sb.appendLine() +} + +private fun formatMissingModuleSetsError(sb: StringBuilder, error: MissingModuleSetsError) { + sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product '${error.context}' references non-existent module sets${AnsiColors.RESET}") + sb.appendLine() + for (setName in error.missingModuleSets.sorted()) { + sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Module set '${AnsiColors.BOLD}$setName${AnsiColors.RESET}' does not exist") + } + sb.appendLine() + sb.appendLine("${AnsiColors.BLUE}Fix: Remove the reference or define the module set${AnsiColors.RESET}") + sb.appendLine() +} + +private fun formatDuplicateModulesError(sb: StringBuilder, error: DuplicateModulesError) { + sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product '${error.context}' has duplicate content modules${AnsiColors.RESET}") + sb.appendLine() + sb.appendLine("${AnsiColors.YELLOW}Duplicated modules (appearing ${AnsiColors.BOLD}${error.duplicates.values.max()}${AnsiColors.RESET}${AnsiColors.YELLOW} times):${AnsiColors.RESET}") + for ((moduleName, count) in error.duplicates.entries.sortedBy { it.key }) { + sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} ${AnsiColors.BOLD}$moduleName${AnsiColors.RESET} (appears $count times)") + } + sb.appendLine() + sb.appendLine("${AnsiColors.BLUE}This causes runtime error: \"Plugin has duplicated content modules declarations\"${AnsiColors.RESET}") + sb.appendLine("${AnsiColors.BLUE}Fix: Remove duplicate moduleSet() nesting or redundant module() calls${AnsiColors.RESET}") + sb.appendLine() +} + +private fun formatMissingDependenciesError(sb: StringBuilder, error: MissingDependenciesError) { + sb.appendLine("${AnsiColors.BOLD}Product: ${error.context}${AnsiColors.RESET}") + sb.appendLine() + + // Group by missing dependency for clearer output + val depToModules = mutableMapOf>() + for ((module, deps) in error.missingModules) { + for (dep in deps) { + depToModules.getOrPut(dep) { mutableSetOf() }.add(module) + } + } + + for ((missingDep, needingModules) in depToModules.entries.sortedByDescending { it.value.size }) { + sb.appendLine(" ${AnsiColors.RED}*${AnsiColors.RESET} Missing: ${AnsiColors.BOLD}'$missingDep'${AnsiColors.RESET}") + sb.appendLine(" Needed by: ${needingModules.sorted().take(5).joinToString(", ")}") + if (needingModules.size > 5) { + sb.appendLine(" ... and ${needingModules.size - 5} more modules") + } + + val containingSets = error.allModuleSets + .filter { ModuleSetTraversal.containsModule(it, missingDep) } + .map { it.name } + + if (containingSets.isNotEmpty()) { + sb.appendLine(" ${AnsiColors.BLUE}Suggestion:${AnsiColors.RESET} Add module set: ${containingSets.joinToString(" or ")}") + } + sb.appendLine() + } +} + +internal fun formatProductDependencyErrorsHeader(sb: StringBuilder) { + sb.appendLine("${AnsiColors.RED}${AnsiColors.BOLD}Product-level validation failed: Unresolvable module dependencies${AnsiColors.RESET}") + sb.appendLine() +} + +internal fun formatProductDependencyErrorsFooter(sb: StringBuilder) { + sb.appendLine("${AnsiColors.BLUE}This will cause runtime errors: \"Plugin X has dependency on Y which is not installed\"${AnsiColors.RESET}") + sb.appendLine() + sb.appendLine("${AnsiColors.BOLD}To fix:${AnsiColors.RESET}") + sb.appendLine("${AnsiColors.BLUE}1.${AnsiColors.RESET} Add the required module sets to the product's getProductContentDescriptor()") + sb.appendLine("${AnsiColors.BLUE}2.${AnsiColors.RESET} Or add individual modules via module()/embeddedModule()") + sb.appendLine("${AnsiColors.BLUE}3.${AnsiColors.RESET} Or add the product to allowUnresolvableProducts if this is intentional") +} + +// endregion diff --git a/platform/build-scripts/product-dsl/src/productDiscovery.kt b/platform/build-scripts/product-dsl/src/productDiscovery.kt index 9821ce32a79e..c383e2443f10 100644 --- a/platform/build-scripts/product-dsl/src/productDiscovery.kt +++ b/platform/build-scripts/product-dsl/src/productDiscovery.kt @@ -4,12 +4,16 @@ package org.jetbrains.intellij.build.productLayout import com.intellij.platform.plugins.parser.impl.elements.ModuleLoadingRuleValue +import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.async import kotlinx.coroutines.awaitAll import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.withContext import kotlinx.serialization.SerialName import kotlinx.serialization.Serializable import org.jetbrains.intellij.build.ModuleOutputProvider +import org.jetbrains.intellij.build.PLUGIN_XML_RELATIVE_PATH +import org.jetbrains.intellij.build.findFileInModuleSources import org.jetbrains.jps.model.java.JavaSourceRootType import org.jetbrains.jps.model.java.JpsJavaExtensionService import java.nio.file.Files @@ -33,7 +37,7 @@ data class ProductConfigurationRegistry(@JvmField val products: Map, @JvmField @SerialName("class") val className: String, - @JvmField val pluginXmlPath: String? = null + @JvmField val pluginXmlPath: String? = null, ) /** @@ -76,7 +80,7 @@ fun findProductPropertiesSourceFile( buildModules: List, productPropertiesClass: Class<*>, moduleOutputProvider: ModuleOutputProvider, - projectRoot: Path + projectRoot: Path, ): String { val className = productPropertiesClass.name @@ -105,17 +109,6 @@ fun findProductPropertiesSourceFile( throw IllegalStateException("Cannot find source file for $productPropertiesClass (searched for $relativePath in modules: $buildModules)") } -/** - * Extracts product validation data from discovered products. - * Returns list of (productName, ProductModulesContentSpec) pairs for validation. - * - * @param discoveredProducts List of discovered products - * @return List of product name and spec pairs for validation - */ -fun extractProductsForValidation(discoveredProducts: List): List> { - return discoveredProducts.map { it.name to it.spec } -} - /** * Generates product XMLs for all products using programmatic content. * Takes discovered products and test product specs, then generates complete plugin.xml files. @@ -166,13 +159,13 @@ internal suspend fun generateAllProductXmlFiles( val spec = discovered.spec ?: return@async null val pluginXmlPath = projectRoot.resolve(pluginXmlRelativePath) - + // Extract ProductProperties class name (works with both ProductProperties and null) val productPropertiesClass = when (val props = discovered.properties) { null -> "test-product" else -> props.javaClass.name } - + generateProductXml( pluginXmlPath = pluginXmlPath, spec = spec, @@ -231,7 +224,7 @@ private fun aggregateAndCleanupOrphanedFiles(moduleSetResults: List + async { extractPluginContent(pluginName, config.moduleOutputProvider) } + } + // TIER 2: Parallel dependency and product generation (can run concurrently with TIER 1) val dependencyJob = async { val moduleSetsByLabel = config.moduleSetSources.mapValues { (_, source) -> @@ -293,21 +296,12 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig) coreModuleSets = moduleSetsByLabel.get("core") ?: emptyList(), moduleOutputProvider = config.moduleOutputProvider, productSpecs = products, - embeddedModulesDeferred = embeddedModulesDeferred, + pluginContentJobs = pluginContentJobs, ) } // TIER 3: Plugin dependency generation for bundled plugins val pluginDependencyJob = async { - val allBundledPlugins = ( - config.discoveredProducts - .asSequence() - .mapNotNull { it.spec?.bundledPlugins } - .flatten() + config.additionalPlugins - ) - .distinct() - .toList() - if (allBundledPlugins.isEmpty()) { null } @@ -322,12 +316,7 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig) false } } - generatePluginDependencies( - plugins = allBundledPlugins, - moduleOutputProvider = config.moduleOutputProvider, - descriptorCache = cache, - dependencyFilter = dependencyFilter, - ) + generatePluginDependencies(plugins = allBundledPlugins, pluginContentJobs = pluginContentJobs, descriptorCache = cache, dependencyFilter = dependencyFilter) } } @@ -360,6 +349,38 @@ suspend fun generateAllModuleSetsWithProducts(config: ModuleSetGenerationConfig) ) } +/** + * Result of extracting plugin content from plugin.xml. + * Contains everything needed for both validation and dependency generation. + */ +data class PluginContentInfo( + @JvmField val pluginXmlPath: Path, + @JvmField val pluginXmlContent: String, + @JvmField val contentModules: Set, + /** Lazy JPS production dependencies - only called by plugin dep gen, not validation */ + @JvmField val jpsDependencies: () -> List, +) + +/** + * Extracts content modules from a plugin's plugin.xml. + * Returns null if plugin.xml not found or has module references with '/'. + */ +private suspend fun extractPluginContent( + pluginName: String, + moduleOutputProvider: ModuleOutputProvider, +): PluginContentInfo? { + val jpsModule = moduleOutputProvider.findModule(pluginName) ?: return null + val pluginXmlPath = findFileInModuleSources(module = jpsModule, relativePath = PLUGIN_XML_RELATIVE_PATH, onlyProductionSources = true) ?: return null + val content = withContext(Dispatchers.IO) { Files.readString(pluginXmlPath) } + val contentModules = extractContentModulesFromText(content) ?: return null + return PluginContentInfo( + pluginXmlPath = pluginXmlPath, + pluginXmlContent = content, + contentModules = contentModules, + jpsDependencies = { jpsModule.getProductionModuleDependencies(withTests = false).map { it.moduleReference.moduleName }.toList() }, + ) +} + /** * Collects all embedded modules from all product specs. * Used to filter out embedded platform modules from plugin dependencies. diff --git a/platform/build-scripts/product-dsl/validation.md b/platform/build-scripts/product-dsl/validation.md new file mode 100644 index 000000000000..ac755148fb41 --- /dev/null +++ b/platform/build-scripts/product-dsl/validation.md @@ -0,0 +1,315 @@ +# Plugin Model Validation + +This document describes the validation system that ensures module dependencies are resolvable at build time, preventing runtime errors. + +## Overview + +### Why Validation Exists + +Without validation, dependency errors only surface at runtime: +``` +Plugin 'Java' has dependency on 'com.intellij.modules.vcs' which is not installed +``` + +The validation system catches these errors during `Generate Product Layouts`, providing: +- Early detection of missing dependencies +- Clear error messages with suggested fixes +- Module set hierarchy awareness +- Cross-plugin dependency support + +### When Validation Runs + +Validation runs automatically when you execute: +- **IDE**: Run configuration "Generate Product Layouts" +- **CLI**: `UltimateModuleSets.main()` or `CommunityModuleSets.main()` +- **Bazel**: `bazel run //platform/buildScripts:product-model-tool` + +## Two-Tier Validation System + +The build system uses a **two-tier validation approach** to ensure module dependencies are resolvable while avoiding false positives. + +### Tier 1: Product-Level Validation + +**What it validates**: All products with their complete module composition + +**How it works**: +- Collects all modules from a product's complete module set hierarchy +- Validates that each module's dependencies are available within that product context +- Includes bundled plugin content modules in available modules +- Reports errors per-product with clear affected modules + +**Why this tier exists**: Products are the actual deployment units. Most module sets (like `debugger()`, `vcs()`, `xml()`) are **composable building blocks** designed to work together. Validating them in isolation would produce false positives because they intentionally depend on modules from other sets. + +**Example**: +- Product `GoLand` uses: `ide.ultimate`, `ssh`, `rd.common` +- Validation checks: Can all modules in `ide.ultimate` + `ssh` + `rd.common` resolve their dependencies within this combined set? + +### Tier 2: Self-Contained Module Set Validation + +**What it validates**: Module sets marked with `selfContained = true` + +**How it works**: +- Validates the module set in isolation without considering other module sets +- Flattens the entire hierarchy (nested sets all see each other) +- Ensures all dependencies are resolvable within the set itself +- Reports errors specific to that module set + +**Why this tier exists**: Some module sets are designed to be **standalone/self-contained** and used directly by products without composition with other sets. These sets must have all their dependencies available internally. + +**When to use `selfContained = true`**: +- Module set is used directly by products as the primary/only module set +- Module set represents a complete, independent runtime environment +- You want to enforce that the set doesn't leak dependencies on external sets + +**Example**: `core.platform` is self-contained because CodeServer uses it alone. + +```kotlin +fun corePlatform(): ModuleSet = moduleSet( + name = "core.platform", + selfContained = true, // Must be resolvable in isolation + outputModule = "intellij.platform.ide.core" +) { + moduleSet(librariesPlatform()) + moduleSet(rpcMinimal()) + embeddedModule("intellij.platform.core", includeDependencies = true) + // ... +} +``` + +## Cross-Plugin Dependency Validation + +### The Problem + +Bundled plugins can have content modules that optionally depend on modules from non-bundled plugins. For example: + +- `intellij.fullLine.cpp` (bundled in IDEA Ultimate) depends on `intellij.c.core` +- `intellij.c.core` is part of the C/C++ plugin (`intellij.c`), which is **not bundled** + +Without special handling, this triggers a validation error because `intellij.c.core` is not in the product's module sets. + +### Loading Attribute Semantics + +The solution depends on the content module's **loading attribute**: + +| Loading Value | Meaning | Cross-Plugin Deps Allowed? | +|---------------|---------|---------------------------| +| `embedded` | Core module, loaded into main classloader | **No** - must have all deps in product | +| `required` | Required at startup, loaded early | **No** - must have all deps in product | +| `optional` | Loaded on demand if deps available | **Yes** - can depend on non-bundled plugins | +| `on_demand` | Loaded only when explicitly requested | **Yes** - can depend on non-bundled plugins | +| (unspecified) | Default loading behavior | **Yes** - can depend on non-bundled plugins | + +**Key insight**: Critical modules (`embedded`/`required`) cannot depend on non-bundled plugins because those plugins may not be installed. Non-critical modules can safely have cross-plugin dependencies since they won't prevent IDE startup if the dependency is missing. + +### Configuration + +Non-bundled plugins for validation are configured in `UltimateModuleSets.kt`: + +```kotlin +/** + * Non-bundled plugins whose content modules should be available for cross-plugin + * dependency validation. These are plugins that bundled plugins may optionally depend on. + * + * When a bundled plugin's content module (with non-critical loading like optional/on_demand) + * depends on a module from these plugins, the dependency is considered valid. + */ +val NON_BUNDLED_PLUGINS_FOR_VALIDATION: List = listOf( + "intellij.c", // C/C++ support - needed by intellij.fullLine.cpp +) +``` + +### Adding a New Non-Bundled Plugin + +When you encounter a validation error for a cross-plugin dependency: + +1. **Verify the source module is NOT critical** (not `embedded` or `required`) +2. **Find the plugin module name** (not the plugin ID): + ```bash + # Find the .iml file for the plugin + find . -name "*.iml" | xargs grep -l "plugin-descriptor-xml" + ``` +3. **Add to the list**: + ```kotlin + val NON_BUNDLED_PLUGINS_FOR_VALIDATION: List = listOf( + "intellij.c", + "intellij.new.plugin", // Description of why it's needed + ) + ``` + +### How It Works Internally + +1. **Plugin content extraction**: The generator uses `extractPluginContent()` to read each plugin's content modules from its `plugin.xml` + +2. **All plugin modules collection**: At validation start, all plugin content modules (bundled + non-bundled from the config) are collected: + ```kotlin + val allPluginModules = mutableSetOf() + for ((_, job) in pluginContentJobs) { + val info = job.await() + if (info != null) { + allPluginModules.addAll(info.contentModules) + } + } + ``` + +3. **Validation with loading check**: For each module's dependencies: + ```kotlin + val loading = productIndex.moduleLoadings[moduleName] + val isCritical = loading == ModuleLoadingRuleValue.EMBEDDED || + loading == ModuleLoadingRuleValue.REQUIRED + + if (dep in reachableModules) { + // Present in product - valid + } else if (dep in allowedMissing) { + // Explicitly allowed - skip + } else if (dep in allPluginModules && !isCritical) { + // Exists in another plugin AND source is NOT critical - valid + } else { + // Missing dependency - report error + } + ``` + +## allowMissingDependencies + +### What It Is + +Products can specify modules that are allowed to be missing from validation: + +```kotlin +override fun getProductContentModules(): ProductModulesContentSpec = productModules { + moduleSet(CommunityModuleSets.essential()) + // ... + + allowMissingDependencies = setOf( + "intellij.some.optional.module" + ) +} +``` + +### When to Use + +Use `allowMissingDependencies` sparingly for: +- **Modules provided by bundled plugins** that aren't declared as content modules +- **Temporary allowances** during migration (with a TODO to fix) +- **Platform quirks** where a dependency exists at runtime but not in module model + +### When NOT to Use + +Do NOT use for: +- **Cross-plugin dependencies** - use `NON_BUNDLED_PLUGINS_FOR_VALIDATION` instead +- **Avoiding validation errors** - fix the root cause +- **Optional features** - use proper loading attributes instead + +### Best Practices + +1. **Document each entry**: Add a comment explaining why it's needed +2. **Review periodically**: Remove entries that are no longer needed +3. **Prefer proper fixes**: `allowMissingDependencies` is a workaround, not a solution + +## Troubleshooting + +### Common Errors + +#### 1. Missing Dependency in Product + +``` +❌ Product-level validation failed: Unresolvable module dependencies + +Product: GoLand + + ✗ Missing: 'intellij.platform.polySymbols' + Needed by: intellij.platform.vcs.impl + Suggestion: Add module set: symbols +``` + +**Cause**: A module in the product depends on another module not available in the product's composition. + +**Fix options**: +1. Add the suggested module set to the product +2. Add the individual module via `module()` or `embeddedModule()` +3. If it's a cross-plugin dependency with non-critical loading, add the plugin to `NON_BUNDLED_PLUGINS_FOR_VALIDATION` + +#### 2. Self-Contained Set with Unresolvable Dependencies + +``` +❌ Module set 'core.platform' is marked selfContained but has unresolvable dependencies + + ✗ Missing: 'fleet.kernel' + Needed by: intellij.platform.kernel + +💡 To fix: +1. Add the missing modules/sets to 'core.platform' to make it truly self-contained +2. Or remove selfContained=true if this set is designed to compose with other sets +``` + +**Cause**: A self-contained module set doesn't have all its dependencies internally. + +**Fix options**: +1. Add the missing module/module set +2. Remove `selfContained = true` if the set is meant to be composed with others + +#### 3. Duplicate Content Modules + +``` +❌ Product 'GoLand' has duplicate content modules + +Duplicated modules (appearing 2 times): + ✗ intellij.platform.vcs.impl (appears 2 times) + +💡 This causes runtime error: "Plugin has duplicated content modules declarations" +Fix: Remove duplicate moduleSet() nesting or redundant module() calls +``` + +**Cause**: The same module appears twice in the product's content. + +**Fix**: Remove the duplicate - either from a module set or from direct `module()` calls. + +### Investigation Tools + +Use the Plugin Model Analyzer MCP for investigation: + +```kotlin +// Find dependency path between modules +mcp__PluginModelAnalyzer__find_dependency_path( + fromModule = "intellij.platform.vcs.impl", + toModule = "intellij.c.core" +) + +// Check which module sets contain a dependency +mcp__PluginModelAnalyzer__suggest_module_set_for_modules( + moduleNames = ["intellij.c.core"] +) + +// Check module reachability within a module set +mcp__PluginModelAnalyzer__check_module_reachability( + moduleName = "intellij.platform.kernel", + moduleSetName = "core.platform" +) + +// Get module info including which products/sets include it +mcp__PluginModelAnalyzer__get_module_info( + moduleName = "intellij.fullLine.cpp" +) +``` + +### Investigation Strategy + +When you encounter a validation error: + +1. **Identify the dependency chain**: Use `find_dependency_path` to understand why the dependency is needed + +2. **Check if cross-plugin dependency**: Is the missing module in a non-bundled plugin? If so, check if the source module's loading allows cross-plugin deps. + +3. **Find the right fix level**: + - Missing module set in hierarchy → add the nested set + - Product missing required set → add to product + - Cross-plugin dep with non-critical loading → add to `NON_BUNDLED_PLUGINS_FOR_VALIDATION` + - Truly missing infrastructure → add to appropriate module set + +4. **Verify the fix**: Run "Generate Product Layouts" again + +## See Also + +- [Module Sets Documentation](module-sets.md) - How module sets work +- [Programmatic Content](programmatic-content.md) - Product content descriptors +- `DependencyValidation.kt` - Validation implementation +- `UltimateModuleSets.kt` - Non-bundled plugins configuration diff --git a/platform/build-scripts/src/org/jetbrains/intellij/build/JetBrainsProductProperties.kt b/platform/build-scripts/src/org/jetbrains/intellij/build/JetBrainsProductProperties.kt index 945eb2bc0ad7..078d8a4dcfef 100644 --- a/platform/build-scripts/src/org/jetbrains/intellij/build/JetBrainsProductProperties.kt +++ b/platform/build-scripts/src/org/jetbrains/intellij/build/JetBrainsProductProperties.kt @@ -32,6 +32,18 @@ fun isCommunityModule(module: JpsModule, context: BuildContext): Boolean { } } +val knownMissingModuleDependencies: List = listOf( + // todo not included into any plugin - investigate why and fix + "intellij.javaee.jpa", + "intellij.rider.plugins.fsharp", + // conditional xi-include + "kotlin.base.scripting.k1", + // todo special module (make it not special) + "intellij.platform.commercial.verifier", + // included using `withModule` + "intellij.python.frontend" +) + /** * Describes a distribution of an IntelliJ-based IDE hosted in the IntelliJ repository. */ diff --git a/platform/jewel/samples/showcase/src/main/resources/intellij.platform.jewel.samples.showcase.xml b/platform/jewel/samples/showcase/src/main/resources/intellij.platform.jewel.samples.showcase.xml index 1fbc621f1692..80b520043ac9 100644 --- a/platform/jewel/samples/showcase/src/main/resources/intellij.platform.jewel.samples.showcase.xml +++ b/platform/jewel/samples/showcase/src/main/resources/intellij.platform.jewel.samples.showcase.xml @@ -1,9 +1,12 @@ - - - - - + + + + + + + + diff --git a/plugins/devkit/devkit-core/resources/intellij.devkit.core.xml b/plugins/devkit/devkit-core/resources/intellij.devkit.core.xml index 325b6f14881f..b1584eb5de79 100644 --- a/plugins/devkit/devkit-core/resources/intellij.devkit.core.xml +++ b/plugins/devkit/devkit-core/resources/intellij.devkit.core.xml @@ -7,6 +7,9 @@ + + + messages.DevKitBundle diff --git a/plugins/devkit/intellij.devkit.workspaceModel/resources/intellij.devkit.workspaceModel.xml b/plugins/devkit/intellij.devkit.workspaceModel/resources/intellij.devkit.workspaceModel.xml index 4049021e0a3d..8c0609f3176d 100644 --- a/plugins/devkit/intellij.devkit.workspaceModel/resources/intellij.devkit.workspaceModel.xml +++ b/plugins/devkit/intellij.devkit.workspaceModel/resources/intellij.devkit.workspaceModel.xml @@ -4,6 +4,9 @@ + + + messages.DevKitWorkspaceModelBundle diff --git a/python/build/src/org/jetbrains/intellij/build/pycharm/PyCharmCommunityProperties.kt b/python/build/src/org/jetbrains/intellij/build/pycharm/PyCharmCommunityProperties.kt index 56031a59807d..3a53e843c781 100644 --- a/python/build/src/org/jetbrains/intellij/build/pycharm/PyCharmCommunityProperties.kt +++ b/python/build/src/org/jetbrains/intellij/build/pycharm/PyCharmCommunityProperties.kt @@ -71,6 +71,9 @@ open class PyCharmCommunityProperties(protected val communityHome: Path) : PyCha // Static includes deprecatedInclude("intellij.platform.extended.community.impl", "META-INF/community-extensions.xml", ultimateOnly = true) deprecatedInclude("intellij.pycharm.community", "META-INF/pycharm-core-customization.xml") + + allowMissingDependencies(knownMissingModuleDependencies) + bundledPlugins(productLayout.bundledPluginModules.toList()) } override suspend fun copyAdditionalFiles(targetDir: Path, context: BuildContext) {