Darya.Rovdo
60804a8248
[eval-plugin] AIML-13 Fix filtering bug
...
(cherry picked from commit 8d077bced577bdc032807497d7a89f65a29110ca)
IJ-MR-183649
GitOrigin-RevId: aacd4baf4138d89d2078cf4c7ba8a383cc8e2d0f
2025-11-26 11:34:43 +00:00
Anton Spilnyy
00e64bd481
[aia-eval] LME-681 Configure eval for codex+gemini in AIA
...
GitOrigin-RevId: 2e5beeb7968b2fab572c5a259d6623da87d7dbc4
2025-11-23 20:09:42 +00:00
Roman Vasiliev
ce636a8321
[evaluation-plugin] LME-610 The refactoring to get rid of a big number of useless thread blocks
...
Merge-request: IJ-MR-179004
Merged-by: Roman Vasiliev <Roman.Vasiliev@jetbrains.com >
(cherry picked from commit 2469fe2d5af7f5286bf08356cc496daad690fe55)
IJ-MR-179004
GitOrigin-RevId: 00bece64b8380f5ae6a0cc5f369e7a1969189e30
2025-10-24 23:02:23 +00:00
Kiro.Kostov
2c89653984
[aia-eval go] LME-552: Add Go swe eval config
...
Using GOLDEN runner
GitOrigin-RevId: caad37e33ddcd4168576d89d2f94e96a1d1bcb49
2025-10-24 20:28:25 +00:00
Nikolai.Palchikov
0050495bf8
[askai] LLM-20502 add metric to track Ask AI feature usage
...
(cherry picked from commit dca77c73db2ccc37f06b589edf20dcf300fb0e0a)
IJ-MR-179680
GitOrigin-RevId: 0ac37a72f5bd876c0f33af4fc165d39986239886
2025-10-24 17:54:40 +00:00
Nikolai.Palchikov
36d7d990a3
[askai] LLM-20502 track smart chat endpoint calls during evaluation
...
(cherry picked from commit 8ed01cf5eb84eef1464adc72ba8e43b96d2e0e97)
IJ-MR-179680
GitOrigin-RevId: ba0645fc445663cb27d5e2689af8af89a713c69e
2025-10-24 17:54:40 +00:00
anton.spilnyy
cbc5854206
[aia-eval] LME-620 Configure Junie for CodeGenerationSweFeature pipeline
...
(cherry picked from commit 7bcc436e87a74833b678ee4d093eb6c5899d7dbf)
IJ-CR-179283
GitOrigin-RevId: 0cea6800b9e903c980f872132ed35c6219489910
2025-10-21 09:03:00 +00:00
Nikolai Bogdanov
db0d10e846
[aia] LME-609 Remove PSI magic from maven execution
...
Merge-request: IJ-MR-178680
Merged-by: Nikolai Bogdanov <nikolai.bogdanov@jetbrains.com >
(cherry picked from commit 33e6ab060c2be0bd2213dbc3ea07a54b0cf8baf9)
IJ-MR-178680
GitOrigin-RevId: 53cbe56202d3586cda0d26260dc9c653422820f4
2025-10-20 19:00:17 +00:00
SergeiDudoladov
4c27610fe0
[LLM-19751] implements text similarity score for eval
...
(cherry picked from commit bfb947fc964568c74ea9466af643e9b0004dee8e)
IJ-CR-178740
GitOrigin-RevId: 22653f8a87363e67d88d34d06c05a082b118b292
2025-10-16 13:58:28 +00:00
Petr Surkov
95c564daa9
[daemon] Fix eval exception
...
(cherry picked from commit 96f33b84c17d00bb5ec43aaceb0fc4112051687a)
IJ-CR-178664
GitOrigin-RevId: b5bb4d846a53b57cd75b541b0e465685e23e8660
2025-10-15 16:32:06 +00:00
Anna Kozlova
3147b31edc
[kotlin] KTIJ-10041 rename fir modules to obey platform rules
...
GitOrigin-RevId: d431e21c180e5f188e57f622295289eea3754a16
2025-10-13 20:19:55 +00:00
Vladimir Krivosheev
67a6dac7f8
IJPL-209476 IJ-CR-146078 kotlinx-datetime-jvm, kotlinx-document-store-mvstore, kotlinx-html-jvm, kotlinx-serialization-protobuf, kotlinx-collections-immutable
...
GitOrigin-RevId: 058331a1e834d7780456f98d003afe56abfc36a0
2025-10-12 11:10:47 +00:00
Vladimir Krivosheev
30413b84c9
IJPL-209476 IJ-MR-175479 extract jettison, jaxen, bouncy-castle
...
GitOrigin-RevId: d9f6e8e745fbfb69604eb02d0f95cad976d4d7f9
2025-10-09 21:36:26 +00:00
Roman Vasiliev
4c93e1c18a
[evaluation-plugin] LLM-19945 Make exception more meaningful
...
GitOrigin-RevId: bdea0f3507d4e75a1d7a54a34ae099967aed0e38
2025-10-09 15:43:52 +00:00
Nikolay Chashnikov
c3d006d45e
[plugin model] use 'internal' visibility for content modules which are used from modules of other plugins (IJPL-207059)
...
These modules and their classes don't have external usages, so they shouldn't be made 'public' at least for now. The 'namespace' is also set to 'jetbrains' for plugins which contain such modules or modules which use them to allow 'internal' visibility to work.
GitOrigin-RevId: 198007e49320075dc27faadde6963e98332296a4
2025-10-08 18:39:06 +00:00
anton.spilnyy
124cb41c77
[aia-eval] swe. base test for pipeline
...
GitOrigin-RevId: 865fddc22ffd4b3ac580786062cfffb002d85977
2025-10-06 15:21:31 +00:00
Vladimir Krivosheev
aef6fcfb61
IJ-MR-175479 IJ-CR-146078 IJPL-209476 intellij.libraries.kotlinx.coroutines.slf4j, icu4j, jackson, ion as product module
...
GitOrigin-RevId: 6ec3fc109676944133e91aff3a82c51572bf4dbc
2025-10-04 19:03:47 +00:00
Vladimir Krivosheev
601a44264f
IJ-MR-175479 IJ-CR-146078 IJPL-209476 gson as product module
...
GitOrigin-RevId: 4e0ec5a56b91ff85cef4343c34f36e18adec1e9e
2025-10-04 19:03:47 +00:00
Vladimir Krivosheev
2e1adf2172
IJPL-209419 don't pack libs that are part of the product modules - extract intellij.libraries.ktor.network.tls
...
GitOrigin-RevId: cbaf43a02bcab4c390a89912366b0ff3580534f2
2025-10-03 15:08:44 +00:00
Dmitry Kozhevnikov
8c7f08fea9
[ask ai] LLM-18896: Implement tool-based approach for IDE actions and settings
...
GitOrigin-RevId: 7a5e76e5d132118ecb0757ef5a49c801cd2c46ed
2025-10-02 22:09:25 +00:00
Dmitry Kozhevnikov
30291ed874
[askai] LLM-18962: initial AskAI eval
...
GitOrigin-RevId: c3831ad69bdf16650d2cd021c548ff129f5b6b53
2025-10-02 22:09:25 +00:00
Dmitry Osinovskiy
713ca39aec
[aia-eval] fallback for calculating datasetName and chunkNamePrefix, because if they are empty there is a cryptic error
...
GitOrigin-RevId: 165bd73a0ecfbf8475b93d8ee597bdab4b8fbc6c
2025-10-01 22:48:32 +00:00
Dmitry Osinovskiy
fab7ea8eb3
[llm-vcs-grouped-diff, aia-eval] Further work on evaluation: import from old formats, run configuration, fixes for eval cases library UI
...
GitOrigin-RevId: b681f00d8e70e8e3d9bdceab018fe488fa86f98f
2025-10-01 01:57:56 +00:00
Dmitry Osinovskiy
67371290eb
[llm-vcs-grouped-diff, aia-eval] Draft of the evaluation for AI Diff
...
GitOrigin-RevId: dd0a53e08791ef70ec66de528a7474bbc096784b
2025-10-01 01:57:56 +00:00
Darya.Rovdo
58d762c838
LLM-20255 [ij next daemon] Add insights type, update dataset
...
GitOrigin-RevId: 77e9288c870d9ab750f07e653be81e20c8f729d0
2025-09-30 17:36:40 +00:00
Nikita.Lyubimov
18fbb9b96b
[ijnext] Add Self Review Verifier feature and Kotlin evaluation setup
...
GitOrigin-RevId: 6175cebd15577e605ffa66e872d0879b383f44f4
2025-09-29 13:04:46 +00:00
Ilia Kirianovskii
f43aa95086
[bazel] Update build files (IJI-3062)
...
GitOrigin-RevId: 98a67396a48bddc3d084cc93c50ae2f2017bfe8c
2025-09-29 00:11:15 +00:00
Roman Shevchenko
73fa5bfe55
Cleanup
...
- moving `CommandLineProcessor` extensions to more appropriate places
- dropping long-obsolete `ApplicationStarter#getCommandName`
- deprecating cumbersome `ApplicationStarter#getCommandNameFromExtension` in favor of constants
- typos
- formatting
GitOrigin-RevId: 2668c9f3474bd78fe97d9c614a2cf3faebbe9eee
2025-09-26 14:55:59 +00:00
Ilia Kirianovskii
6f8920da99
[bazel] Update build files (IJI-3062)
...
GitOrigin-RevId: 2394c1289e33945f7640f249b17cbf34b31fd695
2025-09-23 09:25:59 +00:00
Roman Vasiliev
57f329b3ab
[evaluation-plugin] LME-585 Support unittest framework
...
GitOrigin-RevId: 4ae4f9c6a4ca9b324db826eff611c869899f912d
2025-09-19 16:38:21 +00:00
Roman Vasiliev
c38e37c7b0
[evaluation-plugin] Remove the separate error code metric
...
GitOrigin-RevId: 1df6e29acad337554f16462bbb4dd57bb2bd494d
2025-09-19 16:38:21 +00:00
Roman Vasiliev
928615647c
[evaluation-plugin] LME-585 Adapt test name normalization for keras instances
...
GitOrigin-RevId: 058e99247ed8ffde8d65ead14a417219e5346471
2025-09-19 16:38:21 +00:00
Max Medvedev
dd99a8880a
[cleanup] more nullability, remove unused parameter
...
GitOrigin-RevId: f7fa51c1096e0ebd96c1f5235bcf57446f2f8617
2025-09-18 14:21:14 +00:00
Darya.Rovdo
c038fd0293
[ij-next-ai-diff] Manually check cluster metrics correctness
...
GitOrigin-RevId: af9b08228a7a1bcd8f2251f50f58c187ebec66a0
2025-09-16 12:45:36 +00:00
Darya.Rovdo
6a6fbcdb17
[ij-next-ai-diff] Refine cluster metrics
...
GitOrigin-RevId: b551287b0d0c1f51c0f5efdf8a9cb6dcbebe33e4
2025-09-16 12:45:36 +00:00
Darya.Rovdo
2bd3156947
[ij-next-ai-diff] Add sample weights to cluster metrics
...
GitOrigin-RevId: 6571c04c944df9029a03046364bfe83ace12ea8d
2025-09-16 12:45:36 +00:00
Darya.Rovdo
8d2f89d11e
[ij-next-ai-diff] Add basic cluster sklearn metrics
...
GitOrigin-RevId: 2e1bc63f73e0c64a50f13d724942a7f3f020a3f6
2025-09-16 12:45:36 +00:00
Nikolai Bogdanov
3ae86b2093
[llm-eval] LME-573 Proper gradle build for eval
...
[llm] LME-575 Remove git usages
[llm] LME-573 Fix gradle execution for aia eval
Merge-request: IJ-MR-175542
Merged-by: Nikolai Bogdanov <nikolai.bogdanov@jetbrains.com >
GitOrigin-RevId: b236549d6d1afc086db8b393d05bd5d648455b23
2025-09-15 09:05:57 +00:00
Roman Vasiliev
debdfa606a
[evaluation-plugin] Fix environment initialization problems during merge
...
- remove environment from merge command completely since the command doesn't need it
Merge-request: IJ-MR-175105
Merged-by: Roman Vasiliev <Roman.Vasiliev@jetbrains.com >
GitOrigin-RevId: a70131b5d9e49e9e4fecc2bcaead88821e9c3f93
2025-09-10 16:30:50 +00:00
Nikolai Bogdanov
0cb6973060
[llm] LME-573 Some adjustments for docker eval
...
[llm] LME-573 Mount bin folder to have the HTML report back from docker
[llm] LME-573 If tests are failed - score shouldn't be 1.
[llm] LME-573 If no tests are passed, all tests should be triggered
Merge-request: IJ-MR-175152
Merged-by: Nikolai Bogdanov <nikolai.bogdanov@jetbrains.com >
GitOrigin-RevId: f71a97835e0837477b110e1699662063ff0fa8a4
2025-09-10 15:10:52 +00:00
Roman Vasiliev
76479bf81d
[evaluation-plugin] LME-452 Quick-fix for the appeared problem with docker eel provider
...
GitOrigin-RevId: 6a6ed1b38343684387fde6fd69083e081c0b1dc9
2025-09-02 10:24:26 +00:00
Roman Vasiliev
5564fe0c23
[evaluation-plugin] Pass the test command with the dataset item
...
GitOrigin-RevId: 1ab92171b0c2ac4ba1c14314d0d8036eb2fb31a8
2025-09-02 10:24:26 +00:00
Roman Vasiliev
b9fec562cc
[evaluation-plugin] Implement a simple test runner for python
...
GitOrigin-RevId: 381ca34815eab4e495f51f1c8bdbb716931f8c4b
2025-08-28 11:43:38 +00:00
Roman Vasiliev
36e42137f5
[evaluation-plugin] Fix python interpreter setup
...
GitOrigin-RevId: 2c113da95e5667ec92d4bc8b546a667b3478c238
2025-08-28 11:43:38 +00:00
Roman Vasiliev
e37edf2fdd
[evaluation-plugin] Allow resolving instance id from environment variables
...
GitOrigin-RevId: 24eb529101c350c76aaf39fc8f54829759536ba6
2025-08-28 11:43:38 +00:00
Veselin Roganovic
b86145044d
[ai-completion] LLM: Add i18n plugin dependency and added my evaluation to evaluation plugin plugin-content.yaml
...
GitOrigin-RevId: 4ada722605cbbecfc7a1d7653e8abaeed163ae29
2025-08-27 18:42:58 +00:00
Veselin Roganovic
a65f08639e
[ai-completion] LLM-18323: Removed the name duplication in properties config. Now name is always just the value of the filter type enum
...
GitOrigin-RevId: 70facb152cb264aa888679b697b7378948e1e650
2025-08-27 18:42:58 +00:00
Veselin Roganovic
22b82459f3
[ai-completion] LLM-18323: Smaller updates, like formatting or wording changes
...
GitOrigin-RevId: 23ae7d734ade3a830269d10fa6ab5ed7b1ff467d
2025-08-27 18:42:58 +00:00
Veselin Roganovic
38da1656cf
[ai-completion] LLM-18323: Removed unused import from LookupFilter
...
GitOrigin-RevId: d10afbcd4f0c9314dee2c8d9f505b616ed1766a6
2025-08-27 18:42:58 +00:00
Veselin Roganovic
2da2269b9f
[ai-completion] LLM-18323: Cleaned up the successful cache hit removal, by introducing new concept LookupFilter and creating RemoveSuccessfulCacheHitsFilter and SessionLookupsFilter
...
GitOrigin-RevId: f6be45bb981a5c6d3aea6f05506725646a0c1277
2025-08-27 18:42:58 +00:00