Support Gradle configuration cache on multiple JVMs

Gradle configuration cache requires task state to be round-trip serializable. As outlined at #987, this is a signficant constraint on plugin developers and it slows down the end-user experience of change-test-change-test on a single JVM daemon. We built a workaround (JvmLocalCache) that complies with Gradle's requirement by making our tasks round-trip serializable without requiring much work from plugin developers or the end user's machine, but it only works within a single JVM.

To work on multiple JVMs, we need to

make every step round-trip serializable
remove JvmLocalCache

FYI Gradle 8 will do a full roundtrip on the first invocation with CC enabled.
See work in progress here gradle/gradle#21985 (comment)
The goal is for this to be the default in 8.0.

Full roundtrip will work great on the same JVM, so Spotless will still work and not work under the same conditions it does currently. EDIT: Compatibility with configuration cache within a single JVM on Gradle 8 is confirmed.

I think we now have a way to incrementally build this until we can eventually remove our workaround. See:

Start making each FormatterStep roundtrip serializable. #1945

In this comment, I'm going to maintain a list of remaining TODOs before we will fully support Gradle's configuration cache model without any workarounds. If you'd like to help: comment in this issue when you start on a given step, follow-up with a PR (draft is fine) and I will update the checklist below

remove LineEnding.GIT_ATTRIBUTES (at least for Gradle) because very difficult to make it serializable Support Gradle configuration cache on multiple JVMs #1274
- we can keep GIT_ATTRIBUTES_FAST_ALLSAME which is the default anyway
lib(generic.PipeStepPair) Serializable refactor pipe/fence #1954
lib(generic.EndWithNewlineStep)
lib(generic.IndentStep) Start making each FormatterStep roundtrip serializable. #1945
lib(generic.Jsr223Step) Make all steps with JarState serializable #2039
lib(generic.LicenseHeaderStep) Port all stragglers except npm #2132
lib(generic.NativeCmdStep) Port all stragglers except npm #2132
lib(generic.ReplaceRegexStep)
lib(generic.ReplaceStep)
lib(generic.TrimTrailingWhitespaceStep)
lib(groovy.RemoveSemicolonsStep)
lib(antlr4.Antlr4FormatterStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(cpp.ClangFormatStep) Serializable foreign exe #1955
extra(cpp.EclipseFormatterStep) Make the Equo-based steps round-trip serializable #1950
lib(gherkin.GherkinUtilsStep) Make all steps with JarState serializable #2039
extra(groovy.GrEclipseFormatterStep) Make the Equo-based steps round-trip serializable #1950
lib(java.GoogleJavaFormatStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(java.ImportOrderStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(java.PalantirJavaFormatStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(java.RemoveUnusedImportsStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
extra(java.EclipseJdtFormatterStep) Make the Equo-based steps round-trip serializable #1950
lib(java.FormatAnnotationsStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(java.CleanthatJavaStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(json.gson.GsonStep) Make all steps with JarState serializable #2039
lib(json.JacksonJsonStep) Make all steps with JarState serializable #2039
lib(json.JsonSimpleStep) Make all steps with JarState serializable #2039
lib(json.JsonPatchStep) Make all steps with JarState serializable #2039
lib(kotlin.KtLintStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(kotlin.KtfmtStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(kotlin.DiktatStep) Start making each FormatterStep roundtrip serializable. #1945
lib(markdown.FreshMarkStep) Make all steps with JarState serializable #2039
lib(markdown.FlexmarkStep) Make all steps with JarState serializable #2039
lib(npm.EslintFormatterStep) Make the npm steps roundtrip serializable #2135
lib(npm.PrettierFormatterStep) Make the npm steps roundtrip serializable #2135
lib(npm.TsFmtFormatterStep) Make the npm steps roundtrip serializable #2135
lib(pom.SortPomStepStep) Make all steps with JarState serializable #2039
lib(protobuf.BufStep) Serializable foreign exe #1955
lib(python.BlackStep) Serializable foreign exe #1955
lib(biome.BiomeStep) Make the npm steps roundtrip serializable #2135
lib(scala.ScalaFmtStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
lib(sql.DBeaverSQLFormatterStep) Port all stragglers except npm #2132
extra(wtp.EclipseWtpFormatterStep) Serializable refactor legacy eclipse steps #1952
lib(yaml.JacksonYamlStep) Make all steps with JarState serializable #2039

Make the Equo-based steps round-trip serializable (#1950 towards #1274)

I checked npm.EslintFormatterStep, npm.PrettierFormatterStep, npm.TsFmtFormatterStep and sql.DBeaverSQLFormatterStep. They all use StepHarness for testing which has roundtrip serialization testing enabled by default. Since their tests do not fail, it should mean they are already serializable, am I correct?

For NativeCmdStep I created a PR #2108 which adds a test using StepHarness.

@Goooler

Thanks mostly to @Goooler's tireless dedication, we have finally shipped:

plugin-gradle 3.0.0.BETA1
plugin-maven 2.44.0.BETA1

These contain a bazillion fixes. Please test and let us know if you encounter any issues!

JUnit build updated ✅

All working fine for me as well, except I used

removeUnusedImports("cleanthat-javaparser-unnecessaryimport")

as seen in the docs. That one seems to not support the config cache yet:

Could not load the value of field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.
> Could not load the value of field `excluded` of `com.diffplug.spotless.java.CleanthatJavaStep` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.

I just removed it as it looks like googleJavaFormat already takes care of that.

@bddckr

I get a similar error to @bddckr if I use any prettier steps, and only while using the Gradle Configuration Cache, like:

Could not load the value of field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

or:

Could not load the value of field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

I've worked around it by declaring Spotless incompatible with the configuration cache, like:

tasks.withType<SpotlessTask>().configureEach {
  notCompatibleWithConfigurationCache(
      "Spotless 7.0.0.BETA1 with prettier steps does not support the configuration cache.")
}

FWIW, I've upgraded to 7.0.0.BETA2 in all my repositories and everything seems to be in order! Great job!

With 7.0.0.BETA4 I think this is now just about fully solved. Last chance for feedback before these APIs all get baked!

Feedback wanted! Ending beta period for 7.0 #2311

Just filed a regression on #2318

CC issue w/ prettier on 7.0.0BETA4: #2372

build: Windows에서 spotlessApply가 shell script를 CRLF로 바꾸지 않게 한다

Navigation Menu

Repository navigation

Support Gradle configuration cache on multiple JVMs #1274

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support Gradle configuration cache on multiple JVMs #1274

Description

Activity

eskatos commented on Sep 28, 2022

nedtwigg commented on Sep 28, 2022

nedtwigg commented on Dec 4, 2023

lehmanju commented on May 6, 2024

nedtwigg commented on Jun 4, 2024

marcphilipp commented on Jun 5, 2024

bddckr commented on Jun 5, 2024

ZachCallear commented on Jun 13, 2024

davidburstrom commented on Sep 18, 2024

nedtwigg commented on Oct 24, 2024

ZacSweers commented on Oct 28, 2024

cloudshiftchris commented on Dec 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions