Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BOM / dependency management support #3924

Merged
merged 43 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
910e8a4
Remove trailing space
alexarchambault Nov 8, 2024
4f5f070
NIT Use allIvyDeps task
alexarchambault Nov 8, 2024
3e8643d
WIP Add BOM support
alexarchambault Nov 8, 2024
f796d0c
Merge branch 'main' into pr/bom-support
alexarchambault Nov 13, 2024
bc5e1b3
more
alexarchambault Nov 13, 2024
425b4aa
Merge branch 'main' into pr/bom-support
alexarchambault Nov 19, 2024
732c659
more
alexarchambault Nov 19, 2024
17259c0
fixup
alexarchambault Nov 19, 2024
a8b86e3
more
alexarchambault Nov 19, 2024
67ff17a
Merge branch 'main' into pr/bom-support
alexarchambault Nov 20, 2024
067d84d
more
alexarchambault Nov 20, 2024
17c3084
more
alexarchambault Nov 21, 2024
c143182
Merge branch 'main' into pr/bom-support
alexarchambault Nov 21, 2024
a45ddd6
fix
alexarchambault Nov 21, 2024
d601c44
Revert UnidocModule stuff
alexarchambault Nov 21, 2024
20f59c5
Merge branch 'main' into pr/bom-support
alexarchambault Nov 21, 2024
8c00ccb
Merge branch 'main' into pr/bom-support
alexarchambault Nov 21, 2024
f53fd8d
tweaking
alexarchambault Nov 21, 2024
c549b21
alexarchambault Nov 21, 2024
06b0c5b
more
alexarchambault Nov 21, 2024
1828975
fmt
alexarchambault Nov 21, 2024
c9054a3
more
alexarchambault Nov 21, 2024
3dbc225
Starting to look good 🤔
alexarchambault Nov 21, 2024
b260f6f
Merge branch 'main' into pr/bom-support
alexarchambault Nov 21, 2024
e13a102
Merge branch 'main' into pr/bom-support
alexarchambault Nov 22, 2024
9c75f73
Merge branch 'main' into pr/bom-support
alexarchambault Nov 22, 2024
97cb113
more
alexarchambault Nov 22, 2024
13fdbca
Merge branch 'main' into pr/bom-support
alexarchambault Nov 25, 2024
8e28493
more
alexarchambault Nov 25, 2024
7950d4f
fmt
alexarchambault Nov 25, 2024
4f78fe8
Merge branch 'main' into pr/bom-support
alexarchambault Nov 27, 2024
03857e0
Merge branch 'main' into pr/bom-support
alexarchambault Nov 27, 2024
745b500
more - needs clean-up, only checking if CI passes
alexarchambault Nov 27, 2024
26b5c54
Merge branch 'main' into pr/bom-support
alexarchambault Nov 28, 2024
f286477
Merge branch 'main' into pr/bom-support
alexarchambault Dec 2, 2024
8aa498a
more
alexarchambault Dec 2, 2024
61b6061
fmt
alexarchambault Dec 2, 2024
5bd61aa
Merge branch 'main' into pr/bom-support
alexarchambault Dec 3, 2024
73f5c10
rework / add doc, remove parentDep stuff
alexarchambault Dec 3, 2024
e5fe19f
Merge branch 'main' into pr/bom-support
alexarchambault Dec 4, 2024
169a747
Address review comments
alexarchambault Dec 4, 2024
2a5ca63
nit
alexarchambault Dec 4, 2024
26116a9
Minor refactoring
alexarchambault Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 200 additions & 0 deletions integration/feature/bom/resources/build.mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
package build
lefou marked this conversation as resolved.
Show resolved Hide resolved

import mill._
import mill.scalalib._
import mill.scalalib.publish._

trait TestPublishModule extends PublishModule {
def pomSettings = PomSettings(
description = artifactName(),
organization = "com.lihaoyi.mill-tests",
url = "https://github.com/com-lihaoyi/mill",
licenses = Seq(License.`Apache-2.0`),
versionControl = VersionControl.github("com-lihaoyi", "mill"),
developers = Nil
)
def publishVersion = "0.1.0-SNAPSHOT"
}

object bom extends Module {
object placeholder extends JavaModule with TestPublishModule {
def bomDeps = Agg(
ivy"com.google.cloud:libraries-bom:26.50.0"
)
def ivyDeps = Agg(
ivy"com.google.protobuf:protobuf-java:_"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we leave out the :_ and just use ivy"com.google.protobuf:protobuf-java"? That would make our syntax consistent with Maven and Gradle, both of which just leave out the version segment without any placeholder

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving out the version entirely would also be consistent with how we currently handle missing versions for contrib plugins. We could remove the special handling and instead auto-include a BOM configuring all contrib plugins.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Things like ivy"com.google.protobuf:protobuf-java" are now accepted.

Note that missing version translates to the version field being "_", which still acts as a placeholder for versions (that makes better error mssages IMO).

So we have

ivy"com.google.protobuf:protobuf-java" == ivy"com.google.protobuf:protobuf-java:_"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ivy"com.google.protobuf:protobuf-java:_" could be arguably read as "any version will do", whereas the missing versions rather means, "version needs to be configured elsewhere and is not provided here".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to make sure the _ used in coursier does not leak into Mill, which is probably not possible, since we forward coursier output in error messages and ivyDepsTree to name some. Out of curiosity, were there any limitations forcing you to choose _ as a marker? Would it be possible to chnage it in coursier?

Copy link
Collaborator Author

@alexarchambault alexarchambault Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it could still be changed in coursier, but for what? I used a marker so that the dependency parsers in coursier don't need much change (compared to accepting things like org:name)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the user experience? For the consistency with Maven and Gradle?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for avoidance of extra work in Mill, since we already use _ as a ubiquitous wildcard, not as a marker for absence. We only have the choise to

  1. convert all dependency representation coming from coursier, which is tedious and leads to a lot of boilerplate code
  2. use _ and off-load the inconsistence to the user
  3. ignore and make an inconsistent UX

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So how we want to proceed now? Since Dep does not fully abstract over the inner dep: Dependency, the version is often used as is, and the forwarder Dep.verison wasn't touched in this PR, so we end up with a mix-up of defined or empty or _ versions.

)

object dependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
placeholder
)
}

object subDependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
dependee
)
}

object check extends JavaModule {
def ivyDeps = Agg(
ivy"com.google.protobuf:protobuf-java:_"
)
}
}

object versionOverride extends JavaModule with TestPublishModule {
def bomDeps = Agg(
ivy"com.google.cloud:libraries-bom:26.50.0"
)
def ivyDeps = Agg(
ivy"com.thesamet.scalapb:scalapbc_2.13:0.9.8"
)

object dependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
versionOverride
)
}

object subDependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
dependee
)
}

object check extends JavaModule {
def ivyDeps = Agg(
ivy"com.thesamet.scalapb:scalapbc_2.13:0.9.8"
)
}
}

object invalid extends Module {
object exclude extends JavaModule {
def bomDeps = Agg(
ivy"com.google.cloud:libraries-bom:26.50.0".exclude(("foo", "thing"))
)
}
}
}

object parent extends JavaModule with TestPublishModule {
def parentDep = Some(ivy"org.apache.spark:spark-parent_2.13:3.5.3")
def ivyDeps = Agg(
ivy"org.apache.commons:commons-compress:_"
)

object dependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
parent
)
}

object subDependee extends JavaModule with TestPublishModule {
def moduleDeps = Seq(
dependee
)
}

object scala extends ScalaModule with TestPublishModule {
def scalaVersion = _root_.scala.util.Properties.versionNumberString
def parentDep = Some(ivy"org.apache.spark::spark-parent:3.5.3")
def ivyDeps = Agg(
ivy"org.apache.commons:commons-compress:_"
)
}

object invalid extends Module {
object exclude extends JavaModule {
def parentDep = Some(
ivy"org.apache.spark:spark-parent_2.13:3.5.3".exclude(("foo", "thing"))
)
}
}
}

object depMgmt extends JavaModule with TestPublishModule {
def ivyDeps = Agg(
ivy"com.thesamet.scalapb:scalapbc_2.13:0.9.8"
)
def dependencyManagement = Agg(
ivy"com.google.protobuf:protobuf-java:4.28.3"
)

object transitive extends JavaModule with TestPublishModule {
def moduleDeps = Seq(depMgmt)
}

object extraExclude extends JavaModule with TestPublishModule {
def ivyDeps = Agg(
ivy"com.lihaoyi:cask_2.13:0.9.4"
)
def dependencyManagement = Agg(
// The exclude should be automatically added to the dependency above
// thanks to dependency management, but the version should be left
// untouched
ivy"com.lihaoyi:cask_2.13:0.9.3"
.exclude(("org.slf4j", "slf4j-api"))
)

object transitive extends JavaModule with TestPublishModule {
def moduleDeps = Seq(extraExclude)
}
}

object exclude extends JavaModule with TestPublishModule {
def ivyDeps = Agg(
ivy"com.lihaoyi:cask_2.13:0.9.4"
)
def dependencyManagement = Agg(
ivy"org.java-websocket:Java-WebSocket:1.5.2"
.exclude(("org.slf4j", "slf4j-api"))
)

object transitive extends JavaModule with TestPublishModule {
def moduleDeps = Seq(exclude)
}
}

object onlyExclude extends JavaModule with TestPublishModule {
def ivyDeps = Agg(
ivy"com.lihaoyi:cask_2.13:0.9.4"
)
def dependencyManagement = Agg(
ivy"org.java-websocket:Java-WebSocket:_"
.exclude(("org.slf4j", "slf4j-api"))
)

object transitive extends JavaModule with TestPublishModule {
def moduleDeps = Seq(onlyExclude)
}
}

object invalid extends Module {
object transitive extends JavaModule {
def dependencyManagement = {
val dep = ivy"org.java-websocket:Java-WebSocket:1.5.3"
Agg(
dep.copy(
dep = dep.dep.withTransitive(false)
)
)
}
}
}

object placeholder extends JavaModule with TestPublishModule {
def ivyDeps = Agg(
ivy"com.google.protobuf:protobuf-java:_"
)
def dependencyManagement = Agg(
ivy"com.google.protobuf:protobuf-java:4.28.3"
)

object transitive extends JavaModule with TestPublishModule {
def moduleDeps = Seq(placeholder)
}
}
}
Loading
Loading