How do I use custom opam repository?

[Blue-Berry]

Hey, thanks for making such an awesome project!

I'm fairly new to nix, so I've probably made some simple errors, I am trying to use the multi-package template to create a nix-shell with Jane street's fork of the ocaml compiler using their opam-repository (https://github.com/janestreet/opam-repository/tree/with-extensions).

To do this Imperatively using opam I would run the following:
opam repo add with-extensions https://github.com/janestreet/opam-repository.git#with-extensions
opam switch create 5.2.0+flambda2 --repos with-extensions,default

This is what I have as my nix flake:

{
  inputs = {
    opam-nix.url = "github:tweag/opam-nix";
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.follows = "opam-nix/nixpkgs";
  };
  outputs = {
    self,
    flake-utils,
    opam-nix,
    nixpkgs,
  } @ inputs:
    flake-utils.lib.eachDefaultSystem (system: let
      with-extensions = pkgs.fetchFromGitHub {
        owner = "janestreet";
        repo = "opam-repository";
        rev = "with-extensions";
        sha256 = "sha256-OPle+vvQ2InaqYCzIzGlVgT1pClMaN9c3YrHyYQW5xA=";
      };
      opam-repository = pkgs.fetchFromGitHub {
        owner = "ocaml";
        repo = "opam-repository";
        rev = "master";
        sha256 = "sha256-HUqY3MuNLUf2etT9X/ZiHQtMQqU+LxwXzrm3AlOP634=";
      };
      pkgs = nixpkgs.legacyPackages.${system};
      on = opam-nix.lib.${system};
      localPackagesQuery =
        builtins.mapAttrs (_: pkgs.lib.last)
        (on.listRepo (on.makeOpamRepo ./.));
      devPackagesQuery = {
        # You can add "development" packages here. They will get added to the devShell automatically.
        # ocaml-lsp-server = "*";
        # ocamlformat = "0.26.2+jst";
      };
      query =
        devPackagesQuery
        // {
          ## You can force versions of certain packages here, e.g:
          ## - force the ocaml compiler to be taken from opam-repository:
          ocaml-base-compiler = "5.2.0+flambda2";
          ## - or force the compiler to be taken from nixpkgs and be a certain version:
          # ocaml-system = "4.14.0";
          ## - or force ocamlfind to be a certain version:
          # ocamlfind = "1.9.2";
        };
      scope = on.buildOpamProject' {repos = [
        with-extensions
        opam-repository
      ];} ./. query;
      overlay = final: prev: {
        # You can add overrides here
      };
      scope' = scope.overrideScope overlay;
      # Packages from devPackagesQuery
      devPackages =
        builtins.attrValues
        (pkgs.lib.getAttrs (builtins.attrNames devPackagesQuery) scope');
      # Packages in this workspace
      packages =
        pkgs.lib.getAttrs (builtins.attrNames localPackagesQuery) scope';
    in {
      legacyPackages = scope';

      inherit packages;

      ## If you want to have a "default" package which will be built with just `nix build`, do this instead of `inherit packages;`:
      # packages = packages // { default = packages.<your default package>; };

      devShells.default = pkgs.mkShell {
        inputsFrom = builtins.attrValues packages;
        buildInputs =
          devPackages
          ++ [
            # You can add packages from nixpkgs here
          ];
      };
    });
}

I don't get any errors running the nix-shell but I don't seem to have any ocaml binaries (eg. ocaml, ocamlc, ocamlopt...).

Any help would be greatly appreciated, thanks!

[balsoft]

This seems to be correctly done. Maybe you need to be running nix develop instead of nix-shell?

[balsoft]

Ok, so this led me to discover a bug in opam-nix; thanks for this! The code for opam file search didn't recognize the package/${name}.${version} format that Jane Street's opam-repository used. I've fixed this, but I think it's distinct from your problem.

[balsoft]

Actually, can you give a link to your code, or some other code that requires JaneStreet's repo? I'd like to reproduce your issue.

[balsoft]

Ah, actually, the reason for your error is that ocaml-base-compiler doesn't have a version called 5.2.0+flambda2. If I specify ocaml-variants = "5.2.0+flambda2"; that seems to at least resolve the versions correctly.

[balsoft]

I've tinkered with this for a bit and it seems that the with-extensions repo is pretty hacky sadly. I've had to add some overrides to get ocaml-variants building, and then build ocamlformat=0.26.2+jst with it:

{
  inputs = {
    opam-nix.url = "github:tweag/opam-nix";
    opam-nix.inputs.opam-repository.follows = "opam-repository";
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.follows = "opam-nix/nixpkgs";
    with-extensions.url = "github:janestreet/opam-repository/with-extensions";
    with-extensions.flake = false;
    opam-repository.url = "github:ocaml/opam-repository";
    opam-repository.flake = false;
  };
  outputs =
    {
      self,
      flake-utils,
      opam-nix,
      nixpkgs,
      with-extensions,
      opam-repository,
    }@inputs:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        on = opam-nix.lib.${system};
        localPackagesQuery =
          builtins.mapAttrs (_: pkgs.lib.last)
          (on.listRepo (on.makeOpamRepo ./.));
        devPackagesQuery = {
          # You can add "development" packages here. They will get added to the devShell automatically.
          # ocaml-lsp-server = "*";
          ocamlformat = "0.26.2+jst";
        };
        query = devPackagesQuery // {
          ## You can force versions of certain packages here, e.g:
          ## - force the ocaml compiler to be taken from opam-repository:
          ocaml-variants = "5.2.0+flambda2";
          ## - or force the compiler to be taken from nixpkgs and be a certain version:
          # ocaml-system = "4.14.0";
          ## - or force ocamlfind to be a certain version:
          # ocamlfind = "1.9.2";
        };
        scope = on.buildOpamProject' {
          repos = [
            opam-repository
            with-extensions
          ];
        } ./. query;
        overlay = final: prev: {
          # You can add overrides here
          init-dune = prev.init-dune.overrideAttrs (oa: {
            nativeBuildInputs = oa.nativeBuildInputs ++ [ pkgs.ocaml-ng.ocamlPackages_4_14.ocaml ];
          });
          init-menhir = prev.init-menhir.overrideAttrs (oa: {
            nativeBuildInputs = oa.nativeBuildInputs ++ [ pkgs.ocaml-ng.ocamlPackages_4_14.ocaml ];
            preBuild = "mkdir -p $out/init_deps/bin; ln -s ${final.init-dune}/init_deps/bin/dune $out/init_deps/bin";
          });
          ocaml-variants = prev.ocaml-variants.overrideAttrs (oa: {
            preBuild = ''
              mkdir -p $out/init_deps/bin
              ln -s ${final.init-menhir}/init_deps/bin/menhir ${final.init-dune}/init_deps/bin/dune $out/init_deps/bin
              export PATH=$PATH:${final.init-menhir}/init_deps/bin:${final.init-dune}/init_deps/bin
              sed -i "s|/usr/bin/env|${pkgs.coreutils}/bin/env|g" Makefile Makefile.* ocaml/Makefile.*
            '';
            nativeBuildInputs = oa.nativeBuildInputs ++ [
              pkgs.autoconf
              pkgs.which
              pkgs.rsync
            ];
          });
          stdio = prev.stdio.overrideAttrs (oa: {
            buildInputs = [ final.base final.sexplib0 ];
          });
          base = prev.base.overrideAttrs (oa: {
            buildInputs = [ final.ocaml_intrinsics_kernel final.sexplib0 ];
          });
        };
        scope' = scope.overrideScope overlay;
        # Packages from devPackagesQuery
        devPackages = builtins.attrValues (pkgs.lib.getAttrs (builtins.attrNames devPackagesQuery) scope');
        # Packages in this workspace
        packages = pkgs.lib.getAttrs (builtins.attrNames localPackagesQuery) scope';
      in
      {
        legacyPackages = scope';

        inherit packages;

        ## If you want to have a "default" package which will be built with just `nix build`, do this instead of `inherit packages;`:
        # packages = packages // { default = packages.<your default package>; };

        devShells.default = pkgs.mkShell {
          inputsFrom = builtins.attrValues packages;
          buildInputs = devPackages ++ [
            # You can add packages from nixpkgs here
          ];
        };
      }
    );
}

(I also cleaned up the dependency fetching a bit)

If the thing you're actually building breaks, it may be because you need to force-add some dependencies, look at stdio and base in the overlay for an example.

Hope this helps!

[balsoft]

Oh, you don't have an opam file in your repo? That would explain why you're not getting ocaml in scope, because technically nothing requested it to be in scope :)

[Blue-Berry]

Ahh right, that makes a lot of sense! Well everything seems to be working great so far, thank you so much for all the help, I really appreciate it! I learnt a lot.

[balsoft]

And thank you for helping find a bug in opam-nix!