host_volume/volume/volume_mount does not work with java driver

[martingms]

Nomad version

Nomad v0.10.1 (0d4e5d949fe073c47a947ea36bfef31a3c49224f)

Operating system and Environment details

Ubuntu 18.04.3 LTS

Issue

host_volume/volume/volume_mount does not work with java driver. The directory is created in the allocation, but not mounted/available.

Reproduction steps

Add host_volume in client config, with corresponding volume and volume_mount in job spec, using the "java" driver. Put a file in the directory, and attempt to find it/read it from within the java program.

Job file (if appropriate)

client.hcl:

data_dir = "/var/lib/nomad"                                                         
                                                                                 
client {                                                                         
  enabled = true                                                                 
                                                                                 
  ...                                                                            
                                                                                 
  host_volume "data" {                                                           
    path      = "/data"                                                          
    read_only = false                                                            
  }                                                                              
}

my-job.nomad:

job "my-job" {                                                                      
  type = "service"                                                                  
                                                                                    
  group "my-group" {                                                                
    count = 1                                                                       
                                                                                    
    volume "data" {                                                                 
      type      = "host"                                                            
      read_only = false                                                             
      source    = "data"                                                            
    }                                                                               
                                                                                    
    task "my-task" {                                                                
      driver = "java"                                                               
                                                                                    
      config {                                                                      
        jar_path = "..."                                                            
        jvm_options = [ ... ]                                                       
        args = [ ... ]                                                              
      }                                                                             
                                                                                    
      artifact {                                                                    
        source = "..."                                                              
                                                                                    
        options = {                                                                 
          checksum = "..."                                                          
        }                                                                           
      }                                                                             
                                                                                    
      volume_mount {                                                                
        volume      = "data"                                                        
        destination = "/data"                                                       
        read_only   = false                                                         
      }                                                                             
                                                                                    
      resources { ... }                                                             
    }                                                                               
  }                                                                                 
} 

Nomad Client logs (if appropriate)

Nomad Server logs (if appropriate)

Seems like nothing about this shows up in the logs

[martingms]

Seems like this might be an issue with the exec-driver as well, based on discussions on gitter.

[tgross]

Thanks for reporting this @martingms! Yeah, that definitely looks like a documentation and validation issue.

[martingms]

Hi @tgross thanks for the quick reply :) What do you mean with documentation and validation issue? Is host_volume/volume_mount not supposed to work with the java driver?

[tgross]

I did some digging and chatted with one of my colleagues and it turns out I'm wrong -- this should work. Sorry about the confusion. Going to mark this as a bug.

[rkettelerij]

Seems like this might be an issue with the exec-driver as well, based on discussions on gitter.

I've got host-volumes working using the exec driver FWIW.

[sfs77]

I have same issue , looking for a resolution

[raypettersen]

This is crucial for us, excellent timing putting this into 0.11.1 👍

[raypettersen]

I just tested 0.11.1, and if I'm not totally lost (been a long day..) it seems that java_driver mount is now working?

[tgross]

This person reported the same: #6709 (comment). I never got a chance to actually reproduce this issue, but because two people reported it in this thread I'm hesitant to close this out until I've done so. I have this on my plate for the next patch release, so lemme take a quick look again in the next day or so to confirm.

[raypettersen]

Let me know if I can try/test anything. The puzzling thing is that I started off with the java_driver, didn't work. Tested with docker, and it worked so I just waited for 0.11.1, but I see this has been moved to 0.11.2. Either way, I just restarted the java-job I had and it seems like the storage popped up. It is not showing in the UI -> Files though. Going through the new "exec" is showing the directory content.

[tgross]

It is not showing in the UI -> Files though. Going through the new "exec" is showing the directory content.

The UI uses the same API as nomad alloc fs. So that sounds like a case of #7365?

[raypettersen]

Certainly does. I tested another java-job and it looks to be the same behavior.

[raypettersen]

Short follow-up, I've had my java-app write to the store successfully. The store is an EFS share that I've mounted locally on all my nomad clients. Seems to work just fine. I feel this issue should be closed (I was personally waiting for it to be resolved, pure luck that I did some extra tests and found it already working), but as you say @tgross it would be good to do some extra checks.

[tgross]

Ok, I was able to confirm that host volumes were working with both exec and java drivers with a few recent versions of Nomad. At this point I'm going to close this out and if anyone is running into this and can provide some more context we can always revisit. Thanks folks!

My relevant client config:

client {
  host_volume "srv" {
    path      = "/path/to/my/stuff"
    read_only = true
  }
}

My java task, using the host volume as the source of the jar we're running:

job "example" {
  datacenters = ["dc1"]

  group "group" {
    volume "srv" {
      type      = "host"
      source    = "srv"
      read_only = true
    }

    task "task" {
      driver = "java"

      config {
        jar_path    = "/local/srv/services-kotlin-0.0.1-all.jar"
        jvm_options = ["-Xmx1024m"]
      }

      volume_mount {
        volume      = "srv"
        destination = "/local/srv"
      }

      env {
        PORT = "9001"
      }

      resources {
        cpu    = 1024
        memory = 1024
      }

    }
  }
}

My exec task, which I verified had the volume mounted via nomad alloc exec :alloc_id /bin/sh

job "example" {
  datacenters = ["dc1"]

  group "group" {
    volume "srv" {
      type      = "host"
      source    = "srv"
      read_only = true
    }

    task "task" {
      driver = "exec"

      config {
        command = "sleep"
        args    = ["3600"]
      }

      volume_mount {
        volume      = "srv"
        destination = "/local/srv"
      }

    }
  }
}

[raypettersen]

Thanks @tgross !

[tgross]

@martingms we had an internal discussion of another bug and I think the original problem you've had here isn't specific to the java driver but related to this #7796. I've got that on my plate for further investigation.

[martingms]

Okay! It's been a while, but I'll test if upgrading to the latest version changes anything. Following #7796 as well.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.