Failed to get filesystem from image: chown /etc/gshadow: operation not permitted

[Gwulior]

Actual behavior
I use Kaniko: "gcr.io/kaniko-project/executor:debug-v0.19.0"
and GitLab Kubernetes Runner to build my Docker image and send it to AWS ECR.
During executing of such Dockerfile:

FROM openjdk:11-jre
RUN useradd -u 230000 unpriv
RUN mkdir /workspace
ARG COMPONENT
ARG VERSION
ARG FOLDER
COPY ./$FOLDER/target/${COMPONENT}-${VERSION}.jar /workspace
RUN chown  -R unpriv /workspace
USER unpriv
WORKDIR /workspace
EXPOSE  8080
CMD [ "java", "-Dspring.profiles.active=cloud", "-jar",  "workspace/${COMPONENT}-${VERSION}.jar" ]

I get such log with error:

INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Retrieving image manifest openjdk:11-jre     
 INFO[0003] Retrieving image manifest openjdk:11-jre     
 INFO[0004] Built cross stage deps: map[]                
 INFO[0004] Retrieving image manifest openjdk:11-jre     
 INFO[0005] Retrieving image manifest openjdk:11-jre     
 INFO[0007] Unpacking rootfs as cmd RUN useradd -u 230000 unpriv requires it. 
 error building image: error building stage: failed to get filesystem from image: chown /etc/gshadow: operation not permitted
sh: exec: line 6: /bin/bash: not found
00:00
 ERROR: Job failed: command terminated with exit code 1

Expected behavior
Previously I ran such build using GitLab Docker runner and it worked. Also after this problem I started my own GitLab docker runner and got such successful log:

Running before_script and script
00:47
 $ echo "{ \"credsStore\": \"ecr-login\" }" > /kaniko/.docker/config.json
 $ /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --build-arg COMPONENT=$COMPONENT  --build-arg VERSION=$VERSION --build-arg FOLDER=$FOLDER --destination $DOCKER_REGISTRY_URL/$COMPONENT:$CI_COMMIT_SHA
 INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Retrieving image manifest openjdk:11-jre     
 INFO[0003] Retrieving image manifest openjdk:11-jre     
 INFO[0004] Built cross stage deps: map[]                
 INFO[0004] Retrieving image manifest openjdk:11-jre     
 INFO[0005] Retrieving image manifest openjdk:11-jre     
 INFO[0006] Unpacking rootfs as cmd RUN useradd -u 230000 unpriv requires it. 
 INFO[0014] Taking snapshot of full filesystem...        
 INFO[0018] Resolving paths                              
 INFO[0019] RUN useradd -u 230000 unpriv                 
 INFO[0019] cmd: /bin/sh                                 
 INFO[0019] args: [-c useradd -u 230000 unpriv]          
 INFO[0019] Taking snapshot of full filesystem...        
 INFO[0019] Resolving paths                              
 INFO[0021] RUN mkdir /workspace                         
 INFO[0021] cmd: /bin/sh                                 
 INFO[0021] args: [-c mkdir /workspace]                  
 INFO[0021] Taking snapshot of full filesystem...        
 INFO[0021] Resolving paths                              
 INFO[0022] ARG COMPONENT                                
 INFO[0022] ARG VERSION                                  
 INFO[0022] ARG FOLDER                                   
 INFO[0022] COPY ./$FOLDER/target/${COMPONENT}-${VERSION}.jar /workspace 
 INFO[0022] Resolving paths                              
 INFO[0022] Taking snapshot of files...                  
 INFO[0022] RUN chown  -R unpriv /workspace              
 INFO[0022] cmd: /bin/sh                                 
 INFO[0022] args: [-c chown  -R unpriv /workspace]       
 INFO[0022] Taking snapshot of full filesystem...        
 INFO[0022] Resolving paths                              
 INFO[0023] USER unpriv                                  
 INFO[0023] cmd: USER                                    
 INFO[0023] WORKDIR /workspace                           
 INFO[0023] cmd: workdir                                 
 INFO[0023] Changed working directory to /workspace      
 INFO[0023] EXPOSE  8080                                 
 INFO[0023] cmd: EXPOSE                                  
 INFO[0023] Adding exposed port: 8080/tcp                
 INFO[0023] CMD [ "java", "-Dspring.profiles.active=cloud", "-jar",  "workspace/${COMPONENT}-${VERSION}.jar" ] 
Running after_script
00:02
Saving cache
00:02
Uploading artifacts for successful job
00:01
 Job succeeded

To Reproduce
Steps to reproduce the behavior:

1. Define build step as in provided GitLab CI definition
2. Add Dockerfile similar to provided one
3. Setup GitLab Kubernetes Shared Runner
4. Start the job

Additional Information
GitLab build:

.docker:
  stage: package
  cache: {}
  image:
    name: 'gcr.io/kaniko-project/executor:debug-v0.19.0'
    entrypoint: [""]
  script:
    - 'echo "{ \"credsStore\": \"ecr-login\" }" > /kaniko/.docker/config.json'
    - '/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --build-arg COMPONENT=$COMPONENT  --build-arg VERSION=$VERSION --build-arg FOLDER=$FOLDER --destination $DOCKER_REGISTRY_URL/$COMPONENT:$CI_COMMIT_SHA'
  only:
    refs:
      - master
  except:
    - schedules

Is it known problem with Kaniko and GitLab Kubernetes Runner?

[kwinkel]

Having the same problem with GitLab Kubernetes runner
might be related to #778

[danielhass]

@Gwulior - I ran into a similar issue with a runner on a k8s cluster with restrictive Pod Security Policies in place. @kwinkel this points into the same direction as the issue you mentioned.

Do you have any PSP activated in your environment?

[kwinkel]

@danielhass Yes, we have a restricted PSP. Currently in contact with our internal administration to activate some linux capabilities (see #778) at the PSP.

[Gwulior]

@danielhass Yes we have restricted PSP, the spec is here:

[danielhass]

@kwinkel would it be possible to share with us your PSP definition if you get it to work with the capabilities from #778?

[kwinkel]

Our administration team added the capabilities DefaultAddCapabilities from #778 to our PSP for the gitlab-runner, which has resolved this issue.

Docs: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#capabilities

[tejal29]

Thanks @kwinkel got confirming this!
I will close this issue now.