A common issue we see with self-hosted runners is that they can leave behind files that were created or modified by the action. This is because the action runs in a container and the container is using a root
user to do its work.
The GitHub documentation says to run the the runner service as root
as well, to have the most compatibility with most runners. This is not a good idea, as it can lead to security issues, so a lot of people run the runner service as a non-root user. This can lead to permission issues when the action touches files in the workspace directory that get’s mounted.
Examples
One common example is the super-linter action. Depending on the checks that run, it can touch files on disc that then get owned by the root
user.
Another example is running the entire job inside of a container, with using the keyword container
on the job level:
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">self-hosted</span>
<span class="na">container</span><span class="pi">:</span> <span class="s">ubuntu:22.04</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span> <span class="s">echo "Hello World"</span>
The checkout action will create a .git
directory in the workspace directory with the repo contents, which will be owned by the root
user as the ubuntu-22.04
container runs as root
. The job itself will complete just fine, but the next time you run another job for this repository on the same runner, the checkout action will try to cleanup the $GITHUB_WORKSPACE directory to get a clean starting point, and will fail with a permission error since the job is not running as root, but as the non-root user the runner service is executing under.
There are multiple ways to tackle this issue:
- Prevent it from happening: Run the runner service as
root
and run the job asroot
as well. This is recommended by GitHub to prevent this from happening, that is how the service has been designed. Most people I talk with do not agree since the user has to much permissions and it can lead to security issues. - Change the action to not run as root, which is not realistic when using actions from the public marketplace.
- Get the user to cleanup inside of the container, or add a cleanup action in their job.
- Add cleanup configuration on the container level configuration in the runner
- Add cleanup configuration on the runner configuration itself
- Do not run the container with any persistence: run as
ephemeral
, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse.
I’ll go over the last 4 options in this post.
Get the user to cleanup inside of the container, or add a cleanup action in their job.
You could hunt for the jobs that cause this issue, and ‘ask’ the user to add a cleanup action in the job itself. There is for example a cleanup action available in the marketplace that runs in a container that uses root, and that cleans up the workspace directory. This is not a good solution, as it requires a lot of manual work, and you will have to keep track of the jobs that cause this issue. The following options are probably manageable in a better way.
Add cleanup configuration on the container level configuration in the runner
You can configure the runner with specific customization commands that get executed before and after the job, as well as other options:
- prepare_job: Called when a job is started.
- cleanup_job: Called at the end of a job.
- run_container_step: Called once for each container action in the job.
- run_script_step: Runs any step that is not a container action. See the entire configuration and examples here. This is rather complex and more suited for configuring sidecars or other complex scenarios. I prefer to use the next option.
Add cleanup configuration on the runner configuration itself
By setting 2 environment variables before the job starts, you can run specific scripts before and after the job starts. The completed job can then be a docker run command that mounts the $GITHUB_WORKSPACE and cleans everything up while running as root
. This is the easiest option in my opinion:
- ACTIONS_RUNNER_HOOK_JOB_STARTED: The script defined in this environment variable is triggered when a job has been assigned to a runner, but before the job starts running.
- ACTIONS_RUNNER_HOOK_JOB_COMPLETED: The script defined in this environment variable is triggered after the job has finished processing. Find more information in the documentation here.
Do not run the container with any persistence
The best advice is to always start runners as ephemeral
, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse. You will need to provide a mechanism to cleanup the Virtual Machine or Container setup where you execute the runner on. The best method I’ve found is to use Actions Runner Controller where the runner is executing inside of a container that gets deleted after the job is done. This will prevent any data to linger around on the runner as well, and gets you a clean execution environment every time.
There might be some valid reasons to still have a persistent runner, but I would recommend to use ephemeral runners as much as possible.