Exposing ssh port in dockerized gitlab-ce

2017-03-19 18:40

One problem of running a dockerized gitlab instance is that the gitlab instance would need an ssh server. If we follow the best practice of docker, we should not run into this problem, but since the gitlab-ce service would heavily rely on the ssh service, it is common practice to bundle the ssh server in the container. Thus, a problem natually arise as to what is the best way to expose this internal ssh port without affecting the existing ssh port on the host server.

There are some discussion online, we have several ways to fix this problem, either by:

But personally, I like neither of them. I would like to have a setup that:

I happened to come across a blog post here, in which someone created a git user on the host system and used a bash script to handle the connection to the container. But that article worked for gogs. For gitlab-ce, we need some modifications.

Background

Before we start, I have a gitlab instance running on my system. The docker-compose.yml looks like this:


gitlab:
  image: gitlab/gitlab-ce:8.17.3-ce.0
  container_name: gitlab
  volumes:
    - /data/var/lib/gitlab/etc:/etc/gitlab
    - /data/var/lib/gitlab/data:/var/opt/gitlab
    - /data/var/lib/gitlab/log:/var/log/gitlab
    - /etc/localtime:/etc/localtime:ro
  ulimits:
    sigpending: 62793
    nproc: 131072
    nofile: 60000
    core: 0
  ports:
    - "9922:22"
    - "0.0.0.0:80:80"
    - "0.0.0.0:443:443"

And here’s what we want to achieve:

  1. The user connect to the ssh port of the host server, using the git user.
  2. SSH correctly authenticates the user, let him/her in, and run some script.
  3. In the script, we initialize another ssh connection to the container using port 9922, forward the authentication information.
  4. The sshd in the container correctly handles the request.

Git user setup on the host server

First, as it has already been stated in the gogs blog post, we need a git user on the host server. Before jumping in and create a user, please start the gitlab container and find the user id and group id of the git user in the container. Currently, both the UID and GID are hardcoded to 998. But this could change in a future release. For this purpose, we could run:


docker exec -it gitlab cat /etc/passwd | awk -F':' '{if($1=="git") printf("uid: %s; gid: %s\n"), $3, $4}'

This will give out the uid and the gid of the git user. You can check whether these IDs has already been used on your host, if not, it is best to re-use these IDs:


groupadd -g 998 git
useradd -m -u 998 -g git -s /bin/sh -d /home/git git

In this way, you would not experience any file permission issues at all.

Secondly, this git user need to share the .ssh directory with the container. Since I’ve already got that directory in my data volumes, I just need to link it here:


su - git
ln -s /data/var/lib/gitlab/data/.ssh /home/git/.ssh

Secondly, this git user would need to ssh to the container. Let’s create a key pair using ssh-kegen.


su - git
ssh-keygen
cd .ssh && mv id_rsa.pub authorized_keys_proxy

This authorized_keys_proxy file will later be mounted into the container.

Other setups on the host server

If you look into the authorized_keys file. You’ll notice that we have a command here:


[git@gitlab .ssh]$ head -n 2 authorized_keys
# Managed by gitlab-shell
command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-3",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3N

This command obviously does not exist on the host system, and we need to create a script that will forward all ssh connection to the container. To be more specific, we need to create the directory(/opt/gitlab/embedded/service/gitlab-shell/bin/) and create a file named gitlab-shell in this directory with the following content:


#!/bin/sh

ssh -i /home/git/.ssh/id_rsa -p 9922 -o StrictHostKeyChecking=no git@127.0.0.1 "SSH_ORIGINAL_COMMAND=\"$SSH_ORIGINAL_COMMAND\" $0 $@"

I’ve used my exposed port here and you might need to change this to fit your environment. After that, don’t forget to chmod +x this file.

But actually, how was the authentication information passed to the sshd in the container? If you add more debug information to the above script, you’ll find that:


$SSH_ORIGINAL_COMMAND=git-upload-pack 'somegroup/somerepo.git'
$0=/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell
$@=key-3

So, the real authentication(key-3) was determined when the user connects to the sshd on the host server, and when we connect to the container, we are not using the ssh key on the user’s computer, so ssh key forward on the client’s side is not required and not used here.

Mounting extra volume

We cannot put the content of the authorized_keys_proxy into the ~/.ssh/authorized_keys file, since that file is managed by gitlab-shell, and a key rebuild would destroy everything in the authorized_keys file. So we need to add another authorized_keys file location in the sshd_config file in the container. Luckily, around half a year ago, someone added this to the sshd_config file used by the sshd in the container, and it was later renamed to /gitlab-data/ssh/authorized_keys. Since this /gitlab-data/ directory was not used in our container, we can just mount our proxy file to that location:


gitlab:
  image: gitlab/gitlab-ce:8.17.3-ce.0
  container_name: gitlab
  volumes:
    - /data/var/lib/gitlab/etc:/etc/gitlab
    - /data/var/lib/gitlab/data:/var/opt/gitlab
    - /data/var/lib/gitlab/log:/var/log/gitlab
    - /etc/localtime:/etc/localtime:ro
    - /home/git/.ssh/authorized_keys_proxy:/gitlab-data/ssh/authorized_keys
  ulimits:
    sigpending: 62793
    nproc: 131072
    nofile: 60000
    core: 0
  ports:
    - "9922:22"
    - "0.0.0.0:80:80"
    - "0.0.0.0:443:443"

you can restart your docker container to test this whole setup now.

Extra setup if your uid/gid is used on the host server.

I’m not going to give any specific code here, but it is still possible to work out. You need to: