Rust vs. C in networking application
Close

15 September 2022

Software Development

Dealing with private repositories, Go modules, Docker and CircleCI

7 minutes reading

Dealing with private repositories, Go modules, Docker and CircleCI

While developing one of our recent projects, as a team, we decided to implement our microservices application in the mono repository. This idea allows us to sync versions between services easily, even in one commit. However, we’re not keeping this solution for ourselves. Seeing great value in it, we open-sourced one of the microservices for the community. 

We decided to change our private GitHub repository to be open to the public when we were ready, but before we had to find answers to questions like: 

  • How to authorize other services to pull dependencies from private repositories, both in local and Docker environments?
  • How to make it work with our CI system - CircleCI?

Keep reading to find out how we nailed it. In the article, you will find a step-by-step guide to implementing our solution. 

Problem statement

NOTE: At the beginning of each paragraph there are links to GitHub repositories containing fully working example solutions at given stage.

https://github.com/goprivate-blog/monorepo/tree/before-split

In our project, we are creating a system which consists of multiple microservices, most of which are written in Go. For easier and faster development we decided to start with monorepo (hosted by GitHub), with the following structure.

.
├── authorizer
│   ├── cmd
│   │   └── authorizer
│   │       └── main.go
│   ├── Dockerfile
│   ├── go.mod
│   ├── go.sum
│   └── pkg
│       └── app
│           └── app.go
├── backend
│   ├── cmd
│   │   └── backend
│   │       └── main.go
│   ├── Dockerfile
│   ├── go.mod
│   ├── go.sum
│   └── pkg
│       └── app
│           └── app.go
├── log
│   ├── go.mod
│   ├── go.sum
│   └── log.go
├── Makefile
├── README.md
└── server
    ├── go.mod
    ├── go.sum
    └── server.go

There are two microservices - ‘authorizer’ and ’backend’. There are also two libraries - ‘log’ and ‘server’. Authorizer depends only on the ‘log’ module, while the backend depends on ‘log’ and ‘server’. Dependencies use the ‘replace’ directive in Go mod to work properly.

replace (
        example.org/log => ../log
        example.org/server => ../server
)

At some point during development, we decided that the ’server’ and ‘log’ packages would be open source. As a first step, we wanted to extract them as a separate (but still private repo) and then after some testing go public.

Requirements

In our development process, we build our services in 3 ways:

  1. Build on local dev machine
  2. Containerized build on local dev machine
  3. Containerized build in CircleCI.

At each of those stages we need to address the following issues:

  1. How to access dependency code from a private GitHub repository?
  2. How to handle custom (‘example.org’) domain names?

These sound easy, but during the repo split we discovered that this matter is far from trivial. In the following paragraphs we will walk through the implementation process describing how to solve issues emerging at each of the levels.

Working on a local environment

https://github.com/goprivate-blog/monorepo/tree/after-split-local-dev

https://github.com/goprivate-blog/log

https://github.com/goprivate-blog/server

There are no surprises at this level. To get this to work there are only three steps needed:

  1. Configure the GitHub SSH key which has access to all the required repositories.
  2. Configure Git to use SSH instead of HTTPS for GitHub:
    git config --global url.ssh://git@github.com/.insteadOf https://github.com/
  3. Set GOPRIVATE=github.com/goprivate-blog/ to tell go get that those repositories need to be accessed directly without Go module mirrors.

Working on a local environment with Docker

https://github.com/goprivate-blog/monorepo/tree/after-split-local-dev-with-docker

https://github.com/goprivate-blog/log

https://github.com/goprivate-blog/server

The main challenge when using private repositories in Docker is to keep the SSH keys used to access repositories safe. We can’t just copy them to the container as someone unauthorized might access them.

Docker allows us to copy files to the container which live only in the context of one ‘RUN’. That allows us to inject the SSH key only for the dependencies download. This key won’t be present in the final image.

FROM golang:1.18-buster AS builder
WORKDIR /src
COPY authorizer/ .

RUN git config --global url.ssh://git@github.com/.insteadOf https://github.com/
RUN mkdir /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts
ENV GOPRIVATE github.com/goprivate-blog

RUN --mount=type=secret,id=sshKey,dst=/root/.ssh/id_ecdsa go mod download

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /authorizer cmd/authorizer/main.go

FROM alpine:3.15.0
COPY --from=builder /authorizer /authorizer
CMD ["/authorizer"]

Makefile:

TAG=$(shell git rev-parse --short HEAD)

build_backend:
        docker build \
                -t backend:$(TAG) \
                -f backend/Dockerfile \
                --secret id=sshKey,src=${HOME}/.ssh/id_ecdsa \
                .

build_authorizer:
        docker build \
                -t authorizer:$(TAG) \
                -f authorizer/Dockerfile \
                --secret id=sshKey,src=${HOME}/.ssh/id_ecdsa \
                .

Working in CircleCI with Docker

https://github.com/goprivate-blog/monorepo/tree/after-split-circleci

https://github.com/goprivate-blog/log

https://github.com/goprivate-blog/server

Unlike in the local dev environment, in CircleCI we would like to have more granular control when accessing GitHub repositories. To achieve this we can use deploy keys, which are limited to a single repository and may be even configured to be read only, which is the expected permission for CI jobs.

Unfortunately, GitHub does not allow adding the same deploy key for multiple repositories. We need to improve the solution from the previous step to make it work.

First, let’s generate and add SSH deploy keys to each of the repositories.

$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/krzysztof/.ssh/id_ecdsa): log_github_key
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in log_github_key
Your public key has been saved in log_github_key.pub
<snip>
~/.../github.com/goprivate-blog$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/krzysztof/.ssh/id_ecdsa): server_github_key
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in server_github_key
Your public key has been saved in server_github_key.pub
<snip>

Adding to GitHub

Adding SSH keys to GitHub

Fig. 1 Adding SSH keys to GitHub

Adding to CircleCI 

To distinguish keys we add a prefix to github.com. Don’t worry that they are not real hostnames, we will override them.

Adding SSH keys to CircleCI

Fig. 2 Adding SSH keys to CircleCI

Additional SSH keys

Fig. 3 Additional SSH keys 

Next, we need to create fake SSH endpoints, to tell the SSH client which key needs to be used to access each of the repositories. To make it reusable, both in Dockerfiles and in CircleCI config, we can wrap it in a simple bash script.

#!/usr/bin/env bash
set -euo pipefail
set -x

log_key_path=${1:-~/.ssh/log}
server_key_path=${2:-~/.ssh/server}

mkdir -p ~/.ssh
# Add github to known hosts
ssh-keyscan github.com >>~/.ssh/known_hosts

# Add "fake" SSH hosts per each private repo
cat <<EOT >>~/.ssh/config
  Host log.github.com
    HostName github.com
    IdentityFile ${log_key_path}
    IdentitiesOnly yes
  Host server.github.com
    HostName github.com
    IdentityFile ${server_key_path}
    IdentitiesOnly yes
EOT

# Configure git to use "fake" SSH hosts
git config \
        --global url."git@log.github.com:goprivate-blog/log".insteadOf \
        "https://github.com/goprivate-blog/log"
git config \
        --global url."git@server.github.com:goprivate-blog/server".insteadOf \
        "https://github.com/goprivate-blog/server"

export GOPRIVATE=github.com/goprivate-blog

Now, change Dockerfiles to:

FROM golang:1.18-buster AS builder

WORKDIR /src
COPY authorizer/ .

RUN --mount=type=secret,id=ssh-script,dst=/root/go_mod_ssh_config.sh \
    --mount=type=secret,id=log-key,dst=/root/.ssh/log \
    --mount=type=secret,id=server-key,dst=/root/.ssh/server \
    --mount=type=ssh <<-EOF
    bash /root/go_mod_ssh_config.sh
    go mod download
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /authorizer cmd/authorizer/main.go
EOF

FROM alpine:3.15.0
COPY --from=builder /authorizer /authorizer
CMD ["/authorizer"]

And update CircleCI config to:

version: 2
jobs:
  build:
    docker:
      - image: cimg/base:2022.08
    steps:
      - checkout
      - setup_remote_docker:
          version: 20.10.14
      - add_ssh_keys:
          fingerprints:
            - "c5:cd:b9:00:1e:4c:99:c0:60:45:57:1d:d2:41:8c:d2" # log.github.com
            - "6f:2d:3c:ae:08:6f:d4:f0:15:5f:b7:4a:89:60:d2:13" # server.github.com
      - run:
          name: Configure private repos for go mod
          command: |
            ./.circleci/scripts/go_mod_ssh_config.sh \
              ~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2 \
              ~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213
      - run:
          name: Build backend Docker image
          command: |
            make build_backend
      - run:
          name: Build authorizer Docker image
          command: |
            make build_authorizer

Working with passphrase-protected SSH keys

https://github.com/goprivate-blog/monorepo/tree/after-split-passphrase-protected-keys

https://github.com/goprivate-blog/log

https://github.com/goprivate-blog/server

When the SSH key is protected with a passphrase this solution won’t work. In such a case, we would need to use SSH Agent to manage our keys. To do so we need to pass the existing SSH Agent socket into the image and point the SSH config to public keys instead of private ones.

The only issue here is that CircleCI only stores private keys, we need to generate private ones in CircleCI:

      - run:
          name: Generate public keys
          command: |
            ssh-keygen -y -f ~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2 \
              > ~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2.pub
            ssh-keygen -y -f ~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213 \
              > ~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213.pub

Summary

Our goal was to make this solution secure and reusable across multiple services. As all this information could not be found in a single place on the internet, this post gathers all the information in one place and explains the entire process we went through to make it work. 

Hopefully, you will benefit from our guide next time you have to deal with private repositories, Go modules, Docker and CircleCI at once.

Krzysztof

Krzysztof Kwapisiewicz

Software Engineer