While developing one of our recent projects, as a team, we decided to implement our microservices application in the mono repository. This idea allows us to sync versions between services easily, even in one commit. However, we’re not keeping this solution for ourselves. Seeing great value in it, we open-sourced one of the microservices for the community.
We decided to change our private GitHub repository to be open to the public when we were ready, but before we had to find answers to questions like:
- How to authorize other services to pull dependencies from private repositories, both in local and Docker environments?
- How to make it work with our CI system - CircleCI?
Keep reading to find out how we nailed it. In the article, you will find a step-by-step guide to implementing our solution.
Problem statement
NOTE: At the beginning of each paragraph there are links to GitHub repositories containing fully working example solutions at given stage.
In our project, we are creating a system which consists of multiple microservices, most of which are written in Go. For easier and faster development we decided to start with monorepo (hosted by GitHub), with the following structure.
.
├── authorizer
│ ├── cmd
│ │ └── authorizer
│ │ └── main.go
│ ├── Dockerfile
│ ├── go.mod
│ ├── go.sum
│ └── pkg
│ └── app
│ └── app.go
├── backend
│ ├── cmd
│ │ └── backend
│ │ └── main.go
│ ├── Dockerfile
│ ├── go.mod
│ ├── go.sum
│ └── pkg
│ └── app
│ └── app.go
├── log
│ ├── go.mod
│ ├── go.sum
│ └── log.go
├── Makefile
├── README.md
└── server
├── go.mod
├── go.sum
└── server.goThere are two microservices - ‘authorizer’ and ’backend’. There are also two libraries - ‘log’ and ‘server’. Authorizer depends only on the ‘log’ module, while the backend depends on ‘log’ and ‘server’. Dependencies use the ‘replace’ directive in Go mod to work properly.
replace (
example.org/log => ../log
example.org/server => ../server
)At some point during development, we decided that the ’server’ and ‘log’ packages would be open source. As a first step, we wanted to extract them as a separate (but still private repo) and then after some testing go public.
Requirements
In our development process, we build our services in 3 ways:
- Build on local dev machine
- Containerized build on local dev machine
- Containerized build in CircleCI
At each of those stages we need to address the following issues:
- How to access dependency code from a private GitHub repository?
- How to handle custom (‘example.org’) domain names?
These sound easy, but during the repo split we discovered that this matter is far from trivial. In the following paragraphs we will walk through the implementation process describing how to solve issues emerging at each of the levels.
Working on a local environment
monorepo@after-split-local-dev
There are no surprises at this level. To get this to work there are only three steps needed:
- Configure the GitHub SSH key which has access to all the required repositories.
- Configure Git to use SSH instead of HTTPS for GitHub:
git config --global url.ssh://git@github.com/.insteadOf https://github.com/ - Set
GOPRIVATE=github.com/codilime/to tellgo getthat those repositories need to be accessed directly without Go module mirrors.
Working on a local environment with Docker
monorepo@after-split-local-dev-with-docker
The main challenge when using private repositories in Docker is to keep the SSH keys used to access repositories safe. We can’t just copy them to the container as someone unauthorized might access them.
Docker allows us to copy files to the container which live only in the context of one ‘RUN’. That allows us to inject the SSH key only for the dependencies download. This key won’t be present in the final image.
FROM golang:1.18-buster AS builder
WORKDIR /src
COPY authorizer/ .
RUN git config --global url.ssh://git@github.com/.insteadOf https://github.com/
RUN mkdir /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts
ENV GOPRIVATE github.com/codilime
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /authorizer cmd/authorizer/main.go
FROM alpine:3.15.0
COPY /authorizer /authorizer
CMD ["/authorizer"]Makefile:
TAG=$(shell git rev-parse --short HEAD)
build_backend:
docker build \
-t backend:$(TAG) \
-f backend/Dockerfile \
--secret id=sshKey,src=${HOME}/.ssh/id_ecdsa \
.
build_authorizer:
docker build \
-t authorizer:$(TAG) \
-f authorizer/Dockerfile \
--secret id=sshKey,src=${HOME}/.ssh/id_ecdsa \
.Working in CircleCI with Docker
blog/monorepo@after-split-circleci
Unlike in the local dev environment, in CircleCI we would like to have more granular control when accessing GitHub repositories. To achieve this we can use deploy keys, which are limited to a single repository and may be even configured to be read only, which is the expected permission for CI jobs.
Unfortunately, GitHub does not allow adding the same deploy key for multiple repositories. We need to improve the solution from the previous step to make it work.
First, let’s generate and add SSH deploy keys to each of the repositories.
$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/krzysztof/.ssh/id_ecdsa): log_github_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in log_github_key
Your public key has been saved in log_github_key.pub
<snip>
~/.../github.com/goprivate-blog$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/krzysztof/.ssh/id_ecdsa): server_github_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in server_github_key
Your public key has been saved in server_github_key.pub
<snip>Adding to GitHub
Fig. 1 Adding SSH keys to GitHub
Adding to CircleCI
To distinguish keys we add a prefix to github.com. Don’t worry that they are not real hostnames, we will override them.
Fig. 2 Adding SSH keys to CircleCI
Fig. 3 Additional SSH keys
Next, we need to create fake SSH endpoints, to tell the SSH client which key needs to be used to access each of the repositories. To make it reusable, both in Dockerfiles and in CircleCI config, we can wrap it in a simple bash script.
#!/usr/bin/env bash
set -euo pipefail
set -x
log_key_path=${1:-/root/.ssh/log.pub}
server_key_path=${2:-/root/.ssh/server.pub}
mkdir -p ~/.ssh
# Add github to known hosts
ssh-keyscan github.com >>~/.ssh/known_hosts
# Add "fake" SSH hosts per each private repo
cat <<EOT >>~/.ssh/config
Host log.github.com
HostName github.com
IdentityFile ${log_key_path}
IdentitiesOnly yes
Host server.github.com
HostName github.com
IdentityFile ${server_key_path}
IdentitiesOnly yes
EOT
# Configure git to use "fake" SSH hosts
git config \
--global url."git@log.github.com:codilime/goprivate-blog-log".insteadOf \
"https://github.com/codilime/goprivate-blog-log"
git config \
--global url."git@server.github.com:codilime/goprivate-blog-server".insteadOf \
"https://github.com/codilime/goprivate-blog-server"
export GOPRIVATE=github.com/codilimeNow, change Dockerfiles to:
FROM golang:1.18-buster AS builder
WORKDIR /src
COPY authorizer/ .
RUN <<-EOF
bash /root/go_mod_ssh_config.sh
go mod download
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /authorizer cmd/authorizer/main.go
EOF
FROM alpine:3.15.0
COPY /authorizer /authorizer
CMD ["/authorizer"]And update CircleCI config to:
version: 2
jobs:
build:
docker:
- image: cimg/base:2022.08
steps:
- checkout
- setup_remote_docker:
version: 20.10.14
- add_ssh_keys:
fingerprints:
- "c5:cd:b9:00:1e:4c:99:c0:60:45:57:1d:d2:41:8c:d2" # log.github.com
- "6f:2d:3c:ae:08:6f:d4:f0:15:5f:b7:4a:89:60:d2:13" # server.github.com
- run:
name: Configure private repos for go mod
command: |
./.circleci/scripts/go_mod_ssh_config.sh \
~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2 \
~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213
- run:
name: Build backend Docker image
command: |
make build_backend
- run:
name: Build authorizer Docker image
command: |
make build_authorizerWorking with passphrase-protected SSH keys
monorepo@after-split-passphrase-protected-keys
When the SSH key is protected with a passphrase this solution won’t work. In such a case, we would need to use SSH Agent to manage our keys. To do so we need to pass the existing SSH Agent socket into the image and point the SSH config to public keys instead of private ones.
The only issue here is that CircleCI only stores private keys, we need to generate private ones in CircleCI:
- run:
name: Generate public keys
command: |
ssh-keygen -y -f ~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2 \
> ~/.ssh/id_rsa_c5cdb9001e4c99c06045571dd2418cd2.pub
ssh-keygen -y -f ~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213 \
> ~/.ssh/id_rsa_6f2d3cae086fd4f0155fb74a8960d213.pubSummary
Our goal was to make this solution secure and reusable across multiple services. As all this information could not be found in a single place on the internet, this post gathers all the information in one place and explains the entire process we went through to make it work.
Hopefully, you will benefit from our guide next time you have to deal with private repositories, Go modules, Docker and CircleCI at once.
