Speeding up your CI pipeline by 50% or more
Are you running docker build in your pipeline? Remember
to enable image caching. Here’s an example for AWS
CodePipeline and ECR, but these steps are roughly the same
for all docker providers. The steps are essentially:
- Write your Dockerfile to be cacheable
- Use your Docker repository as the CI cache
- You’re ready; fetch glory from your collegues
1) Write your Dockerfile to be cacheable ¶
If you have a Dockerfile like this:
COPY * ./
RUN npm install && npm build && npm test
you should rewrite it like this:
# essential: npm install (or similar "get and build dependencies")
COPY package.json package-lock.json ./
RUN npm install
# ...before copying the rest of your source files
COPY src ./src
RUN npm build
RUN npm test
You can test whether you got it correct by first building the Docker image on your own computer, modifying some source file, and rebuilding the image. If you see the text ‘—> Using cache’ after the ’npm install’ step, you’re good.
2) Use your Docker repository as the CI cache ¶
This example uses AWS and ECR. Adjust the examples for your CI as appropriate.
If you have a CI pipeline script like this:
pre_build:
commands:
$(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
build:
commands:
docker build -f Dockerfile \
--target $ECR_TARGET_REPOSITORY_URI:latest .
post_build:
commands:
docker push $ECR_TARGET_REPOSITORY_URI:latest
update this to pull the image, and use that as cache:
pre_build:
commands:
$(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
docker pull $ECR_TARGET_REPOSITORY_URI:latest # <- Changed line number one
build:
commands:
docker build -f Dockerfile \
--cache-from $ECR_TARGET_REPOSITORY_URI:latest \ # <- Changed line number two
--target $ECR_TARGET_REPOSITORY_URI:latest . # <- this tag must match the pull/push
post_build:
commands:
docker push $ECR_TARGET_REPOSITORY_URI:latest # <- Remember to push the correct target
3) You’re ready; fetch glory from your collegues ¶
In my current project, doing this sped up our pipeline from 20 minutes to 10 minutes. Depending on how long you take to fetch dependencies, it might or might not be a huge speedup for you.
PS. if you are using multi-stage builds, build each image separately:
docker pull $REPO::builder || true
docker pull $REPO::app || true
...
docker build -f Dockerfile \
--target build \
--cache-from $REPO:builder \
-t $REPO:builder
docker build -f Dockerfile \
--target app \
--cache-from $REPO:builder \
--cache-from $REPO:app \
-t $REPO:app \
-t $REPO:latest
docker push $REPO:builder
docker push $REPO:app
docker push $REPO:latest