1
\$\begingroup\$

Introduction

I'm writing a script in YAML for building ASP.NET Core 2.2 project using GitLab Continuous Integration. In all YAML samples I could find (and there are not many) for building .NET Core based applications using GitLab CI I could see something like this:

before_script:
 - 'dotnet restore'

before_script is running a dependency restore before every job. It makes sense because if you use free GitLab runners (like I do) every job is executed on a different machine. There is no possibility to preserve state from previous jobs (with exception on cache and artifacts, but I'll get to that later). What that means is that on the next job, all previously restored packages will be gone, and they need to be restored again. And on the next job once again. And again on every job in the pipeline which needs the packages. I noticed a redundancy there. A redundancy that takes precious time, because a huge project with a lot of third-party packages takes a while for a full restore.

Using job artifacts

I found a way to preserve those packages and then pass them to the next job via GitLab artifacts:

restore:
 stage: restore
 script:
  - 'dotnet restore --packages .nuget/'
 artifacts:
  paths:
   - 'src/**/obj/*'
   - '.nuget/'

Let's break it down. With dotnet restore --packages .nuget/ I explicitly specify a custom directory for packages to be restored. Then I specify two paths which GitLab CI will be interested in when creating a job artifacts. dotnet restore creates a few files with metadata about packages inside a obj/ directory, so these will be needed as well. I include them in src/**/obj/*. Finally, I include the .nuget/ directory which after dotnet restore should contain all restored dependencies.

Note: A dependency restore saves the path where the packages will be kept inside <PROJECT_NAME>/obj/project.assets.json file. After that, there is no need for explicitly specifying where the restored packages are e.g. when building the project.

Eventually, in the next job I use previously created job artifacts by specifying a job dependecy. In that way, GitLab CI knows that it should download job artifacts from the dependant job.

build:
 stage: build
 script:
  - 'dotnet build --no-restore'
 dependencies:
  - restore

Whole YAML script:

image: microsoft/dotnet:2.2-sdk

variables:
 SOURCE_CODE_DIRECTORY: 'src'
 BINARIES_DIRECTORY: 'bin'
 OBJECTS_DIRECTORY: 'obj'
 NUGET_PACKAGES_DIRECTORY: '.nuget'

stages:
 - restore
 - build

restore:
 stage: restore
 script:
  - 'dotnet restore --packages="$NUGET_PACKAGES_DIRECTORY"'
 artifacts:
  paths:
   - '$SOURCE_CODE_DIRECTORY/**/$OBJECTS_DIRECTORY/*'
   - '$NUGET_PACKAGES_DIRECTORY/'

build:
 stage: build
 script:
  - 'dotnet build --no-restore'
 dependencies:
  - restore

Feedback

Please tell me what you think, any weaknesses of my approach, code smells, or maybe a better solution. All kind of constructive feedback appreciated.

\$\endgroup\$
1
\$\begingroup\$

I had a wrong concept about GitLab artifacts. After a good read on GitLab docs, especially the section distinguishing artifacts and cache, I deduced that I should use cache instead of artifacts as it was designed precisely for storing restored dependencies. Artifacts are meant for passing build output and binaries.

I also removed the restore stage, placing the dotnet restore command in a global before_script. Cache can fail and in such scenario the script should gracefully fallback to default 'download-from-internet' behaviour. With --no-restore option enabled it would not happen. Thus, I removed that option from dotnet build command. It won't make a noticeable difference with successfully download cache as a dependency restore with already downloaded packages will execute in next-to-no-time.

Finally, I added cache key, which will keep cache bundles separate for branches and stages.

Updated script:

image: microsoft/dotnet:2.2-sdk

variables:
 SOURCE_CODE_DIRECTORY: 'src'
 BINARIES_DIRECTORY: 'bin'
 OBJECTS_DIRECTORY: 'obj'
 NUGET_PACKAGES_DIRECTORY: '.nuget'

stages:
  - build

cache:
 key: '$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG'
 paths:
  - '$SOURCE_CODE_DIRECTORY/*/$OBJECTS_DIRECTORY/project.assets.json'
  - '$SOURCE_CODE_DIRECTORY/*/$OBJECTS_DIRECTORY/*.csproj.nuget.*'
  - '$NUGET_PACKAGES_DIRECTORY'

before_script:
  - 'dotnet restore --packages $NUGET_PACKAGES_DIRECTORY'

build:
 stage: build
 script:
  - 'dotnet build --no-restore'
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Thanks for you post, Is caching project.assets.json is necessary? \$\endgroup\$ – Altiano Gerung Feb 28 at 8:41
  • 1
    \$\begingroup\$ project.assets.json contains a list of all dependencies, with the dependency tree, packages version, frameworks and quite importantly where to they were restored as the restore destination location can be changed (in fact, I'm doing just the thing in the script above). This file will be created during restore or build, so changes made to your dependencies list will be overwritten by newer version if needed, then uploaded back to cache. Without the file, the CLI tool won't know about the locations of restored dependencies even if you load them from cache, so yes, it is necessary. \$\endgroup\$ – Prolog Feb 28 at 17:03
  • 1
    \$\begingroup\$ Some additional good reads on the topic: Example GitLab CI template for .NET Core project with extensive commentary and a good article explaining the reason and purpose of project.assets.json. \$\endgroup\$ – Prolog Feb 28 at 17:13
  • \$\begingroup\$ And how about, *.csproj.nuget.*, what is that for? \$\endgroup\$ – Altiano Gerung Feb 29 at 4:53
  • 1
    \$\begingroup\$ As the case with project.assets.json was quite clear, it is not so with files matched with *.csproj.nuget.* expression. It usually matches 4 files. I do not know their entire purpose, but I remember I've experienced some issues when I wasn't caching them. Still, I'm not so sure about this one, so you are free to try without them. Note though, that some parts of the script above I achieved with try and see strategy which is clearly not the best approach to write code, but hey, it works! \$\endgroup\$ – Prolog Feb 29 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.