Introduction to artifact repository management

• Artifact Repository
  • Artifacts are apps built into a single file
  • Different formats
    • JAR, WAR, ZIP, TAR, etc
  • An artifact repository is where you store artifacts
  • Artifact repository needs to support this specific format
  • There is a repository for each file/artifact type
• Artifact Repository Manager
  • Different apps produce different types of artifacts
  • You need different repositories for them
  • Would be tedious if we use different software to manage each repository of a specific type
  • e.g. Nexus is the most popular
    • For private or internal use within a company
  • Allows you to upload and store different built artifacts
  • Can retrieve (download) artifacts later
  • Is a central storage of artifacts
• There are public repository managers
• e.g. Libs/Frameworks you use a dependencies
• e.g. Maven Central Repository, npm
• With Nexus
  • Host your own repositories
  • Proxy repository
  • Open source and commercial
  • Multiple repos for different formats
• Features of repository Managers
  • Integrate with LDAP
    • configure access mgmt
  • Flexible and powerful REST API for integration with other tools
    • Not manually used
    • Used with build automation / CI CD
      • After Jenkins builds, it pushes to Nexus
• Backup and restore
• Multiple format support
• Metadata tagging (labelling and tagging artifacts)
• Cleanup policies
• Search functionality
• User token support for system user authentication

Install and run Nexus on Cloud Server

• Nexus is created on its own server
• Requires more memory so server should have more capacity
  • Used 8GB/4 CPUs on DigitalOcean
• Add firewall rule and SSH into server
  • Install java 8 for Nexus

apt update
apt install openjdk-8-jre-headless
apt install net-tools

Install Nexus

• Installed in the /opt directory

cd /opt
wget https://download.sonatype.com/nexus/3/latest-unix.tar.gz # Download tar file
tar -zxvf latest-unix.tar.gz # Untar it

• 2 folders are created
  • Nexus folder: contains runtime and application of Nexus
    • Can start service from Nexus folder
    • Services shouldn’t have root user privileges
    • Create your own user for service
    • Give only the required permission for that specific service
  • Sonatype-work: contains own config for Nexus and data
    • Custom config and data can be kept after upgrading
    • Subdirectories depending on your Nexus configuration
    • IP addresses that accessed Nexus
    • Logs of Nexus App
    • Your uploaded files and metadata
    • Can use this folder for backups
• Change both directories' owner to nexus user

adduser nexus
chown -R nexus:nexus nexus-3.28.1-01
chown -R nexus:nexus sonatype-work

vim nexus-3.28.1-01/bin/nexus.rc
run_as_user="nexus"

su - nexus
/opt/nexus-3.28.1-01/bin/nexus start

ps aux | grep nexus
netstat -lnpt

• Running on port 8081
• You need to open that port on DigitalOcean
• Open inbound connection on 8081

Introduction To Nexus

• Admin is created
  • Details in /opt/sonatype-work/nexus3/admin.password
  • Username is admin
  • Disable anonymous access

Repository Types

• Can have multiple repositories of different formats
• By default, some repos have already been created (most used)
• Each repository has a type
  • Hosted
  • Group
  • Proxy
• Proxy Repo
  • A repo that is linked to a remote repository
  • If a component is requested, it will go through the proxy instead of remote
  • If the component is already available on nexus, it uses it
  • If not, it goes to the remote repository and the component is retrieved and stored in nexus' cache
  • Advantage of using proxy over fetching from the remote directly is that it gives devs a simple repository endpoint
  • Comes with maven-central and nuget.org proxy repositories
  • Best practice
    • Set up proxy repositories for the default remote repositories for that specific repo type
    • Your apps will get packages or dependencies from your proxy repository
    • Can configure repositories for Docker and npm
• Hosted Repo
  • Is the primary storage for artifacts and components
  • Company artifacts and components are stored in the hosted repos
  • Best practice
  • eg. for Java
  • Can have maven-releases hosted repository and maven-snapshots hosted repository
  • Development versions can be stored as snapshots in the maven-snapshots (internal development versions)
  • When the app is ready, can be stored in maven-releases
  • maven-releases
    • Where org publishes internal releases
    • Can use for 3rd party components not available in public repositories
• Group Repository
  • Allows you to combine multiple repositories and other repository groups in a single repository

Publish Artifact to repository

• Upload Jar file to Nexus
  • Upload Far file to existing hosted repository on Nexus
  • Maven/Gradle command for pushing to remote repository
  • Configure both tools to connect to Nexus (Nexus Repo URL + Credentials)
  • Create Nexus User with permission to upload
• Create Nexus User
  • Create role
    • Give the least necessary permission
    • eg. nx-java
      • nx-repository-view-maven2-snapshots-*
    • Have 2 prefixes
      • admin
        • Used for administrators of nexus
      • view

Gradle Project

plugins {
    id 'java'
    id 'org.springframework.boot' version '2.2.2.RELEASE'
    id 'io.spring.dependency-management' version '1.0.8.RELEASE'
}

group 'com.example'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

// Changes
apply plugin: 'maven-publish' // for publishing to maven formatted repositories. For both maven and gradle projects

publishing {
    publications {
        maven(MavenPublication) {
            artifact("build/libs/my-app-$version"+".jar") {
                extension 'jar'
            }
        }
    }

    repositories {
        maven {
            name 'nexus'
            url "http://46.101.137.161:8081/repository/maven-snapshots/"
            credentials {
                username project.repoUser
                password project.repoPassword
            }
            allowInsecureProtocol = true
        }
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation group: 'net.logstash.logback', name: 'logstash-logback-encoder', version: '5.2'
    testImplementation group: 'junit', name: 'junit', version: '4.12'
}

rootProject.name = 'my-app'

repoUser = alfredasare
repoPassword = abcd

Gradle project JAR upload

./gradlew build

./gradlew publish

• Publish command was made available after adding plugin

Maven Project

• Configure with Nexus
• Use pom.xml file

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>java-maven-app</artifactId>
    <version>1.1.0-SNAPSHOT</version>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-deploy-plugin</artifactId>
                    <version>2.8.2</version>
                </plugin>
            </plugins>
        </pluginManagement>

        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-deploy-plugin</artifactId>
            </plugin>

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.5.RELEASE</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- to handle any Java version mismatch, add the following configuration for maven-compiler-plugin -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <distributionManagement>
        <snapshotRepository>
            <id>nexus-snapshots</id>
            <url>http://46.101.137.161:8081/repository/maven-snapshots/</url>
        </snapshotRepository>
    </distributionManagement>

</project>

• To set credentials
  • Create settings.xml in .m2 directory in your home directory

<settings>
  <servers>
   <server>
    <id>nexus-snapshots</id>
    <username>alfredasare</username>
    <password>abcd</password>
   </server>
  </servers>
</settings>

mvn package

mvn deploy

Nexus REST API

• Used to query nexus repository for different information
  • Components
  • Repositories
  • Versions
• This information is needed in the CI/CD pipeline
• When you are pushing multiple artifacts per day
• How to access API
  • cURL or wget to execute http request
  • Provide user and credentials of Nexus user
  • Use Nexus user with the required permissions

# List repos
curl -u alfredasare:password -X GET 'http://46.101.137.161:8081/service/rest/v1/repositories'

# List all components
curl -u admin:password -X GET 'http://46.101.137.161:8081/service/rest/v1/components?repository=maven-snapshots'

# Get info for component
curl -u admin:password -X GET 'http://46.101.137.161:8081/service/rest/v1/components/bWF2ZW4tc25hcHNob3RzOjg4NDkxY2QxZDE4NWRkMTM2MzI2OGYyMjBlNDVkN2Rl'

• Assets are files of the component

Blob store

• Is the storage for Nexus that is used to store all the uploaded files
• A blob store is an internal storage mechanism for binary parts of artifacts
• Storage can be local like on the droplet or on the Cloud (S3)
• Each blob store can be used by one or multiple repositories or repository groups
• Can create blob stores per repo or make repos share blob stores
• Data stored in /opt/sonatype-work/nexus3/blobs/default/content
• Type
  • A blob store's storage backend
  • File: File system-based storage
  • S3: Cloud-based storage. Allows blobs to be stored in AWS
• State field
  • Indicates the state of the blob
  • started: running as expected
  • failed: has config issue
• Blob count
  • Number of blobs
• Blob stores can't be modified
• Blob store used by a repository can't be deleted
• Need to decide
  • How many blob stores you create
  • With which sizes
  • Which ones you'll use for which repos
  • How much space each repo will need
• One repo cannot use multiple blob stores

Component vs Asset

• Component
  • Abstract
  • What we are uploading
• Assets
  • Actual physical packages/files
• 1 component = 1 or more assets
• Docker Format gives assets unique identifiers and calls them Docker layers
• Docker Layers == Assets
• Assets can be reused for different components
• E.g. can have 2 Docker Images => 2 Components, but share same assets
• The term component refers to any type or format that you upload

Cleanup Policies and Scheduled tasks

• Rules that will clean up artifacts from the repos if they are old or haven’t been used for a while
• After creating policy, attach policy to repository
• Interesting info on cleanup policy
  • Cleaned up components will actually not get deleted
  • They'll be marked for deletion. This is called soft delete
  • If we want to actually delete those items, we would need to compact the blob store
• Can manually execute tasks