The days of giant, discrete batches of code making up an application are behind us. Modern software has undergone an industrial revolution of sorts with its own version of interchangeable parts. Libraries and artifacts have quickly become an essential part of development.
This shift from a monolithic application code base, to applications built on 100s of smaller parts, has directly led to a dramatic decrease in release times, as well as the advent of philosophies like Continuous Delivery, and DevOps. One of the biggest things that is still neglected, is how to properly store, and access these pieces.
The importance of the storage and management issue is compounded, because so many of these pieces will be changed, updated, swapped out, or removed as an application matures. This is why a Universal Package Manager is the most critical link in any Modern Software Toolchain.
What is a Universal Package Manager?
A Universal Package Manager (UPM) holds the components of an application, in a secure and configurable location. These items can include: build artifacts, configuration details, packages, legacy versions of applications, etc. UPMs are also sometimes called repository managers.
UPMs manage feeds, automatically updating and caching external libraries. Internal / proprietary, feed data and artifact data is also managed. Meta-data is also maintained by the UPMs, including things like creation time of build artifacts, license agreements for libraries, etc. Meta-data in UPMs is often configurable, depending on the needs of the user for security and compliance reasons.
Why UPMs really matter
Security and compliance is often the leading requirement in regulated industry (banking, health care, etc.), and a UPM assists greatly in these environments. By enforcing LDAP, Domain Forest Directory, or User Sign-in, UPMs allow for feed restrictions based on credentials. This data is also logged by the UPM, allowing audits to be conducted.
Caching packages in a UPM provides an added level of control by utilizing features for license filtering and vulnerability scanning. Storing components in one location reduces the risk of security violations regardless of package type.
Additionally, local caching negates the risks associated with public repositories, no longer requiring reliability of third party packages that can be edited or deleted at any time, like in the left-pad breakdown. Caching also reduces internal bandwidth by pulling packages locally, and not from a public repository.
Since there are hundreds of types of licenses, license filtering should be integrated into a UPM. The ability to block or allow packages based on license type ensures software created with public packages doesn’t run afoul of licensing agreements, which could open up an organization to costly litigation.
Finally, because a UPM can store build artifacts, everyone on the team uses the same artifacts when testing, which helps to limit the refrain of ‘it worked on my machine’.
UPMs and DevOps
Used correctly, a Universal Package Manager helps further the technical and cultural goals of DevOps.
UPM’s often become the central hub for development by allowing Build and Deploy tools to push artifacts to, and draw artifacts from it. Infrastructure tools also draw elements from the UPM, providing both the Dev, and Ops teams with better visibility over the entire software development lifecycle.
Implementing a UPM allows for more automation, artifacts go to the same place each time a new one is created, making the build and deploy pipeline more static, with fewer changing variables – enabling the ability to reproduce past builds (since UPMs keep old versions) for easier roll-backs.
Features to look for
Aside from some of the features mentioned above (User restrictions, meta-data retention, license filtering, vulnerability scanning, and accessibility by other tools), there are a few other features that a UPM should have.
First, the ability to store multiple package and library types (NuGet, npm, bower, etc.), as well as extensibility via a non-standard type of feed, that accepts different types of files by wrapping them in a different format (like a zipfile).
Second, because a UPM is the critical link in your toolchain, it needs to be user friendly, easy to configure, and easy to maintain so all critical files are stored in it.
Lastly, the ability to scale depending on your organizational needs. A single instance, referred to as a node, is usually good enough for a small organization. Load-balancing and high availability versions are essential for mid and large organizations who need constant access to their UPM. These versions often employ multiple instances, with replication capabilities, as well as indexing and web-nodes.
Not using a UPM isn’t just an anti-pattern it is downright negligent. There are many excellent options for UPMs, so there’s no reason not to start using one.
