Pipeline: Custom Package Managers

While working at my current employer I had the chance to experiment with building a tool for helping deployment and management of pipeline utilities and plug-ins. One of the first things I tried was building a package manager similar to pip or a linux distro packlage manager like yum or apt-get. 

A multi-project facility is very well suited for such a tool. Imagine being about to install plugins and utilities from deployment server using a CLI where a possible command is

pman install nuke-sapphire

This snippet may install the nuke sapphire plugins for a project and maybe even be able to specify a version or a nuke version. 

The actually implementation of this can be as simple or as complex as you need for your facility. I envision a package manager that not only install and updates code for a project but one that also can test code within the install environment to keep bad utlities from being installed. For language libraries in python, javascript, or even java, automating testing with test suites is fairly straight forward though this can really introduce complexity if you plan on having Department TDs in the mix. 

For R&D departments and business and logic layers code and libraries can be maintained in a stricter fashion and made to adhere to a packaging and structure syntax (i.e. Python Setup tools styles). This strict structure breaks down for scripts closer to presentation layer (department td work) where project specific changes are made. In this case building a seperate system to handle this seems completely reasonable, however! IMO, its a waste of resources as you can get a happy middle of the road. 

In my design of implementation: 

  • Tracking/Managment Layer -- Essentially a ORM that takes care of managing packages and their locations in some SQL or PostgreSQL database. 
  • Installation/Updating/Versioning Layer -- purely handles the installation of packages and upgrades.
  • Manager CLI -- Command Line Interface Implementation

Using 'pip', 'django' and 'yum' as my example of how to manage packages (and for CLI design) we can get a very nice, common, package installation for install everything from scripts to compiled plugins and even have those in installation groups (like linux package managers)

Finally, to maintain agility in development for dept. TDs, it is improtant that the creation of a package (for managment in the system) is simplfied to allow for fast iterations. To do this I think the best path is to adopt a git style but insert folder denoted with a '.' (i.e. '.pman'). This would allow for a install.py file in the .pman folder to be used by the installation api to install the package and perform a number of other tasks (like testing and building). This would allow for very complex installs but a fallback to a simple pull or copy (depending on the repo type)