Pipeline: Custom Package Managers
While working at my current employer I had the chance to experiment with building a tool for helping deployment and management of pipeline utilities and plug-ins. One of the first things I tried was building a package manager similar to pip or a linux distro packlage manager like yum or apt-get.
A multi-project facility is very well suited for such a tool. Imagine being about to install plugins and utilities from deployment server using a CLI where a possible command is
pman install nuke-sapphire
This snippet may install the nuke sapphire plugins for a project and maybe even be able to specify a version or a nuke version.
For R&D departments and business and logic layers code and libraries can be maintained in a stricter fashion and made to adhere to a packaging and structure syntax (i.e. Python Setup tools styles). This strict structure breaks down for scripts closer to presentation layer (department td work) where project specific changes are made. In this case building a seperate system to handle this seems completely reasonable, however! IMO, its a waste of resources as you can get a happy middle of the road.
In my design of implementation:
- Tracking/Managment Layer -- Essentially a ORM that takes care of managing packages and their locations in some SQL or PostgreSQL database.
- Installation/Updating/Versioning Layer -- purely handles the installation of packages and upgrades.
- Manager CLI -- Command Line Interface Implementation
Using 'pip', 'django' and 'yum' as my example of how to manage packages (and for CLI design) we can get a very nice, common, package installation for install everything from scripts to compiled plugins and even have those in installation groups (like linux package managers)
Finally, to maintain agility in development for dept. TDs, it is improtant that the creation of a package (for managment in the system) is simplfied to allow for fast iterations. To do this I think the best path is to adopt a git style but insert folder denoted with a '.' (i.e. '.pman'). This would allow for a install.py file in the .pman folder to be used by the installation api to install the package and perform a number of other tasks (like testing and building). This would allow for very complex installs but a fallback to a simple pull or copy (depending on the repo type)