Gallery

Blog | en| ru|

XDG Tree Management - Configurable Multi-Tree XDG Overlay under Ubuntu/bash

Contents

In many cases, it could be useful to create a Frankenbuntu from your home-grade vanilla mainstream Linux distribution. In general, every large-scale program built and installed from source requires its own XDG tree to be safely isolated and divided from all other software, especially that provided by the system. Even though this approach has disadvantages, such as limited distribution compatibility, shared scripts connection issues, etc., it also has the main advantage of extreme flexibility. Together with its containerless nature, it seems to be ideal for on-site research builds. And it is extremely easy - only one bash script is required to install and connect to the system as many multilayered XDG overlay trees as needed.

Prerequisites

On my system, I use the following XDG tree overlay structure:

/util
    | /bin
    |
    | /lib
    |   | /pkgconfig
    |
    | /share
    |
    | /<special-software-1>
    |   | /bin
    |
    |   | /lib
    |       | /pkgconfig
    |
    |   | /share
    |
    | /<special-software-2>
    |
    ...

Here /util is a master overlay directory. The idea is to concentrate all the customised software built on the given system, in the directory which contains the skeleton libraries needed to build all the other custom software.

The target here is to have the alias-linked, configurable overlay structure, where the order of mentioned aliases configures the order of overlay linkage. For example, this ability is extremely useful for linkage management between several versions of the Qt library built from source. If it is needed to change the overlay order, changing the order of elements in the corresponding bash array does the trick.

Architecture

The overlay management script util.sh is installed in the directory /etc/profile.d, the typical way for most Ubuntu/Debian-based distributions. As the first step, it defines the structure of a standard XDG tree:

BIN_DIR="bin"
LIB_DIR="lib"
INC_DIR="include"
PKG_DIR="lib/pkgconfig" # Needed to exclude non-portable directory management commands
SHR_DIR="share"

Then, the master overlay directory should be defined explicitly:

BASE_PREFIX="/util"

Installing Overlays

The overlay order is explicitly defined by an order of elements in the dedicated ROOT_PREFIX array:

declare -a ROOT_PREFIX
ROOT_PREFIX=( "llvm" "ffmpeg" "qt6" "go" "fpc" "rust/cargo" "boost-1.87.0" "qt5" )

If needed, this array can be rewritten to obtain the different overlay order. As the script is installed in /etc/profile.d, the new settings are applied after re-login.

Afterwards, the algorithm is simple. It resolves the root directories, appends XDG affixes and delimiters:

declare -a ROOT_DIRS
ROOT_DIRS+=( "${BASE_PREFIX}" )

for root_prefix in "${ROOT_PREFIX[@]}"
do
    ROOT_DIRS+=( "${BASE_PREFIX}/${root_prefix}" )
done

BIN_DIRS=""
LIB_DIRS=""
INC_DIRS=""
PKG_DIRS=""
SHR_DIRS=""

for root_dir in "${ROOT_DIRS[@]}"
do
    BIN_DIRS+="${root_dir}/${BIN_DIR}:"
    LIB_DIRS+="${root_dir}/${LIB_DIR}:"
    INC_DIRS+="${root_dir}/${INC_DIR}:"
    PKG_DIRS+="${root_dir}/${PKG_DIR}:"
    SHR_DIRS+="${root_dir}/${SHR_DIR}:"
done

The last step - exporting bash environment variables (platform/GCC-dependent, subject to change):

export PATH=${BIN_DIRS}${PATH}
export C_INCLUDE_PATH=${INC_DIRS}${C_INCLUDE_PATH}
export CPLUS_INCLUDE_PATH=${INC_DIRS}${CPLUS_INCLUDE_PATH}
export LD_LIBRARY_PATH=${LIB_DIRS}${LD_LIBRARY_PATH}
export PKG_CONFIG_PATH=${PKG_DIRS}${PKG_CONFIG_PATH}
export XDG_DATA_DIRS=${SHR_DIRS}${XDG_DATA_DIRS}

Python Overlays

Overlaying Python libraries can be a headache, especially due to the extremely high presence of version fallbacks between Python applications. Moreover, this process is version-specific, and every installed Python version has to have its own overlay. Anyhow, the operation itself is doable, and it could be very useful to keep the fixed set of Python libraries with proven compatibility and frozen versions. The overlay installation approach used here is based on XDG-compatible interpreter configuration scenario, sitecustomize.py, which is installed in /etc/python<version.affix>, most likely the modern system will have /etc/python3 and /etc/python3.12, rarely /etc/python2.7 on some distributions. This scenario should be installed version-wise, and all of them are serving the identical task. Due to that, these files should be written as minimally as possible. In my case, I just added these lines to the beginning of the files:

import sys
sys.path.append('/util/bin')
import PYTHONSTARTUP

yielding all the version difference handling to the PYTHONSTARTUP.py script. However, the older distributions containing both Python 2.7 and Python 3 require the specialised script to support the site-packages directory, whereas the newer ones do not need it, as well as Python 2 support.

The legacy version of the PYTHONSTARTUP script presented below shared the installation between two XDG trees: /util and /util/qt. The newer version does not need this; it automatically determines the exact Python version to find where the requested library should be installed.

Legacy Version for Mixed Python 2.7 / 3.9 Installation (Ubuntu 18-22)


# Needs distinguish between 'dist-packages' and 'site-packages' 
# for all the versions and all the XDG roots available

PYTHON2_UTILS = [   '/util/lib/python2.7/dist-packages',
                    '/util/qt/lib/python2.7/dist-packages' ]
PYTHON2_SITE_UTILS =    [   '/util/lib/python2.7/site-packages',
                            '/util/qt/lib/python2.7/site-packages' ]

PYTHON3_UTILS = [   '/util/lib/python3.10/dist-packages',
                    '/util/qt/lib/python3.10/dist-packages' ]
PYTHON3_SITE_UTILS =    [   '/util/lib/python3.10/site-packages',
                            '/util/qt/lib/python3.10/site-packages' ]

# Version tracker

import sys
major_ver = sys.version_info[0]
additional_directories = []
#print('Python {} customized bootstrap by twdragon'.format(major_ver))

# Setting sys.path (PYTHONPATH equivalent)

# Python 2
if major_ver == 2:
    if len(PYTHON2_UTILS) != 0:
        oldpath = sys.path
        sys.path = []
        for i in PYTHON2_UTILS:
            sys.path.append(i)
        for i in oldpath:
            sys.path.append(i)
    if len(PYTHON2_SITE_UTILS) != 0 :
        from site import addsitedir
        for d in PYTHON2_SITE_UTILS: 
            addsitedir(d)

# Python3
elif major_ver == 3:
    if len(PYTHON3_UTILS) != 0:
        oldpath = sys.path
        sys.path = []
        for i in PYTHON3_UTILS:
            sys.path.append(i)
        for i in oldpath:
            sys.path.append(i)
    if len(PYTHON3_SITE_UTILS) != 0:
        from site import addsitedir
        for d in PYTHON3_SITE_UTILS: addsitedir(d)

if len(additional_directories) != 0:
    for d in additional_directories:
        sys.path.append(d)

Version for Python 3

import sys
major_ver = sys.version_info[0]

# TODO: Additional root directory should be customizable via the environment
PYTHON_UTILS_ROOT = '/util/python/' + str(major_ver)
PYTHON_UTILS_PREFIX = '/dist-packages'

# No longer needed, backward compatibility hook
PYTHON_SITE_UTILS_PREFIX = '/site-packages'

additional_directories = list()

# print('Python {} customized bootstrap by twdragon'.format(major_ver))

PYTHON_UTILS_DIR = [ PYTHON_UTILS_ROOT + PYTHON_UTILS_PREFIX ]
PYTHON_SITE_UTILS_DIR = PYTHON_UTILS_ROOT + PYTHON_SITE_UTILS_PREFIX # Legacy

sys.path = PYTHON_UTILS_DIR + sys.path

# Legacy
from site import addsitedir
addsitedir(PYTHON_SITE_UTILS_DIR)

After these modifications, the system is available (in most cases) to load and handle the software from the specified XDG trees and Python distribution directories.