Exodus is a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another. This is useful in situations where you don't have root access on a machine or where a package simply isn't available for a given Linux distribution. For example, CentOS 6.X and Amazon Linux don't have packages for Google Chrome or aria2. Server-oriented distributions tend to have more limited and outdated packages than desktop distributions, so it's fairly common that one might have a piece of software installed on their laptop that they can't easily install on a remote machine.
With exodus, transferring a piece of software that's working on one computer to another is as simple as this.
exodus aria2c | ssh intoli.com
Exodus handles bundling all of the binary's dependencies, compiling a statically linked wrapper for the executable that invokes the relocated linker directly, and installing the bundle in ~/.exodus/
on the remote machine.
You can see it in action here.
Table of Contents
- The Problem Being Solved - An overview of some of the challenges that arise when relocating binaries.
- Installation - Instructions for installing exodus.
-
Usage
- The Command-Line Interface - The options supported by the command-line utility.
- Usage Examples - Common usage patterns, helpful for getting started quickly.
- How It Works - An overview of how exodus works.
- Known Limitations - Situations that are currently outside the scope of what exodus can handle.
- Development - Instructions for setting up the development environment.
- Contributing - Guidelines for contributing.
- License - License details for the project.
The Problem Being Solved
If you simply copy an executable file from one system to another, then you're very likely going to run into problems. Most binaries available on Linux are dynamically linked and depend on a number of external library files. You'll get an error like this when running a relocated binary when it has a missing dependency.
aria2c: error while loading shared libraries: libgnutls.so.30: cannot open shared object file: No such file or directory
You can try to install these libraries manually, or to relocate them and set LD_LIBRARY_PATH
to wherever you put them, but it turns out that the locations of the ld-linux linker and the glibc libraries are hardcoded.
Things can very quickly turn into a mess of relocation errors,
aria2c: relocation error: /lib/libpthread.so.0: symbol __getrlimit, version
GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
segmentation faults,
Segmentation fault (core dumped)
or, if you're really unlucky, this very confusing symptom of a missing linker.
$ ./aria2c
bash: ./aria2c: No such file or directory
$ ls -lha ./aria2c
-rwxr-xr-x 1 sangaline sangaline 2.8M Jan 30 21:18 ./aria2c
Exodus works around these issues by compiling a small statically linked launcher binary that invokes the relocated linker directly with any hardcoded RPATH
library paths overridden.
The relocated binary will run with the exact same linker and libraries that it ran with on its origin machine.
Installation
The package can be installed from the package on pypi.
Running the following will install exodus
locally for your current user.
pip install --user exodus-bundler
You will then need to add ~/.local/bin/
to your PATH
variable in order to run the exodus
executable (if you haven't already done so).
This can be done by adding
export PATH="~/.local/bin/:${PATH}"
to your ~/.bashrc
file.
Optional/Recommended Dependencies
It is also highly recommended that you install gcc and one of either musl libc or diet libc on the machine where you'll be packaging binaries. If present, these small C libraries will be used to compile small statically linked launchers for the bundled applications. An equivalent shell script will be used as a fallback, but it carries significant overhead compared to the compiled launchers.
Usage
Command-Line Interface
The command-line interface supports the following options.
usage: exodus [-h] [-c CHROOT_PATH] [-a DEPENDENCY] [-d] [--no-symlink FILE]
[-o OUTPUT_FILE] [-q] [-r [NEW_NAME]] [--shell-launchers] [-t]
[-v]
EXECUTABLE [EXECUTABLE ...]
Bundle ELF binary executables with all of their runtime dependencies so that
they can be relocated to other systems with incompatible system libraries.
positional arguments:
EXECUTABLE One or more ELF executables to include in the exodus
bundle.
optional arguments:
-h, --help show this help message and exit
-c CHROOT_PATH, --chroot CHROOT_PATH
A directory that will be treated as the root during
linking. Useful for testing and bundling extracted
packages that won run without a chroot. (default:
None)
-a DEPENDENCY, --add DEPENDENCY, --additional-file DEPENDENCY
Specifies an additional file to include in the bundle,
useful for adding programatically loaded libraries and
other non-library dependencies. The argument can be
used more than once to include multiple files, and
directories will be included recursively. (default:
[])
-d, --detect Attempt to autodetect direct dependencies using the
system package manager. Operating system support is
limited. (default: False)
--no-symlink FILE Signifies that a file must not be symlinked to the
deduplicated data directory. This is useful if a file
looks for other resources based on paths relative its
own location. This is enabled by default for
executables. (default: [])
-o OUTPUT_FILE, --output OUTPUT_FILE
The file where the bundle will be written out to. The
extension depends on the output type. The
"{{executables}}" and "{{extension}}" template strings
can be used in the provided filename. If omitted, the
output will go to stdout when it is being piped, or to
"./exodus-{{executables}}-bundle.{{extension}}"
otherwise. (default: None)
-q, --quiet Suppress warning messages. (default: False)
-r [NEW_NAME], --rename [NEW_NAME]
Renames the binary executable(s) before packaging. The
order of rename tags must match the order of
positional executable arguments. (default: [])
--shell-launchers Force the use of shell launchers instead of attempting
to compile statically linked ones. (default: False)
-t, --tarball Creates a tarball for manual extraction instead of an
installation script. Note that this will change the
output extension from ".sh" to ".tgz". (default:
False)
-v, --verbose Output additional informational messages. (default:
False)
Examples
Piping Over SSH
The easiest way to install an executable bundle on a remote machine is to pipe the exodus
command output over SSH.
For example, the following will install the aria2c
command on the intoli.com
server.
exodus aria2c | ssh intoli.com
This requires that the default shell for the remote user be set to bash
(or a compatible shell).
If you use csh
, then you need to additionally run bash
on the remote server like this.
exodus aria2c | ssh intoli.com bash
Explicitly Adding Extra Files
Additional files can be added to bundles in a number of different ways.
If there is a specific file or directory that you would like to include, then you can use the --add
option.
For example, the following command will bundle nmap
and include the contents of /usr/share/nmap
in the bundle.
exodus --add /usr/share/nmap nmap
You can also pipe a list of dependencies into exodus
.
This allows you to use standard Linux utilities to find and filter dependencies as you see fit.
The following command sequence uses find
to include all of the Lua scripts under /usr/share/nmap
.
find /usr/share/nmap/ -iname '*.lua' | exodus nmap
These two approaches can be used together, and the --add
flag can also be used multiple times in one command.
Auto-Detecting Extra Files
If you're not sure which extra dependencies are necessary, you can use the --detect
option to query your system's package manager and automatically include any files in the corresponding packages.
Running
exodus --detect nmap
will include the contents of /usr/share/nmap
as well as its man pages and the contents of /usr/share/zenmap/
.
If you ever try to relocate a binary that doesn't work with the default configuration, the --detect
option is a good first thing to try.
You can also pipe the output of strace
into exodus
and all of the files that are accessed will be automatically included.
This is particularly useful in situations where shared libraries are loaded programmatically, but it can also be used to determine which files are necessary to run a specific command.
The following command will determine all of the files that nmap
accesses while running the set of default scripts.
strace -f nmap --script default 127.0.0.1 2>&1 | exodus nmap
The output of strace
is then parsed by exodus
and all of the files are included.
It's generally more robust to use --detect
to find the non-library dependencies, but the strace
pattern can be indispensable when a program uses dlopen()
to load libraries programmatically.
Also, note that any files that a program accesses will be included in a bundle when following this approach.
Never distribute a bundle without being certain that you haven't accidentally included a file that you don't want to make public.
Renaming Binaries
Multiple binaries that have the same name can be installed in parallel through the use of the --rename
/-r
option.
Say that you have two different versions of grep
on your local machine: one at /bin/grep
and one at /usr/local/bin/grep
.
In that situation, using the -r
flag allows you to assign aliases for each binary.
exodus -r grep-1 -r grep-2 /bin/grep /usr/local/bin/grep
The above command would install the two grep
versions in parallel with /bin/grep
called grep-1
and /usr/local/bin/grep
called grep-2
.
Manual Extraction
You can create a compressed tarball directly instead of the default script by specifying the --tarball
option.
To create a tarball, copy it to a remote server, and then extract it in ~/custom-location
, you could run the following.
# Create the tarball.
exodus --tarball aria2c --output aria2c.tgz
# Copy it to the remote server and remove it locally.
scp aria2c.tgz intoli.com:/tmp/aria2c.tgz
rm aria2c.tgz
# Make sure that `~/custom-location` exists.
ssh intoli.com "mkdir -p ~/custom-location"
# Extract the tarball remotely.
ssh intoli.com "tar --strip 1 -C ~/custom-location -zxf /tmp/aria2c.tgz"
# Remove the remote tarball.
ssh intoli.com "rm /tmp/aria2c.tgz"
You will additionally need to add ~/custom-location/bin
to your PATH
variable on the remote server.
This can be done by adding the following to ~/.bashrc
on the remote server.
export PATH="~/custom-location/bin:${PATH}"
Adding to a Docker Image
Tarball formatted exodus bundles can easily be included in Docker images by using the ADD instruction.
You must first create a bundle using the --tarball
option
# Create and enter a directory for the Docker image.
mkdir jq
cd jq
# Generate the `exodus-jq-bundle.tgz` bundle.
exodus --tarball jq
and then create a Dockerfile
file inside of the jq
directory with the following contents.
FROM scratch
ADD exodus-jq-bundle.tgz /opt/
ENTRYPOINT ["/opt/exodus/bin/jq"]
The Docker image can then be built by running
docker build -t jq .
and jq
can be run inside of the container.
docker run jq
This simple image will include only the jq
binary and dependencies, but the bundles can be included in existing docker images in the same way.
For example, adding
ENV PATH="/opt/exodus/bin:${PATH}"
ADD exodus-jq-bundle.tgz /opt/
to an existing Dockerfile
will make the jq
binary available for use inside the container.
How It Works
There are two main components to how exodus works:
-
Finding and bundling all of a binary's dependencies.
-
Launching the binary in such a way that the proper dependencies are used without any potential interaction from system libraries on the destination machine.
The first component is actually fairly simple.
You can invoke ld-linux with the LD_TRACE_LOADED_OBJECTS
environment variable set to 1
and it will list all of the resolved library dependencies for a binary.
For example, running
LD_TRACE_LOADED_OBJECTS=1 /lib64/ld-linux-x86-64.so.2 /bin/grep
will output the following.
linux-vdso.so.1 => (0x00007ffc7495c000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f89b2f3e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f89b2b7a000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f0e95e8c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f89b3196000)
The linus-vdso.so.1
dependency refers to kernel space routines that are exported to user space, but the other four are shared library files on disk that are required in order to run grep
.
Notably, one of these dependencies is the /lib64/ld-linux-x86-64.so.2
linker itself.
The location of this file is typically hardcoded into an ELF binary's INTERP
header and the linker is invoked by the kernel when you run the program.
We'll come back to that in a minute, but for now the main point is that we can find a binary's direct dependencies using the linker.
Of course, these direct dependencies might have additional dependencies of their own.
We can iteratively find all of the necessary dependencies by following the same approach of invoking the linker again for each of the library dependencies.
This isn't actually necessary for grep
, but exodus does handle finding the full set of dependencies for you.
After all of the dependencies are found, exodus puts them together with the binary in a tarball that can be extracted (typically into either /opt/exodus/
or ~/.exodus
).
We can explore the structure of the grep
bundle by using tree combined with a sed
one-liner to truncate long SHA-256 hashes to 8 digits.
Running
alias truncate-hashes="sed -r 's/([a-f0-9]{8})[a-f0-9]{56}/\1.../g'"
tree ~/.exodus/ | truncate-hashes
will show us all of the files and folders included in the grep
bundle.
/home/sangaline/.exodus/
├── bin
│ └── grep -> ../bundles/3124cd96.../usr/bin/grep
├── bundles
│ └── 3124cd96...
│ ├── lib64
│ │ └── ld-linux-x86-64.so.2 -> ../../../data/dfd5de26...
│ └── usr
│ ├── bin
│ │ ├── grep
│ │ ├── grep-x -> ../../../../data/7477c1a7...
│ │ └── linker-dfd5de26...
│ └── lib
│ ├── libc.so.6 -> ../../../../data/6d0e1d45...
│ ├── libpcre.so.1 -> ../../../../data/a0862ebc...
│ └── libpthread.so.0 -> ../../../../data/85cb56a5...
└── data
├── 6d0e1d45...
├── 7477c1a7...
├── 85cb56a5...
├── a0862ebc...
└── dfd5de26...
8 directories, 13 files
You can see that there are three top-level directories within ~/.exodus/
: bin
, bundles
, and data
.
Let's cover these in reverse-alphabetical order, starting with the data
directory.
The data
directory contains the actual files from the bundles with names corresponding to SHA-256 hashes of their content.
This is done so that multiple versions of a file with the same filename can be extracted in the data
directory without overwriting each other.
On the other hand, files that do have the same content will overwrite each other.
This avoids the need to store multiple copies of the same data, even if the identical files appear in different bundles or directories.
Next, we have the bundles
directory, which is full of subfolders that also have SHA-256 hashes as names.
The hashes this time are determined based on the combined directory structure and content of everything included in the bundle.
The hash provides a unique fingerprint for the bundle and allows multiple bundles to be extracted without their directory contents mixing.
Inside of each bundle subdirectory, the original directory structure of the bundle's contents on the host machine is mirrored.
For this particular grep
bundle, there are lib64
, usr/bin
, and usr/lib
directories.
A more complicated bundle could include additional files from /usr/share
, /opt/local
, a user's home directory, or really anywhere on the system (see the --add
and --detect
options).
The files in both lib64
and usr/lib
simply consist of symlinks to the actual library files in the top-level data/
directory.
The usr/bin
directory is a little more complicated.
The grep
file isn't actually the original grep
binary, it's a special executable that exodus
constructs called a "launcher."
A launcher is a tiny program that invokes the linker and overrides the library search path in such a way that our original binary can run without any system libraries being used and causing issues due to incompatibilities.
The linker in this case is the linker-dfd5de26...
file.
This is located in the same directory so that resource paths can be resolved relative to the running executable.
Finally, the grep-x
symlink points to the actual grep
binary that was bundled and extracted in the top-level data/
directory (this is the ELF file that the linker interprets).
When a C compiler and either musl libc or diet libc are available, exodus will compile a statically linked binary launcher.
If neither of these are present, it will fall back to using a shell script to perform the task of the launcher.
This adds a little bit of overhead relative to the binary launchers, but they are helpful for understanding what the launchers do.
Here's the shell script version of the grep-launcher
, for example.
#! /bin/bash
current_directory="$(dirname "$(readlink -f "$0")")"
executable="${current_directory}/./grep-x"
library_path="../../lib64:../lib64:../../lib:../lib:../../lib32:../lib32"
library_path="${current_directory}/${library_path//:/:${current_directory}/}"
linker="${current_directory}/./linker-dfd5de2638cea087685b67786050dcdc33aac7b67f5f8c2753b7da538517880a"
exec "${linker}" --library-path "${library_path}" --inhibit-rpath "" "${executable}" "$@"
You can see that the launcher first constructs the full paths for all of the LD_LIBRARY_PATH
directories, the executable, and the linker based on its own location.
It then executes the linker with a set of arguments that allow it to search the proper library directories, ignore the hardcoded RPATH
, and run the binary with any command-line arguments passed along.
This serves a similar purpose to something like patchelf that would modify the INTERP
and RPATH
of the binary, but it additionally allows for both the linker and library locations to be specified based solely on their relative locations.
This is what allows for the exodus bundles to be extracted in ~/.exodus
, /opt/exodus/
, or any other location, as long as the internal bundle structure is preserved.
Continuing on with our reverse-alphabetical order, we finally get to the top-level bin
directory.
The top-level bin
directory consists of symlinks of the binary names to their corresponding launchers.
This allows for the addition of a single directory to a user's PATH
variable in order to make the migrated exodus binaries accessible.
For example, adding export PATH="~/.exodus/bin:${PATH}"
to a ~/.bashrc
file will add all of these entry points to a user's PATH
and allow them to be run without specifying their full path.
Known Limitations
There are several scenarios under which bundling an application with exodus will fail. Many of these are things that we're working on and hope to improve in the future, but some are fundamentally by design and are unlikely to change. Here you can see an overview of situations where exodus will not be able to successfully relocate executables.
-
Non-ELF Binaries - Exodus currently only supports completely bundling ELF binaries.
Interpreted executable files, like shell scripts, can be included in bundles, but they're shebang interpreter directives will not be changed.
This generally means that they will be interpreted using the system version of
bash
,python
,perl
, or whatever else. The problem that exodus aims to solve is largely centered around the dynamic linking of ELF binaries, so this is unlikely to change in the foreseeable future. - Incompatible CPU Architectures - Binaries compiled for one CPU architecture will generally not be able to run on a CPU of another architecture. There are some exceptions to this, for example x64 processors are backwards compatible with x86 instruction sets, but you will not be able to migrate x64 binaries to an x86 or an ARM machine. Doing so would require processor emulation, and this is definitely outside the scope of the exodus project. If you find yourself looking for a solution to this problem, then you might want to check out QEMU.
-
Incompatible Glibc and Kernel Versions - When glibc is compiled, it is configured to target a specific kernel version.
Trying to run any software that was compiled against glibc on a system using an older kernel version than glibc's target version will result in a
FATAL: kernel too old
error. You can check the oldest supported kernel version for a binary by runningfile /path/to/binary
. The output should include a string likefor GNU/Linux 2.6.32
which signifies the oldest kernel version that the binary is compatible with. As a workaround, you can create exodus bundles in a Docker image using an operating system image which supports older kernels (e.g. use an outdated version of the operating system). -
Driver Dependent Libraries - Unlike some other application bundlers, exodus aims to include all of the required libraries when the bundle is created and to completely isolate the transported binary from the destination machine's system libraries.
This means that any libraries which are compiled for specific hardware drivers will only work on machines with the same drivers.
A key example of this is the
libGLX_indirect.so
library which can link to eitherlibGLX_mesa.so
orlibGLX_nvidia.so
depending on which graphics card drivers are used on a given system. Bundling dependencies that are not locally available on the source machine is fundamentally outside the scope of what exodus is designed to do, and this will never change.
Development
The development environment can be setup by running the following.
# Clone the repository.
git clone git@github.com:intoli/exodus.git
cd exodus
# Create and enter a virtualenv.
virtualenv .env
. .env/bin/activate
# Install the development requirements.
pip install -r development-requirements.txt
# Install the exodus package in editable mode.
pip install -e .
The test suite can then be run using tox.
tox
Contributing
Contributions are welcome, but please create an issue on the issue tracker first to discuss the contribution first. New feature additions should include tests and it's a requirement that all tests must pass before pull requests are merged.
License
Exodus is licensed under a BSD 2-Clause License and is copyright Intoli, LLC.