Softsync
Sync softly
Softsync helps you create and manage symbolic links to real files. But, rather than
store the links as separate files, as with traditional symlinks (think: ln -s source target
),
the links are stored in a single manifest file, per directory. These kinds of links
are called "softlinks".
Softsync comprises a small collection of commands. The main one is the cp
command.
Use it to create new softlinks to real files (or other softlinks). It can also be
used to "materialise" softlinked files into copies of their real counterparts (in
either real or symbolic link form).
What's the point? This utility may be of use for simulating the benefits of symbolic links where the underlying storage supports the concept of files and directories, but does not provide direct support for symbolic links. A good example of this is Amazon's S3, which has no native method of storing a symbolic link to another object (and none of the many suggested workarounds to this are suitable for your use-case).
Requires
python >= 3.6
pip >= 21.3.1
Install
pip install softsync
alias ss=softsync
Usage
softsync -h
Usage: softsync cmd [-h] [args...]
commands:
cp
rm
ls
repair
cp
The cp
command is the central softsync command. It has three main functions,
depending on what parameters are passed. First, it creates softlinks, either to
real files or to other softlinks. Second, it can reconstruct a root for just a
selection of files, into another root, following softlinks as required in order
to ensure that the real files that are softlinked to are present in the
reconstructed root. And third, it can materialise real (or symlinked) copies
of softlinked files from one root into another.
softsync cp -h
usage: softsync cp [-h] [-R src[:dest]] [-f] [-r] [-c] [-s modes] [-v] [--dry]
src-path [dest-path]
positional arguments:
src-path
dest-path
optional arguments:
-h, --help show this help message and exit
-R src[:dest], --root src[:dest]
root dir(s)
-f, --force copy over duplicates
-r, --recursive recurse into sub-directories
-c, --reconstruct reconstruct file hierarchy
-s modes, --sync modes
any of: symbolic,hardlink,copy
-v, --verbose verbose output
--dry dry run only
rm
The rm
command can be used to remove existing softlinks (or even real files,
if you really want to).
softsync rm -h
usage: softsync rm [-h] [-R root] [-f] [-r] [-v] [--dry] path
positional arguments:
path
optional arguments:
-h, --help show this help message and exit
-R root, --root root root dir
-f, --force copy over duplicates
-r, --recursive recurse into sub-directories
-v, --verbose verbose output
--dry dry run only
ls
The ls
command can be used to produce listings the files in a directory,
including any softlinks present.
softsync ls -h
usage: softsync ls [-h] [-R root] path
positional arguments:
path
optional arguments:
-h, --help show this help message and exit
-R root, --root root root dir
repair
The repair
command can be used to re-write the softsync manifest file in a
given directory according to the current contents of the directory - this may
be necessary if the softlink manifest gets into conflict with the real files
present.
softsync repair -h
usage: softsync repair [-h] [-R root] [-r] [-v] [--dry] path
positional arguments:
path
optional arguments:
-h, --help show this help message and exit
-R root, --root root root dir
-r, --recursive recurse into sub-directories
-v, --verbose verbose output
--dry dry run only
Note: none of the commands support the --recursive option, yet.
Examples
Start with a directory containing some regular files and folders, like this:
./
├── alpha/
│ └── foo/
│ ├── hello.txt
│ └── world.txt
├── omega/
└── zeta/
This represents three "root" directories, alpha
, omega
and zeta
. The first
starts out with some normal files. The second two are empty, for now.
First make a soft copy of the foo/hello.txt
file into a new bar
subdirectory,
within the alpha
root:
softsync cp -R alpha foo/hello.txt bar
This will yield:
./
├── alpha/
│ ├── foo/
│ │ ├── hello.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
├── omega/
└── zeta/
Where the new softsync manifest file ./alpha/bar/.softsync
will contain:
{
"softlinks": [
{
"name": "hello.txt",
"link": "../foo/hello.txt"
}
]
}
Then make a soft copy of the foo/world.txt
file, but this time giving the
copy a different name:
softsync cp -R alpha foo/world.txt bar/mars.txt
Now the manifest will contain:
{
"softlinks": [
{
"name": "hello.txt",
"link": "../foo/hello.txt"
},
{
"name": "mars.txt",
"link": "../foo/world.txt"
}
]
}
The ls
command can be used to list the contents of a directory, eg:
softsync ls -R alpha foo
Yields:
hello.txt
world.txt
And:
softsync ls -R alpha bar
Yields:
hello.txt -> ../foo/hello.txt
mars.txt -> ../foo/world.txt
Now, reconstruct a root from an existing one for just the file or files you are interested in:
softsync cp -R alpha:omega bar/mars.txt --reconstruct
Yields:
./
├── alpha/
│ ├── foo/
│ │ ├── hello.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
├── omega/
│ ├── foo/
│ │ └── world.txt
│ └── bar/
│ └── .softsync
└── zeta/
Where the softsync manifest file in ./omega/bar/.softsync
will have the
same contents as in the first step above.
Now reconstruct based on the hello.txt
file, but this time using the
symbolic option:
softsync cp -R alpha:omega bar/hello.txt --reconstruct --sync=symbolic
This yields:
./
├── alpha/
│ ├── foo/
│ │ ├── hello.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
├── omega/
│ ├── foo/
│ │ ├── hello.txt -> ../../alpha/foo/world.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
└── zeta/
The ./omega/bar/.softsync
manifest file will now contain softlinks for
both files.
Finally, make materialised copies of files that may exist only as
softlinks, optionally using the symbolic
option to produce symlinks
if desired:
softsync cp -R omega:zeta bar/hello.txt
softsync cp -R omega:zeta bar/mars.txt --sync=symbolic
Yields:
./
├── alpha/
│ ├── foo/
│ │ ├── hello.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
├── omega/
│ ├── foo/
│ │ ├── hello.txt -> ../../alpha/foo/world.txt
│ │ └── world.txt
│ └── bar/
│ └── .softsync
└── zeta/
└── bar/
├── hello.txt
└── mars.txt -> ../../omega/foo/world.txt
Where the new hello.txt
is a regular copy of the original hello.txt
file,
and mars.txt
is a symlink pointing to the original world.txt
file.
The cp
command supports the normal globbing patterns characters
in the source path, e.g: *.txt
and h?llo.*
, etc. Note you will
probably need to single quote glob patterns to prevent the shell from
expanding them first.
The cp
command also supports copying all the files in a directory,
just pass the directory itself as the source path parameter.
Programmatic usage
The command line interface is just that, an interface. All the commands can be used programmatically by importing the softsync API into your Python code.
The following is the programmatic equivalent of a couple of the commands from the CLI examples above (and assuming the same working directory):
from pathlib3x import Path
from softsync.common import Root, Options, Sync
from softsync.commands.cp import softsync_cp
# softsync cp -R alpha foo/world.txt bar/mars.txt
root = Root("alpha")
src_path = Path("foo/world.txt")
dest_path = Path("bar/mars.txt")
files = softsync_cp(
root,
src_path=src_path,
dest_path=dest_path,
)
for file in files:
print(file)
# softsync cp -R alpha:omega bar/mars.txt --reconstruct
src_root = Root("alpha")
dest_root = Root("omega")
src_path = Path("bar/mars.txt")
options = Options(
reconstruct=True,
)
files = softsync_cp(
src_root=src_root,
src_path=src_path,
dest_root=dest_root,
options=options,
)
for file in files:
print(file)
# softsync cp -R omega:zeta bar/mars.txt --sync=symbolic
src_root = Root("omega")
dest_root = Root("zeta")
src_path = Path("bar/mars.txt")
options = Options(
sync=[Sync.SYMBOLIC],
)
files = softsync_cp(
src_root=src_root,
src_path=src_path,
dest_root=dest_root,
options=options,
)
for file in files:
print(file)
When used programmatically, the API is even more flexible. For example, it can be provided with a file name mapping function, which will be used when copying multiple files from source to destination. Custom file filtering functions can also be can be used to select which files to copy.