Tools for Manipulating NEF [NMR Exchange Format] Files and Foreign File Access

NEF-Pipelines is a set of command line (currently... there maybe a gui later!) tools for maniulating [NEF] or NMR Exchange Format files which can be used to move NMR meta data [peaks, shifts, sequences etc] between NMR data processing programs.The programs provide basic tools for manipulating nef files:

Please note NEF-Pipelines is beta software and is quite new so if you do find problems put in an issue on the issue tracker. Even better if you can help improve the software do get in touch, there is much to do!

• molecular chains: listing, renaming and cloning molecular chains
• save-frames [tables]: deleting, inserting, listing and pretty printing (tabulate)
• headers creation / updating NEF headers with correct UUIDs and history
• streaming: NEF files into a pipeline
• testing: self testing of NEF pipelines

It also provides tools for importing and exporing non NEF files from the following programs [transcoding / translators]

• fasta
  • sequence [read and write]
• mars
  • shifts [write]
• nmrpipe
  • sequences [read]
  • shifts [read]
  • peaks [read]
• nmrview / nmrfx
  • peaks [read and write]
  • sequence [read]
  • shifts [read]
• pales
  • rdc restraints [write]
  • rdc restraints template [write]
• pdbx
  • sequence [read]
• xplor
  • sequence [read]
  • dihedral restraints [read]
  • distance restraints 'single atom selections [read]

if you have ccpn assign its simple to install NEF-Pipelines using a supplied macro install_nef_pipelines.sh

# if assign is on your path type

install_nef_pipelines

# else if this fails and ccpnmr/3.2.1/bin/assign is where the ccpn assign program is on disk run:

ccpnmr/3.2.1/bin/install_nef_pipelines

NEF-Pipelines can also be installed using pip and pipx

Firstly install pipx if you need to, using the commands for your OS ...

python3 -m pip install --user pipx
python3 -m pipx ensurepath

1. make sure you have python3 installed
2. check if you have pipx installed by typing pipx --version in a command prompt
3. if you don't have pipx installed, install it by typing

python3 -m pip install --user pipx
python3 -m pipx ensurepath

4. close your terminal window and open a new one an type

follow the instruction for linux distro of choice inside WSL2...

sudo apt install pipx python3-venv

sudo dnf install pipx

sudo zypper install python3-pipx

pipx install nef-pipelines

after the installation has completed you can type

nef

and should see output that starts

Usage: nef [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                        ✔ ╱ took 14s  ╱ nef_pipelines  ╱ at 23:29:36 

Options:
...

pipx upgrade nef-pipelines

All commands also have a help options triggered using --help or when no input is detected. So for example the nef command on its own produces

Usage: nef [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.

Commands:
  chains   - carry out operations on chains
  fasta    - read and write fasta sequences
  frames   - carry out operations on frames in nef frames
  header   - add a header to the stream
  mars     - export mars [shifts and sequences]
  nmrpipe  - read nmrpipe [peaks shifts & sequencess]
  nmrview  - read and write nmrview [peaks, sequences & shifts]
  pales    - read and write pales/dc [rdcs]
  pdb      - read pdb [sequences]
  stream   - stream a nef file
  test     - run the test suite

Individual nef pipeline componenets can be used standalone by using the --in or --pipe parameter which will either read a foreign file for translators or read a NEF file for commands that manipulate NEF files. So for example to lists the frames in NEF frame you can type

nef frames list --in test_data/pales_test_1.nef

this will produce

entry pales_test

nef_molecular_system  nef_rdc_restraint_list_test_1

which shows the entry name followed by the frames in the file test_data/pales_test_1.nef. However, the real power in NEF-Pipelines comes from combining pipeline commands together or with command line tools. For example

nef header                                   \
| nef fasta import sequence tailin1.fasta    \
| src/nef nmrview import shifts tailin1.out  \
| src/nef nmrview import peaks tailin1.out   \
> tailin1.nef

will create a valid NEF header followed by the tailin1 sequence as a molecular system and then import shifts and peaks for talin1 before writing a new NEF file tailin1.nef

The commands provided by NEF-Pipelines are hierarchical in nature. All commands are call by the NEF command but there are sub commands so for example to import a sequence froma pdb file tou would type nef psb import sequence followed by the name of the PDB file and relevant options. This can look lomng winded bu NEF-Pipelines support command completion so typing a double tab will list all availabe sub commands.

For example

nef<tab><tab>

shows

chains   fasta    frames   header   mars     nmrpipe  nmrview  pales    pdb      stream   test

and

nef nmrview import<tab<tab>

shows

peaks sequence shifts

The origional NEF paper is not a good description of the format more an outline of ideals and needs. The CCPN website has a good guide to nef and development takes place on github via the NEF consortium repository which includes an annotated NEF example and a dictionary defining the format. NEF is a STAR format which means it has similarities to NMRStar files used by the BMRB and PDBx / mmCIF used by the RCSB / PDB because its uses the same underling format. However, its is not directly interchangeable with these file formats as it has a different syntax and intention / underlying use easy and accurate NMR data interchange] as opposed to archiving for example.