github.com/NeowayLabs/nash/ast

Nash stands for Nash shell.


Keywords
nash, nash-scripts, programming-language, shell
License
Apache-2.0
Install
go get github.com/NeowayLabs/nash/ast

Documentation

nash

Build Status codecov.io

Nash is a Linux system shell that attempts to be more safe and give more power to user. It's safe in the sense of it's far more hard to shoot yourself in the foot, (or in the head). It gives power to the user in the sense that you really can use nash to script deploys taking advantage of linux namespaces, cgroups, unionfs, etc, in an idiomatic way.

It's more safe for devops/sysadmins because it doesn't have the unsafe features of all former shells and it aim to have a very simple, safe and frozen syntax specification. Every shell feature considered harmful was left behind, but some needed features are still missing.

Nash is inspired by Plan 9 rc shell, but with very different syntax and purpose.

Show time!

asciicast

Go ahead:

go get github.com/NeowayLabs/nash/cmd/nash
# Make sure GOPATH/bin is in yout PATH
nash
λ> echo "hello world"
hello world

Pipes works like borne shell and derivations:

λ> cat spec.ebnf | wc -l
108

Output redirection works like Plan9 rc, but not only for filenames. It supports output redirection to tcp, udp and unix network protocols.

# stdout to log.out, stderr to log.err
λ> ./daemon >[1] log.out >[2] log.err
# stderr pointing to stdout
λ> ./daemon-logall >[2=1]
# stdout to /dev/null
λ> ./daemon-quiet >[1=]
# stdout and stderr to tcp address
λ> ./daemon >[1] "udp://syslog:6666" >[2=1]
# stdout to unix file
λ> ./daemon >[1] "unix:///tmp/syslog.sock"

To assign command output to a variable exists the '<=' operator. See the example below:

fullpath <= realpath $path | tr -d "\n"
echo $fullpath

To avoid problems with spaces in variables being passed as multiple arguments to commands, nash pass the contents of each variable as a single argument to the command. It works like enclosing every variable with quotes before executing the command. Then the following example do the right thing:

fullname = "John Nash"
./ci-register --name $fullname --option somevalue

On bash you need to enclose the $fullname variable in quotes to avoid problems.

Nash syntax does not support shell expansion from strings. There's no way to do things like the following in nash:

echo "The date is: $(date +%D)"

Instead you need to assign each command output to a proper variable and then concat it with another string when needed. In nash, the example above must be something like that:

today <= date "+%D"
echo "The date is: " + $today

The concat operator (+) could be used between variables and literal strings.

Functions can be declared with "fn" keyword:

fn cd(path) {
    fullpath <= realpath $path | tr -d "\n"
    cd $path
    PROMPT="[" + $fullpath + "]> "
    setenv PROMPT
}

And can be invoked as a normal function invocation:

λ> cd("/etc")
[/etc]>

Functions are commonly used for nash libraries, but when needed it can be bind'ed to some command name. Using the cd function below, we can override the builtin cd with that command with bindfn statement.

λ> # bindfn syntax is:
λ> # bindfn <function-name> <cmd-name>
λ> bindfn cd cd
λ> cd /var/log
[/var/log]>

The only control statements available are if, else and for. But for isn't implemented yet, because it was not required on my personal projects yet. In the same way, nash doesn't support shell expansion at if condition. For check if a directory exists you must use:

-test -d $rootfsDir    # if you forget '-', the script will be aborted here
                       # if path not exists

if $status != "0" {
        echo "RootFS does not exists."
        exit $status
}

Nash stops executing the script at first error found and, in the majority of times, it is what you want (specially for deploys). But Commands have an explicitly way to bypass such restriction by prepending a dash '-' to the command statement. For example:

fn cleanup()
        -rm -rf $buildDir
        -rm -rf $tmpDir
}

The dash '-' works only for operating system commands, other kind of errors are impossible to bypass. For example, trying to evaluate an unbound variable aborts the program with error.

λ> echo $PATH
/bin:/sbin:/usr/bin:/usr/local/bin:/home/user/.local/bin:/home/user/bin:/home/user/.gvm/pkgsets/go1.5.3/global/bin:/home/user/projects/3rdparty/plan9port/bin:/home/user/.gvm/gos/go1.5.3/bin
λ> echo $bleh
ERROR: Variable '$bleh' not set

Container features

Below are some facilities for namespace management inside nash. Make sure you have USER namespaces enabled in your kernel:

zgrep CONFIG_USER_NS /proc/config.gz
CONFIG_USER_NS=y

If it's not enabled you will need root privileges to execute every example below...

Creating a new process in a new USER namespace (u):

λ> id
uid=1000(user) gid=1000(user) groups=1000(user),98(qubes)
λ> rfork u {
     id
}
uid=0(root) gid=0(root) groups=0(root),65534

Yes, Linux supports creation of containers by unprivileged users. Tell this to the customer success of your container-infrastructure-vendor. :-)

The default UID mapping is: Current UID (getuid) => 0 (no range support). I'll look into more options for this in the future.

Yes, you can create multiple nested user namespaces. But kernel limits the number of nested user namespace clones to 32.

λ> rfork u {
    echo "inside first container"

    id

    rfork u {
        echo "inside second namespace..."

        id
    }
}

You can verify that other types of namespace still requires root capabilities, see for PID namespaces (p).

λ> rfork p {
    id
}
ERROR: fork/exec ./nash: operation not permitted

The same happens for mount (m), ipc (i) and uts (s) if used without user namespace (u) flag.

The c flag stands for "container" and is an alias for upmnis (all types of namespaces). If you want another shell (maybe bash) inside the namespace:

λ> rfork c {
    bash
}
[root@stay-away nash]# id
uid=0(root) gid=0(root) groups=0(root),65534
[root@stay-away nash]# mount -t proc proc /proc
[root@stay-away nash]#
[root@stay-away nash]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  34648  2748 pts/4    Sl   17:32   0:00 -rcd- -addr /tmp/nash.qNQa.sock
root         5  0.0  0.0  16028  3840 pts/4    S    17:32   0:00 /usr/bin/bash
root        23  0.0  0.0  34436  3056 pts/4    R+   17:34   0:00 ps aux

Everything except the rfork is like a common shell. Rfork will spawn a new process with the namespace flags and executes the commands inside the block on this namespace. It has the form:

rfork <flags> {
    <statements to run inside the container>
}

OK, but how scripts should look like?

Take a look in the script below:

#!/usr/bin/env nash
# Library for build Linux Stali rootfs and execute a container in the image
# For more info about stali, see: http://sta.li/
# Requires: build-essential mawk

imageName    = "stali-x86_64"
imageDir     = "/home/" + $USER + "/.nash/images/stali"
toolchainDir = $imageDir + "/toolchain"
srcDir       = $imageDir + "/src"
rootfsDir    = $imageDir + "/rootfs-x86_64"

echo "Build stali rootfs"
echo "Requires: build-essential mawk"

fn cleanup() {
    -rm -rf $toolchainDir
    -rm -rf $srcDir
    -rm -rf $rootfsDir
}

fn download() {
    git clone --depth=1 http://git.sta.li/toolchain
    git clone --depth=1 http://git.sta.li/src
}

fn stali_fsbuild(service) {
    cp -a $service /tmp/

    -mkdir -p $imageDir

    cd $imageDir

    cleanup()
    download()

    mkdir $rootfsDir
    STALI_SRC = $PWD + "/src"

    cd src

    sed -i.bak "s#^DESTDIR=#DESTDIR=" + $rootfsDir + "#g" config.mk
    make
    make install

    cd ..

    serviceName <= basename $service | tr -d "\n"
    cp -a "/tmp/"+$serviceName $rootfsDir

    tar cvf $imagename+".tar" $rootfsDir
    bzip2 $imageName+".tar"

    echo "Stali image generated: " + $imageName + ".tar.bz2"
}

# Run the $exec filename inside a container with $image rootfs
fn stali_run(image, exec) {
    targetDir = "/tmp/stali-rootfs"
    -mkdir $targetDir
    tar xvf $image -C $targetDir

    # Create a container with User, pid, mount, ipc and uts namespaces
    rfork upmis {
        # mount the process filesystem inside PID namespace
        mount -t proc proc /proc
        cp /etc/resolv.conf $targetDir+"/etc/resolv.conf"
        cp /etc/hosts $targetDir+"/etc/hosts"

        chroot $targetDir $exec
    }
}

Invoke the functions stali_fsbuild for build the image and stali_run to execute the container image. Or bindfn the functions for some command.

Pass "/bin/sh" to stali_run if you want a shell inside Stali.

I know, I know, lots of questions in how to handle the hard parts of deploy. My answer is: Coming soon.

Didn't work?

I've tested in the following environments:

Linux 4.1.13 (amd64)
Fedora release 23

Linux 4.3.3 (amd64)
Archlinux

Linux 4.1.13 (amd64)
Debian 8

Language specification

The specification isn't complete yet, but can be found here. The file spec_test.go makes sure it is sane.

Some Bash comparisons

Bash Nash Description
GOPATH=/home/user/gopath GOPATH="/home/user/gopath" Nash enforces quoted strings
GOPATH="$HOME/gopath" GOPATH=$HOME+"/gopath" Nash doesn't do string expansion
export PATH=/usr/bin PATH="/usr/bin"
setenv PATH
setenv operates only on valid variables
export showenv
ls -la ls -la Simple commads are identical
ls -la "$GOPATH" ls -la $GOPATH Nash variables shouldn't be enclosed in quotes, because it's default behaviour
./worker 2>log.err 1>log.out ./worker >[2] log.err >[1] log.out Nash redirection works like plan9 rc
./worker 2>&1 ./worker >[2=1] Redirection map only works for standard file descriptors (0,1,2)

Security

The PID 1 of every namespace created by nash is the same nash binary reading commands from the parent shell via unix socket. It allows the parent namespace (the script that creates the namespace) to issue commands inside the child namespace. In the current implementation the unix socket communication is not secure yet.

Concept

Nowadays everyone agrees that a good deploy requires containers, but why this kind of tools (docker, rkt, etc) and libraries (lxc, libcontainer, etc) are so bloated and magical?

In the past, the UNIX sysadmin had the complete understanding of the operating system and the software being deployed. All of the operating system packages/libraries going to production and the required network configurations in every machine was maintained by several (sometimes un-mantainable) scripts. Today we know that this approach have lots of problems and the container approach is a better alternative. But in the other end, we're paying a high cost for the lose of control. The container-technologies in the market are very unsafe and few people are worrying about. No one knows for hundred percent sure, how the things really works because after every release it's done differently. On my view it's getting worse and worse...

Before Linux namespace, BSD Jails, Solaris Zones, and so on, the sysadmin had to fight the global view of the operating system. There was only one root mount table, only one view of devices and processes, and so on. It was a mess. This approach then proved to be much harder to scale because of the services conflicts (port numbers, files on disk, resource exhaustion, etc) in the global OS interface. The container/namespace idea creates an abstraction to the process in a way that it thinks it's the only process running (not counting init), it is the root (or no) and then, the filesystem of the container only has the files required for it (nothing more).

What's missing is a safe and robust shell for natural usage of namespace/container ideas for everyone (programmers, sysadmins, etc).

Nasn is a way for you, that understand the game rules, to make reliable deploy scripts using the good parts of the container technologies. If you are a programmer, you can use a good language to automate the devops instead of relying on lots of different technologies (docker, rkt, k8s, mesos, terraform, and so on). And you can create libraries for code-reuse.

It's only a simple shell plus a keyword called rfork. Rfork try to mimic what Plan9 rfork does for namespaces, but with linux limitations in mind.

Motivation

I needed to create test scripts to be running on different mount namespaces for testing a file server and various use cases. Using bash in addition to docker or rkt was not so good for various reasons. First, docker prior to version 1.10 doesn't support user namespaces, and then my make test would requires root privileges, but for docker 1.10 user namespace works still requires to it being enabled in the daemon flags (--userns-remap=?) making more hard to work on standard CIs (travis, circle, etc)... Another problem was that it was hard to maintain a script, that spawn docker container scripts inheriting environment variables from parent namespace (or host). Docker treats the container as a different machine or VM, even calling the parent namespace as "host". This breaks the namespace sharing/unsharing idea of processes. What I wanted was a copy of the missing plan9 environment namespace to child namespaces.

Want to contribute?

Open issues and PR :) The project is in an early stage, be patient because things can change in the future.