Lexical analysis functions, tokenisers, transcribers: an arbitrary assortment of lexical and tokenisation functions useful for writing recursive descent parsers, of which I have several. There are also some transcription functions for producing text from various objects, such as `hexify` and `unctrl`.


Keywords
python2, python3
Licenses
GPL-3.0/GPL-3.0+
Install
pip install cs.lex==20240211

Documentation

This a mirror of my personal kit, containing thousands of scripts and hundreds of Python modules and assorted other code.

Some relevant links:

An autoindex of the command scripts is in the file 1INDEX.txt.

=Installation=

The Python modules on PyPI can just be installed with:

pip install cs.module_name

Installing the whole source tree can be done as follows:

  • Obtain the tree, for example by cloning this repository into a suitable directory such as /opt/css, which I will assume for the sake of example below.

  • Configure your environment to use the scripts:

    . /opt/css/env.sh

    That line should be in your .profile or other environment setup script. It's pretty simple; it adds the css components to the END of your various PATHs. It is expected that this should be one of the last things in your setup, so that is merely adds to your environment instead of preempting things. If your install point is not /opt/css, set the environment variable $OPTCSS to the install point before sourcing the $OPTCSS/env.sh.

The configuration script mostly does the following:

PATH=$PATH:/opt/css/bin
MANPATH=$MANPATH:/opt/css/man
PYTHONPATH=$PYTHONPATH:/opt/css/lib/python
CLASSPATH=$CLASSPATH:/opt/css/lib/java/au.com.zip.cs.jar
PERL5LIB=$PERL5LIB:/opt/css/lib/perl
export PATH MANPATH PYTHONPATH CLASSPATH PERL5LIB

Several of the scripts expect the following shell environment variables:

  • $HOST: your machine's short name. Defaults to $HOSTNAME without the trailing components.
  • $HOSTNAME: your machine's fully qualified domain name. Defaults to hostname.
  • $ARCH: your architecture, in the form vendor.cpu.os, for example sun.sparc.solaris or sgi.mips.irix.
  • $SYSTEMID: a name for your machines' administrative domain. For example, I use home for my home LAN.
  • $USER: your login name.
  • $SITENAME: your email domain (eg cskk.id.au for me).
  • $EMAIL: your email address (just who@where, no "<>"s); normally $USER@$SITENAME. There are some others, but those cover the common stuff.

Licence: You're free to use, modify and redistribute these scripts provided that:

  • you leave my code marked as mine and your modifications (if any) marked as yours; it's enough to prefix your code with a terse comment like: # change text - reason text - yourname <youremail> - date
  • you make recipients aware that the scripts can be obtained for free from my own web page

Warranty: None whatsoever!

These scripts work for me, but any of them may have bugs or be arbitrarily dependent on my own login environment. While I try to make most of them usable by others (and am happy to hear suggestions or bug reports), they're for my use and may well not meet your needs.

But feel free to hack them to meet your needs.