files' catalogue manager


Keywords
file, manager, tags
License
GPL-3.0
Install
pip install Katal==0.2.4

Documentation

(1) Katal (0.3.3)

A (Python3/GPLv3/OSX-Linux-Windows/CLI) project, using no additional modules than the ones installed with Python3. This project requires the installation of one file (katal.py).

Create a catalogue from various source directories, without any duplicate. Add some tags to the files.

  • license : GPL-3.0 (License :: OSI Approved :: GNU General Public License v3 (GPLv3))
  • status : Beta (Development Status :: 5 - Production/Stable)

(2) purpose and caveats

Read a directory, select some files according to a configuration file (leaving aside the duplicates), copy the selected files in a target directory. Once the target directory is filled with some files, a database is added to the directory to avoid future duplicates. You can add new files to the target directory by using Katal one more time, with a different source directory.

Katal can also be used to create a "mirror database" from a source, without copying the files in the target directory (see the mode='nocopy' option, defined in the configuration file).

Caveats :

  • Katal uses sqlite3 databases since sqlite3 is supported out-of-the-box by Python
  • Katal is slow, very slow : you could speed up the execution by reducing the amount of console messages (see the --verbosity option) but the core is absolutely inefficient.
  • Katal isn't immune to unexpected shutdowns/program's stops.
  • please use the --usentfsprefix option if you read files from a NTFS volume.

(3) installation and tests

Don't forget : Katal is Python3 project, not a Python2 project !

$ pip3 install katal

or

$ wget https://raw.githubusercontent.com/suizokukan/katal/master/katal/katal.py
Since katal.py is a stand-alone file, you may place this file in the target directory.

tests :
Beware ! Due to the limitations of OSX and Windows systems, these tests are to be runned
on Linux systems.

$ python -m unittest tests/tests.py

or

$ nosetests

(4) workflow

$ katal -h
... will display all known parameters.
$ katal --version
... wil display the version of the current script.

note : using the --off option allows the user to use Katal without modifying the target 
directory; with --off, no file will be written and the database will not change.

note : Katal NEVER simply deletes a file : the script move ALWAYS the file to be deleted in its
trash directory.

Create the target directory :

$ katal --new myworkingdirectory

It may be convenient to go inside the target directory, e.g. :
$ cd myworkingdirectory

If Katal wasn't installed on the computer ($ pip3 install katal), it may be convenient to copy the 
script katal.py inside the target directory, e.g. :
$ cp katal.py myworkingdirectory

Then, modify the .ini file (myworkingdirectory/.katal/katal.ini) and choose a source :

Inside the .ini file, modify the following lines :

[source]
path : ~/src/

[target]
mode : copy   # 'copy', 'move' or 'nocopy'

->  'copy'   : source files are copied into the target directory .
->  'move'   : source files are moved into the target directory .
->  'nocopy' : no source file is copied into the target directory (the
               target database being updated).

Take a look at the files stored in the source directory :

$ katal -si
or
$ katal --sourceinfos
Some informations are displayed : how many files lie in the source directory, what are
the extensions, and so on.

Choose which files have to be selected; modify the .ini file :

(example 1 : all .jpg (any size) and all .bmp greater then 5MB.)

eval : filter1 or filter2

[source.filter1]
iname : .*\.jpg$

[source.filter2]
iname : .*\.bmp$
size : >= 5MB

(example 2 : all jpg,tif,bmp,png greater then 1MB.)
eval : filter1 and (filter2 or filter3 or filter4 or filter5)
[source.filter1]
size : >1MB
[source.filter2]
iname : .*\.jpg$
[source.filter3]
iname : .*\.tif$
[source.filter4]
iname : .*\.bmp$
[source.filter5]
iname : .*\.png$

Choose a name for the files in the target directory, e.g. :
name of the target files : birthday__%%dd__%%i.%%e
(%%dd : date (e.g. 2015_09_25_06_50)
(%%i : database index, like "0", "1", "2"...)
(%%e : extension, like "jpg")

Let's see what would happen if the script tried to select the files :

$ katal --select
... and answer 'yes' to the final question if all the details are ok to you.
The files will be copied/moved/left (see the mode option) in the target directory.

If you want a bit-to-bit check if two files have the same hashes, use the --strictcmp option.

See the result (ti : target informations)

$ katal -ti

If want to start from scratch just keeping the configuration file, you may delete all target 
files and flush the database with :
$ katal --reset
... and modify the configuration file and use again --select

Let's tag some files :

$ katal --addtag=tree --to=*.jpg
$ katal --addtag=birthday --to=myfile.bmp

Let's search some tags :

$ katal --findtag=tree

Let's search some files and copy the selected files in new directory :

$ katal --findtag=birthday --copyto=backup_birthday

Let's remove all the files without any tag :

$ katal --rmnotags
... the files will be copied in the trash directory (myworkingdirectory/.katal/trash)

Let's remove a file :

$ katal --targetkill=myfile.jpg

Let's remove some files :

$ katal --targetkill=file*.jpg

Let's copy a target directory into another one, which didn't exist :

$ katal --new=../target2

modify the configuration file in target2 : e.g. you may change the name of the files.

Then :
$ katal --rebase=../target2

Check if a file external to the target directory is already stored in it :

$ katal --whatabout=myfile

Check if the files external to the target directory are already stored in it :

$ katal --whatabout=mydir/

Display informations about a file belonging to the target directory :

$ katal --whatabout=myfile

(5) project's author and project's name

Xavier Faure (suizokukan / 94.23.197.37) : suizokukan @T orange D@T fr

The name Katal is derived from the Ancient Greek κατάλογος ("enrolment, register, catalogue").

(6) arguments

usage: katal.py [-h] [--add] [--addtag ADDTAG] [-cfg CONFIGFILE] [--cleandbrm]
                [--copyto COPYTO] [-dlcfg {local,home}] [--findtag FINDTAG]
                [--infos] [-n NEW] [--off] [--rebase REBASE] [--reset]
                [--rmnotags] [--rmtags] [-s] [--settagsstr SETTAGSSTR] [-si]
                [--strictcmp] [--targetpath TARGETPATH] [-ti] [-tk TARGETKILL]
                [--to TO] [--usentfsprefix] [--verbosity {none,normal,high}]
                [--version] [--whatabout WHATABOUT]

optional arguments:
  -h, --help            show this help message and exit
  --add                 # Select files according to what is described in the
                        configuration file then add them to the target
                        directory. This option can't be used with the --select
                        one.If you want more informations about the process,
                        please use this option in combination with --infos .
                        (default: False)
  --addtag ADDTAG       # Add a tag to some file(s) in combination with the
                        --to option. (default: None)
  -cfg CONFIGFILE, --configfile CONFIGFILE
                        # Set the name of the config file, e.g. config.ini
                        (default: None)
  --cleandbrm           # Remove from the database the missing files in the
                        target path. (default: False)
  --copyto COPYTO       # To be used with the --findtag parameter. Copy the
                        found files into an export directory. (default: None)
  -dlcfg {local,home}, --downloaddefaultcfg {local,home}
                        # Download the default config file and overwrite the
                        file having the same name. This is done before the
                        script reads the parameters in the config file. Use
                        'local' to download in the current directory, 'home'
                        to download in the user's HOME directory. (default:
                        None)
  --findtag FINDTAG     # Find the files in the target directory with the
                        given tag. The tag is a simple string, not a regex.
                        (default: None)
  --infos               # Display informations about the source directory
                        given in the configuration file. Help the
                        --select/--add options to display more informations
                        about the process : in this case, the --infos will be
                        executed before --select/--add (default: False)
  -n NEW, --new NEW     # Create a new target directory (default: None)
  --off                 # Don't write anything into the target directory or
                        into the database, except into the current log file.
                        Use this option to simulate an operation : you get the
                        messages but no file is modified on disk, no directory
                        is created. (default: False)
  --rebase REBASE       # Copy the current target directory into a new one :
                        you rename the files in the target directory and in
                        the database. First, use the --new option to create a
                        new target directory, modify the .ini file of the new
                        target directory (modify [target]name of the target
                        files), then use --rebase with the name of the new
                        target directory (default: None)
  --reset               # Delete the database and the files in the target
                        directory (default: False)
  --rmnotags            # Remove all files without a tag (default: False)
  --rmtags              # Remove all the tags of some file(s) in combination
                        with the --to option. (default: False)
  -s, --select          # Select files according to what is described in the
                        configuration file without adding them to the target
                        directory. This option can't be used with the --add
                        one.If you want more informations about the process,
                        please use this option in combination with --infos .
                        (default: False)
  --settagsstr SETTAGSSTR
                        # Give the tag to some file(s) in combination with the
                        --to option. Overwrite the ancient tag string. If you
                        want to empty the tags' string, please use a space,
                        not an empty string : otherwise the parameter given to
                        the script wouldn't be taken in account by the shell
                        (default: None)
  -si, --sourceinfos    # Display informations about the source directory
                        (default: False)
  --strictcmp           # To be used with --add or --select. Force a bit-to-
                        bit comparisionbetween files whose hashid-s is equal.
                        (default: False)
  --targetpath TARGETPATH
                        # Target path, usually '.' . If you set path to .
                        (=dot character), it means that the source path is the
                        current directory (=the directory where the script
                        katal.py has been launched) (default: .)
  -ti, --targetinfos    # Display informations about the target directory
                        (default: False)
  -tk TARGETKILL, --targetkill TARGETKILL
                        # Kill (=move to the trash directory) one file from
                        the target directory.DO NOT GIVE A PATH, just the
                        file's name, without the path to the target directory
                        (default: None)
  --to TO               # Give the name of the file(s) concerned by
                        --settagsstr. wildcards accepted; e.g. to select all
                        .py files, use '*.py' . Please DON'T ADD the path to
                        the target directory, only the filenames (default:
                        None)
  --usentfsprefix       # Force the script to prefix filenames by a special
                        string required by the NTFS for long filenames, namely
                        \\?\ (default: False)
  --verbosity {none,normal,high}
                        # Console verbosity : 'none'=no output to the console,
                        no question asked on the console; 'normal'=messages to
                        the console and questions asked on the console;
                        'high'=display discarded files. A question may be
                        asked only by using the following arguments : --new,
                        --rebase, --reset and --select (default: normal)
  --version             # Show the version and exit
  --whatabout WHATABOUT
                        # Say if the file[the files in a directory] already in
                        the given as a parameter is in the target directory
                        notwithstanding its name. (default: None)

(7) history / future versions

See the roadmap.txt file.

(8) technical details

(8.0) structure

see chart.txt

(8.1) exit codes

             0      : success
            -1      : can't read the parameters from the configuration file
            -2      : a KatalError exception has been raised
            -3      : an unexpected exception exception has been raised

(8.2) configuration file

The informations stored in the configuration file are written in the CFG_PARAMETERS global
variable. CFG_PARAMETERS is filled by read_parameters_from_cfgfile() and is a
configparser.Configparser object.
The default name of the configuration file is set by the DEFAULT_CONFIGFILE_NAME global variable
and may be set by the commande line (see --configfile option).

[log file]    : parameters about the logfile
use log file  : True/False; if False, all the messages are written only to the console.
name          : no quotation mark here !

[display]         : parameters about the way informations are displayed
target filename.max length on console : (max length of the file names displayed)
source filename.max length on console : (max length of the file names displayed)

[source]          : parameters about source directory 

[source.filterN]   : N is an integer greater or equal to  1; [source.filter1], [source.filter2], ...
name/iname         : a regex; e.g. for all files with an .jpg extension : .*\.jpg$
                     (iname : case insensitive); "^c.5$" is different from "c.5" !
size              : a symbol plus an integer
                    e.g., either >999, either <999, either =999, either <=999 either >=999

                    use the following suffixes :

                       "kB" : 1000
                       "KB" : 1000
                       "MB" : 1000**2
                       "GB" : 1000**3
                       "TB" : 1000**4
                       "PB" : 1000**5
                       "EB" : 1000**6
                       "ZB" : 1000**7
                       "YB" : 1000**8
                       "KiB": 1024
                       "MiB": 1024**2
                       "GiB": 1024**3
                       "TiB": 1024**4
                       "PiB": 1024**5
                       "EiB": 1024**6
                       "ZiB": 1024**7
                       "YiB": 1024**8

[target]                  : parameters about target directory 
path                      : a string without any quotation mark.

name of the target files  : a string using keywords which will be replaced by their value
                            see the create_target_name() function.

            known keywords :

            %%h  : hashid (e.g. GwM5NKzoZ76oPAbWwX7od0pI66xZrOHI7TWIggx+xFk=)

            %%f  : source name (with the path), without the extension

            %%ff : source name (with the path), without the extension
                   [see below, no forbidden characters]

            %%p  : source path

            %%pp : source path
                   [see below, no forbidden characters]

            %%e  : source extension, without the dot character (e.g. jpg / no dot character !)

            %%ee : source extension, without the dot character (e.g. jpg / no dot character !)
                   [see below, no forbidden characters]

            %%s  : size (e.g. 123)

            %%dd : date (e.g. 2015_09_25_06_50)
                   [see below, no forbidden characters]

            %%i  : database index (e.g. 123)

            %%t  : timestamp of the file (integer, e.g. 1445584562)

            %%ht : timestamp of the file (hexadecimal, e.g. 5629DED0)

            n.b. : keywords with a reduplicated letter (%%pp, %%ff, ...) are builded against
                   a set of illegal characters, replaced by "_". 

(8.3) logfile

Can be filled with many informations (verbosity="high") or less informations (verbosity="low"). See in documentation:configuration file the explanations about the log verbosity.

The logfile is opened by logfile_opening. The program writes in it via msg() if _for_logfile is set to True. If the logfile becomes too large (see the "[log file]maximal size" value) the logfile if backuped and flushed.

(8.4) selection

SELECT is filled by fill_select(), a function called by action__select(). SELECT is a dictionary with hashid as keys and SELECTELEMENT as values.

Definition of SELECTELEMENT :
  SELECTELEMENT = namedtuple('SELECTELEMENT', ["complete_name",
                                               "path",
                                               "filename_no_extens",
                                               "extension",
                                               "size",
                                               "date"])

  !!! an "extension" stored in SELECTELEMENT does not start with a dot (".") !!!

FILTERS is a dictionary with a (int)filter_index as a key and a dict as values.
This dict may be empty or contain the following keys/values : 
  FILTERS["name"] = re.compile(...)
  FILTERS["size"] = a string like ">999", initial symbol in ('=', '<', '>', '<=', '>=')
  FILTERS["date"] = a string like '>2015-09-17 20:01', initial symbol in ('=', '<', '>', '<=', '>=')
FILTERS is filled by read_filters().

(8.5) database

In every target directory a database is created and filled. Its name is set by the global variable DATABASE_NAME. TARGET_DB is initialized by read_target_db(); hashid:(partialhashid, size, fullname)

o hashid varchar(44) PRIMARY KEY UNIQUE : hashid (of all the file)
o partialhashid varchar(44)             : hashid (of the beginning of the file)
o size integer                          : size
o name text UNIQUE                      : (target) name
o sourcename text                       : complete path + name + extension
o sourcedate integer                    : epoch time
o tagsstr text                          : a list of tags separated by the TAG_SEPARATOR
                                          symbol.

(8.6) trash directory

the deleted files are placed in a trashed directory placed inside the target directory. The trash name is defined in the configuration file.

(8.7) the functions

o  action__add()                        : add the source files to the target
                                          path.
o  action__addtag()                     : add one tag to the tags' string of the given files
o  action__cleandbrm()                  : remove from the database the missing files
o  action__downloadefaultcfg()          : download the default configuration file
o  action__findtag()                    : display the files tagged with the _tag parameter
                                          which is a simple string, not a regex.
o  action__infos()                      : display informations about the source
                                          and the target directory
o  action__new()                        : create a new target directory
o  action__rebase()                     : copy a target directory into a new one
o  action__rebase__files()              : --rebase : select the files to be copied.
o  action__rebase__write()              : --rebase : write the files into the new target direc.
o  action__reset()                      : --reset : remove the database and the files in the
                                          target directory
o  action__rmnotags()                   : Remove all files if they have no tags.
o  action__rmtags()                     : remove the entire tags' string of some files
o  action__select()                     : fill SELECT and SELECT_SIZE_IN_BYTES and
                                          display what's going on.
o  action__settagsstr()                 : modify the tags string in the target directory,
                                          overwriting ancient tags.
o  action__target_kill()                : delete a filename from the target directory
                                          and from the database
o  action__whatabout()                  : is a file/[are the files in a dir] already in the
                                          target directory ?
o  add_keywords_in_targetstr()          : replace some keywords by the value given as parameters
                                          in order to make strings used to create the target files
o  backup_logfile()                     : copy a logfile into a backuped file.
o  check_args()                         : check the arguments of the command line.
o  create_empty_db()                    : create an empty database.
o  create_subdirs_in_target_path()      : create the expected subdirectories in ARGS.targetpath .
o  create_target_name()                 : create the name of a file (a target file)
                                          from various information (filename, ...)
o  create_target_name_and_tags()        : create the name and the tags of a file (a target file)
                                          from various information (filename, ...)
o  create_target_tags()                 : create the tags of a file (a target file)
                                          from various information (filename, ...)
o  draw_table()                         : draw a table with some <_rows> and fill it with _data.
o  eval_filter_for_a_file()             : evaluate a file according to a filter
o  fill_select()                        : fill SELECT and SELECT_SIZE_IN_BYTES from
                                          the files stored in SOURCE_PATH.
o  fill_select__checks()                : final checks at the end of fill_select()
o  get_disk_free_space()                : return the available space on disk
o  get_database_fullname()              : return the full name of the db stored in ARGS.targetpath
o  get_filename_and_extension()         : return (filename_no_extension, extension)
o  get_logfile_fullname()               : return the logfile fullname.
o  goodbye()                            : display the goodbye message
o  hashfile64()                         : return the footprint of a file, encoded
                                          with the base 64.
o  logfile_opening()                    : open the log file
o  main()                               : main entry point
o  main__actions()                      : call the different actions required by the arguments
o  main__actions_tags()                 : call the different actions required by the arguments
o  main__warmup()                       : initializations
o  modify_the_tag_of_some_files()       : modify the tag(s) of some files
o  msg()                                : display a message on console, write the
                                          same message in the log file.

o  normpath()                           : return a human-readable, normalized version of a path
o  possible_paths_to_cfg()              : return a list of the (str)paths to the config file
o  read_command_line_arguments()        : read the command line arguments
o  read_parameters_from_cfgfile()       : read the configuration file
o  read_filters()                       : initialize FILTERS from the configuration file
o  read_target_db()                     : read the database stored in the target
                                          directory and initialize TARGET_DB.
o  remove_illegal_characters()          : replace some illegal characters by the
                                          underscore character.
o  shortstr()                           : shorten a string
o  show_infos_about_source_path()       : display informations about source path
o  show_infos_about_target_path()       : display informations about target path
o  size_as_str()                        : return a size in bytes as a human-readable
                                          string
o  tagsstr_repr()                       : return an improved representation of a tags string
o  is_ntfs_prefix_mandatory()           : return True if the _path is a path in a systemfile
                                          requiring the NTFS prefix for long filenames.
o  thefilehastobeadded__db()            : return True if the file isn't already known in the
                                          database
o  thefilehastobeadded__filters()       : return True if a file can be choosed and added to
                                        : the target directory
o  thefilehastobeadded__filt_date()     : a part of thefilehastobeadded__filters()
o  thefilehastobeadded__filt_name()     : a part of thefilehastobeadded__filters()
o  thefilehastobeadded__filt_size()     : a part of thefilehastobeadded__filters()
o  welcome()                            : display a welcome message on screen
o  welcome_in_logfile()                 : display a welcome message in the log file
o  where_is_the_configfile()            : return the config file name from ARGS.configfile or
                                          from the paths returned by possible_paths_to_cfg().