(1) Pimydoc

Pimydoc : [P]lease [i]nsert my doc[umentation]

A Python3/GPLv3/OSX-Linux-Windows/CLI project, using no additional modules than the ones installed with Python3.

Insert documentation in (text|code) files and update it.

This project is written in Python but the files to be documented may be written in any language : Python, C++, Java...

Example :

-> inside "pimydoc", documentation source file.
[pimydoc]
STARTSYMB_IN_DOC :¤ 
[ressource::001]
An interesting ressource.
[ressource::002]
Another interesting ressource.

 -> inside a (C++) source file :
void foo(arg):
    /*
       ressource::001
    */
    // ressource::002
    func("...")

-> inside a (Python) source file :
def foo(arg):
    """
       ressource::001
    """
    # ressource::002
    print("...")

The source files become :

void foo(arg):
    /*
       ressource::001
       ¤ An interesting ressource.

    */
    // ressource::002
    // ¤ An interesting ressource.
    func("...")

def foo(arg):
    """
       ressource::001
       ¤ An interesting ressource.
    """
    # ressource::002
    # ¤ another interesting ressource.
    print("...")

(2) purpose and file format

Pimydoc inserts in source files the text paragraphs stored in "pimydoc", the documentation source file. Moreover, the script updates the text paragraphs already present in the source files if the documentation source file has changed.

(2.1) documentation source file (docsrc) format

The file "pimydoc" is mandatory : it contains the documentation to be inserted in the source directory. After an optional header beginning with the string "[pimydoc]", the documentation itself is divided along doctitles written in brackets, like "[doctitle1]", followed by docline, a line followed by linefeed character(s).

### a simple example of a "documentation source file" :
[pimydoc]
REGEX_SOURCE_FILTER : .+py$

[doctitle1]
This line will be added as the first line of the "doctitle1"  
This line will be added as the first line of the "doctitle1"  

[doctitle2]
This line will be added as the first line of the "doctitle2"

This example will find all Python files (see REGEX_SOURCE_FILTER) and add after each "doctitle1" mention the two lines given, and add after each "doctitle2" the line given.

(2.1.1) a special case

If a docline is not followed by some linefeed character(s) (e.g. if this line is the last of the file), the script will automatically add a linefeed. The added linefeed is either the last linefeed detected by reading the file, either "\r\n" for Windows systems either "\n" in the other cases.

(2.1.2) comments

By default, comments are the lines beginning with the "###" string. Comments added at the end of a line like "some stuff ### mycomment" are not allowed. You can change the comment startstring : see the COMMENT_STARTSTRING in the documentation source file.

(2.1.3) header

The header is optional and always begin with the "[pimydoc]" string. The header's content is made of single lines following the "KEY:VALUE" format. The available keys are (the values are given as examples) :

REGEX_SOURCE_FILTER : .+py$
REGEX_FIND_DOCTITLE : ^\[(?P<doctitle>.+)\]$
STARTSYMB_IN_DOC :| | 
PROFILE_PYTHON_SPACENBR_FOR_A_TAB : 4
REMOVE_FINAL_SPACES_IN_NEW_DOCLINES : True

! Beware, the syntax "KEY = VALUE" is not supported.

• REGEX_SOURCE_FILTER : .+py$

Python regex describing which file in the source directory have to be read and -if required- modified.

Examples :

REGEX_SOURCE_FILTER : .+py$ ... for Python files

REGEX_SOURCE_FILTER : .+cpp$|.+h$ ... for C++ files

Beware, the format string "*.py" is not supported.

• REGEX_FIND_DOCTITLE : ^[(?P.+)]$

Python regex describing in the documentation source file, NOT in the source files the the way the doctiles appear. The group name (?P) is mandatory.

Examples :

REGEX_FIND_DOCTITLE : ^[(?P.+)]$

... if doctitles appear as "[mydoctitle]" in the documentation source file.

REGEX_FIND_DOCTITLE : ^<(?P.+)>$

... if doctitles appear as "" in the documentation source file.

• STARTSYMB_IN_DOC :| |

Characters appearing in source files juste before a doc line. The STARTSYMB_IN_DOC characters may be preceded by other characters, like spaces, "#", "//", and so on :

"""
   | | doctitle
"""

# | | doctitle

// | | doctitle

/*
   | | doctitle
*/

You may want to add a space before and after STARTSYMB_IN_DOC; there's a difference between "STARTSYMB_IN_DOC :| |" and "STARTSYMB_IN_DOC :| | ".

• PROFILE_PYTHON_SPACENBR_FOR_A_TAB : 4

For Python files : a tabulation in a docline may be replaced by spaces. The PROFILE_PYTHON_SPACENBR_FOR_A_TAB key sets the number of spaces replacing each tabulation. If you don't want any replacement, set this key to 0.

Examples :

PROFILE_PYTHON_SPACENBR_FOR_A_TAB : 4

(standard; see https://www.python.org/dev/peps/pep-0008/#tabs-or-spaces)

PROFILE_PYTHON_SPACENBR_FOR_A_TAB : 0

(no replacement)

• REMOVE_FINAL_SPACES_IN_NEW_DOCLINES : True

If set to True, the leading spaces at the end of docline added by Pimydoc are removed.

Examples :

REMOVE_FINAL_SPACES_IN_NEW_DOCLINES : True

REMOVE_FINAL_SPACES_IN_NEW_DOCLINES : False

• COMMENT_STARTSTRING : #~

In the documentation source file, characters at the beginning of a line which is read as a comment and skipped. The default value is "###".

(2.2) how to add doctitles in the files stored in the source directory

You juste have to add on a line the name of a doctitle (do not follow the format described in REGEX_FIND_DOCTITLE, e.g. "[ressource::001]", this regex being used only in the documentation source file). The documentation (the doclines) is added by inserting the doclines after the line containing the doctitle. Pimydoc will add before each docline the same characters appearing before the doctitle.

By example :

-> inside "pimydoc", documentation source file.
[ressource::001]
An interesting ressource.

-> inside a source file :
def foo(arg):
    """
       ressource::001
    """
    print("...")

The source file becomes :

def foo(arg):
    """
       ressource::001
       ¤ An interesting ressource.
    """
    print("...")

(2.3) profiles

According to the extension of the files read in the source directories, Pimydoc slightly changes the way it adds and updates the documentation. The known profiles are :

• "Python"

for files written in Python2 or Python3.

• see PROFILE_PYTHON_SPACENBR_FOR_A_TAB .

• "CPP" (i.e. C++)

nothing is changed compared to the default profile.

• "default"

default profile

(3) installation and tests

Don't forget : Pimydoc is Python3 project, not a Python2 project !

$ pip3 install pimydoc

or

$ wget https://raw.githubusercontent.com/suizokukan/pimydoc/master/pimydoc/pimydoc.py
Since pimydoc.py is a stand-alone file, you may place this file in the target directory.

How to run the tests :

$ python3 -m unittest tests/tests.py

or

$ nosetests

(4) workflow

$ pimydoc -h
... displays all known parameters.
$ pimydoc --version
... displays the version of the current script.

basic options

$ pimydoc
... searches a "pimydoc" file in the current directory, reads it and parses
    the current directory, searching files to be modified.

$ pimydoc --downloadpimydoc
... downloads the default documentation source file (named 'pimydoc') and exits.

$ pimydoc --sourcepath path/to/the/targetpath --docsrcfile name_of_the_docsrc_file
... gives to the script the name of the source path (=to be modified) and
    the name of the documentation source file (e.g. "pimydoc")

trick : how to modify the start string in a source directory ?

three steps :

$ pimydoc --remove     (or -r)
(modify the documentation source file : modify the STARTSYMB_IN_DOC key)
$ pimydoc

advanced options

removing the documentation added by Pimydoc

$ pimydoc --remove
$ pimydoc -r
... will remove the documentation added by Pimydoc.

security mode

$ pimydoc --securitymode
$ pimydoc -s
... will not delete the backup files created by Pimydoc before modifying the
    source files.

verbosity

$ pimydoc --verbose 0
... displays only error messages

$ pimydoc --verbose 1
$ pimydoc -vv
... both display only error messages and info messages

$ pimydoc --verbose 2
$ pimydoc -vvv
... both display all messages, including debug messages

(5) project's author and project's name

Xavier Faure (suizokukan / 94.23.197.37) : suizokukan @T orange D@T fr

Pimydoc : [P]lease [i]nsert my doc[umentation]

(6) arguments

usage: pimydoc.py [-h] [--sourcepath SOURCEPATH] [--docsrcfile DOCSRCFILE]
                  [--verbose {0,1,2}] [-vv] [-vvv] [--version] [--remove]
                  [--securitymode] [--downloadpimydoc]

optional arguments:
  -h, --help            show this help message and exit
  --sourcepath SOURCEPATH
                        source path of the code (default:
                        /home/suizokukan/projets/pimydoc/pimydoc)
  --docsrcfile DOCSRCFILE
                        source documentation file name (default:
                        /home/suizokukan/projets/pimydoc/pimydoc/pimydoc)
  --verbose {0,1,2}     degree of verbosity of the output. 0 : only the error
                        messages; 1 : all messages except debug messages; 2 :
                        all messages, including debug messages; See also the
                        -vv and -vvv options. (default: 1)
  -vv                   verbosity set to 1 (all messages except debug
                        messages). (default: False)
  -vvv                  verbosity set to 2 (all messages, including debug
                        messages). (default: False)
  --version, -v         show the version and exit
  --remove, -r          Remove every pimydoc line in all the files of the
                        source directory (default: False)
  --securitymode, -s    Security mode : backup files created by Pimdoc are not
                        deleted (default: False)
  --downloadpimydoc     download a default pimydoc file in the current
                        directory and exit (default: False)

(7) history / future versions

v 0.2.7(beta) (2016_10_08) fixed a message in rewrite_new_targetfile()

• fixed a message in rewrite_new_targetfile()

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.6(beta) (2016_09_18) warning for doctitles defined but never read

• a warning appears if a doctitle is defined
  in the documentation source file but never appears
  in the source directory

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.5(beta) (2016_09_17) COMMENT_STARTSTRING in the documentation source file

• comment startstring can now be defined in the documentation
  source file (see COMMENT_STARTSTRING in the documentation).
• added two debug messages in rewrite_new_targetfile().

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.4(beta) (2016_09_08) : fixed a minor bug in DocumentationSource.init()

• DocumentationSource.__init__() : if the documentation source file is
  a directory, an error is raised to prevent the directory to be opened.

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.3(beta) (2016_09_08) : fixed the default value of REGEX_FIND_DOCTITLE

• Settings.__init__() : fixed the default value of REGEX_FIND_DOCTITLE
  to "^\\[(?P<doctitle>.+)\\]$"

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.2(beta) (2016_09_08) : fixed a bug in DocumentationSource.newline()

• DocumentationSource.newline() handles correctly the case where a doctitle
  is ill-formed.
• improved messages : the line number is now marked by the word "line".    

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2.1(beta) (2016_09_08) : fixed an error in pimydoc

• added the accidently deleted "STARTSYMB_IN_DOC :| |" in pimydoc

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.2(beta) (2016_09_08) : messages display improvement

• added an info message at the end of the download_default_pimydoc() function.
• improved the final message of the pimydoc() function.
• improved a message in the rewrite_new_targetfile() function.

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.9(beta) (2016_09_08) : fixed issue #002, --downloadpimydoc option

• rewrite_new_targetfile : a new exception is caught (UnicodeDecodeError)
  when reading a binary file
• --downloadpimydoc option : the function download_default_pimydoc()
  downloads from the default "pimydoc" file.

• updated the documentation

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.8(beta) (2016_09_07) : fixed issue #001

• rewrite_new_targetfile() : fixed the way a linefeed is added to a docline
  without linefeed : either the last linefeed characters read in the file,
  either \r\n on Windows systems, either "\n".
• rewrite_new_targetfile() opens the files in binary mode to avoid that
  Windows OS modified the linefeed characters to be added at the end
  of doclines. Without this modification, the tests can't pass on Windows
  systems since the test files use the \n linefeed, NOT the \r\n one. 
• open() uses the option "encoding='utf-8" so that the script doesn't use
  the cp1252 encoding on Windows systems.
• (tests.py) the PATH_TO_CURRENT_TEST directory is removed at the
  beginning of each test function.
• added a rewrite_new_targetfile__line() function to size down the
  rewrite_new_targetfile() function.

• improved the documentation

• unittests : 2 tests (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.7(beta) (2016_08_29) : documentation improved/project in beta stage.

• improved the documentation

• fixed a bug in the tests : the filenames have to be sorted in order to
  hash a directory exactly the same way on different computers.
• Fixed a bug in rewrite_new_targetfile() : lines containing STARTSYMB_IN_DOC or
  STARTSYMB_IN_DOC.rstrip() have to be discarded.
  Fixed the corresponding tests.
• Fixed a bug in tests/tests.py : files whose name ends in "~" are discarded.

• removed from the pimydoc() function the "just_remove_pimydoc_lines" argument,
  since the value can be deduced from args.remove
• added a test : Tests.test_pimydoc_function__r() to test the --remove argument.  
• added the file : tests/__init__.py
• removed the unused constant STARTSYMB_IN_DOC__ESCAPE
• rewrite_new_targetfile() : added a debug message    
• added tests/test8/ which should have been added in 0.1.6
• improved error message in newline()
• modified rewrite_new_targetfile() : if PROFILE_PYTHON_SPACENBR_FOR_A_TAB
  is set to 0, no tabulation is replaced.
• DocumentationSource.newline() is now a method (not a subfunction
  of DocumentationSource.__init__() anymore)

• unittests : 1 test (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.6(alpha) (2016_08_28) : a docline can be inside a commentary beginning with "#" or "//"

• modified rewrite_new_targetfile() to handle doclines to be added after
  some symbols like "#" (Python) or "//" (C/C++). Added some tests to test
  this feature.

• simplified the documentation source file : no more "REGEX_FIND_PIMYDOC*"
  keys.
• improved the documentation

• unittests : 1 test (passed) 
• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.5(alpha) (2016_08_27) first unittests

• add the "tests/" directory

• fixed a bug in DocumentationSource.__init__() : .format() syntax has to
  be used anywhere except for logging messages.
• fixed a bug in remove_and_return_linefeed() : I forgot to return an empty
  linefeed if the last characters are not "\n", "\r" or "\r\n"
• fixed a bug in rewrite_new_targetfile() : if a docline doesn't end with
  a linefeed character, we have to add it manually in the target files.
• fixed a bug in rewrite_new_targetfile() : the file opened is now explicitly
  closed.
• added the README.rst file : the file is required by Pypi and should have
  been added since v 0.1.4.
• documentation improved

- unittests : 1 test (passed) 
- raw Pylint invocation : 10.0/10.0 for all scripts.
- version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

v 0.1.4(alpha) (2016_08_26) first version available on Pypi

• Pimydoc is now available on Pypi : https://pypi.python.org/pypi/Pimydoc

• improved the documentation

• raw Pylint invocation : 10.0/10.0 for all scripts.
• version packaged and sent to Pypi (https://pypi.python.org/pypi/Pimydoc)

older versions :

+ 0.1.3(alpha) :
    • pimydoc is callable from a Python script.
    • in rewrite_new_targetfile(), FileNotFoundError is catched to prevent
      an error by reading some special files like temp files.
    • in pimydoc_a_file(), added a call to shutil.copystat() to correctly
      set the permissions of the new target file.
    • in DocumentationSource.__init__() added a test about the existence
      of the file to be read.
    • improved messages displayed by the script

    • raw Pylint invocation : 10.0/10.0

+ 0.1.2(alpha)
    • improved documentation
    • added the LICENSE.txt file
    • raw Pylint invocation : 10.0/10.0

+ 0.1.1(alpha)
    • added -vv, -vvv options
    • --verbosity > --verbose
    • raw Pylint invocation : 10.0/10.0

+ 0.1(alpha)     
    • added --securitymode/-s option
    • improved documentation
    • raw Pylint invocation : 10.0/10.0

+ 0.0.9(alpha) : two profiles are available : "Python" and "C++";
                 removed two useless constants (SOURCEPATH, DOCSRCFILENAME);
                 the exit values are {0, -1, -2};
                 try/except added around calls to re.compile() and around .group();
                 improved documentation;

+ 0.0.8(alpha) : added the --remove/-r option.

+ 0.0.7(alpha) : redefined --verbosity={0,1,2}, default=1;
                 improved documentation

+ 0.0.6(alpha) : removed the useless constant VERBOSITY;
                 improved the documentation;
                 the regexes are now compiled

+ 0.0.5(alpha) : fixed a bug. If REMOVE_FINAL_SPACES_IN_NEW_DOCLINES is set to "True", some
                 empty lines wanted by the definition in pimydoc may be cut. E.g. :
                    ( space character being replaced by _ )
                    STARTSYMB_IN_DOC :|_|_ 
                    REMOVE_FINAL_SPACES_IN_NEW_DOCLINES : True

                    [doc001]
                    line1
                                    <--- this an empty line

                    ... the expected result is :

                    But the rewrite_new_targetfile() function needs to identify the added lines : searching
                    "|_|_" wouldn't be sufficient to find to second line, hence the need to search
                    with REGEX_FIND_PIMYDOCLINE_IN_CODE2, i.e. REGEX_FIND_PIMYDOCLINE_IN_CODE
                    without final spaces (\_ here, since the REGEX_* constants are re.escape'd).

+ 0.0.4(alpha) : added a function : remove_and_return_linefeed(); 
                 improved the way final spaces are handled at the end of the added lines;
                 added a new setting : REMOVE_FINAL_SPACES_IN_NEW_DOCLINES

+ 0.0.3(alpha) : improved documentation; pylinted the code; REGEX_SOURCE_FILTER; project's name set to "pimydoc";
                 improved the way the regexes are read in the documentation source file.

+ 0.0.2(alpha) : --verbosity levels reduced to 3; improved documentation

+ 0.0.1(alpha) : Settings class, no more call to eval()

+ 0.0.0(alpha) : proof of concept

(8) technical details

(8.1) exit values :

See the file named "pimydoc", section "exit codes"