gien

gien is a tool for exporting the contents of any Github issue tracker to a local mbox file or Maildir directory as email, thereby mapping all issues and comments to email threads.

The structure of the exported email threads mimics Github's email notifications: There is a root message for every issue, and all comments on that issue reply to that root.

The tool can also download images which are referenced using the markdown syntax ![description](image-url) in issues or comments and attach to the correct email message.

We believe that this tool will prove itself helpful when leaving Github, or just for archiving bug trackers.

Email format

The exported emails have the content type multipart/alternative and consist of a MIME attachment in text/html format containing a markdown rendering of the Github message text and a text/plain MIME attachment containing the original body text.

Because I'm just using python-markdown for now, some Github-flavoured things could be rendered incorrectly.

Including some image attachments (-d, --download-images option), a typical message has the following structure:

  I     1 <no description>                          [multipa/alternativ, 7bit, 1.5K] 
  I     2 ├─><no description>                       [text/html, 7bit, us-ascii, 0.6K] 
  I     3 └─><no description>                       [text/plain, 7bit, us-ascii, 0.6K] 
  A     4 ddd1852a-71c5-11e5-9ec2-0da058cea7ff.png  [image/png, base64, 106K] 
  A     5 e1db00ce-71c5-11e5-8909-bd2991156351.png  [image/png, base64, 123K]

Issue tracker mirroring

When specifying an output mailbox that already exists and contains messages, the program will just append to that file or Maildir. You can use that behaviour to poll the bug tracker of any Github repository for updates for any real email client will help you to delete any dupes.

More intelligent features catering to the purpoes of mirroring issue trackers can be expected in the feature, including partial updates. There is no ETA, though.

Wiki mirroring

Repository wikis can be cloned just like the repos themselves and yield a collection of markdown files corressponding to the individual 'pages'.

Currently, when specifying the --wiki option, wiki pages are threaded just like the issues but with a [WIKI] tag.

Supporting the integration of assets like images as email attachments is a planned feature.

Multithreading

By specifying the --threads option, you can increase the number of parallel network requests made when archiving issue trackers. The thread count defaults to 4. This significantly speeds up the archiving process.

Since local processing of the data is very fast and the mirroring of the wiki relies on first executing a git clone command to fetch the remote data and then process all the data locally, gien does not implement parallel processing for wiki data (yet; but it's low priority because there's little to gain).

Authentication

Using HTTP basic authentication is not optional at the moment because of the API rate limit, which in pretty much all cases is too low for me.

Dependencies

• python3
• python-pygithub
• python-markdown
• python-progressbar
• python-pygit2
• python-requests

Installation

pip3 install gien

On Arch Linux, you can simply install the package gien from the AUR.

Synopsis

usage: gien [-h] [-I] [-W] [-d] [-i {all,open,closed}] [-l] [-o OUTPUT]
                   -p PASSWORD -r REPOSITORY [-t THREADS] -u USER
                   [--mailbox-type {mbox,maildir}]

Export Github issue trackers to local email storage

optional arguments:
  -h, --help            show this help message and exit
  -I, --archive-issues  Enable issue archiving.
  -W, --archive-wiki    Enable wiki archiving. Defaults to off.
  -d, --download-images
                        Enable the downloading of image attachments to issues
                        and comments.
  -i {all,open,closed}, --issues {all,open,closed}
                        Filter issues by state. Defaults to all.
  -l, --labels          If the issue has labels, add them to the email
                        Subject: header. If the issue has been marked as
                        closed, at a [CLOSED] label to the subject.
  -o OUTPUT, --output OUTPUT
                        Path to the output mbox file or Maildir. Will be
                        created if it doesn't exist.
  -p PASSWORD, --password PASSWORD
                        Github API authentication: password
  -r REPOSITORY, --repository REPOSITORY
                        Github repository name the issue tracker of which
                        shall be exported. Example: 2ion/gien
  -t THREADS, --threads THREADS
                        Number of worker threads. Defaults to 4.
  -u USER, --user USER  Github API authentication: user
  --mailbox-type {mbox,maildir}
                        Specify the mailbox type to use. Defaults to mbox.

Usage example

Specify -I|--archive-issues to archive issues and/or -W|--archive-wiki to archive wiki data. If neither of these options has been given, an empty or unmodified mailbox will be created.

./gien -I --user $user --password $password -r 2ion/forecast -i open -o out.mbox

License

gien -- export Github issue tracker & wiki contents to local email storage
Copyright (C) 2016 Jens John <jjohn@2ion.de>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.