ReGet: retransmit less

Inspired by lack of functionality of wget this tool solves two annoying issues

• wget -N -O somefile doesn't work as you may expect
• there is no way to atomically replace target file with updated version

Okay, second problem easily solved like this

#!/bin/bash

TMPFILE=$(mktemp -q)

(wget -qO "$TMPFILE" "$1" && mv "$TMPFILE" "$2") || rm "$TMPFILE"

But what about first issue? It needs much more complicated shell-magic, introduces more troubles and requires additional HTTP request to target server.

Features

• Built on top of requests.
• Keep response metadata for next run.
• Send only needed headers from previous run.
• Atomic update of destination file.

Installation

$ pip install reget

Usage

As console tool:

$ reget http://example.org /tmp/example

will download server response to /tmp/example and create /tmp/example.json with headers needed for next run.

As python package

>>> import reget
>>> reget.get('http://example.org', '/tmp/example')
<Response [200]>
>>> reget.get('http://example.org', '/tmp/example')
<Response [304]>

Note that subsequent calls with the same url save_path arguments will respect 304 Not Modified.

Known issues

• Noticable startup time

TODO

• Option to use single directory for response metadata.
• Accept multiple <url> <save_path> from file/stdin.