twitch-scraper

A program that helps download live twitch streams.


Keywords
twitch, devops
Licenses
MIT/Apache-2.0

Documentation

$$$$$$$$\ $$\      $$\ $$$$$$\ $$$$$$$$\  $$$$$$\  $$\   $$\          
\__$$  __|$$ | $\  $$ |\_$$  _|\__$$  __|$$  __$$\ $$ |  $$ |         
   $$ |   $$ |$$$\ $$ |  $$ |     $$ |   $$ /  \__|$$ |  $$ |         
   $$ |   $$ $$ $$\$$ |  $$ |     $$ |   $$ |      $$$$$$$$ |         
   $$ |   $$$$  _$$$$ |  $$ |     $$ |   $$ |      $$  __$$ |         
   $$ |   $$$  / \$$$ |  $$ |     $$ |   $$ |  $$\ $$ |  $$ |         
   $$ |   $$  /   \$$ |$$$$$$\    $$ |   \$$$$$$  |$$ |  $$ |         
   \__|   \__/     \__|\______|   \__|    \______/ \__|  \__|         
 $$$$$$\   $$$$$$\  $$$$$$$\   $$$$$$\  $$$$$$$\  $$$$$$$$\ $$$$$$$\  
$$  __$$\ $$  __$$\ $$  __$$\ $$  __$$\ $$  __$$\ $$  _____|$$  __$$\ 
$$ /  \__|$$ /  \__|$$ |  $$ |$$ /  $$ |$$ |  $$ |$$ |      $$ |  $$ |
\$$$$$$\  $$ |      $$$$$$$  |$$$$$$$$ |$$$$$$$  |$$$$$\    $$$$$$$  |
 \____$$\ $$ |      $$  __$$< $$  __$$ |$$  ____/ $$  __|   $$  __$$< 
$$\   $$ |$$ |  $$\ $$ |  $$ |$$ |  $$ |$$ |      $$ |      $$ |  $$ |
\$$$$$$  |\$$$$$$  |$$ |  $$ |$$ |  $$ |$$ |      $$$$$$$$\ $$ |  $$ |
 \______/  \______/ \__|  \__|\__|  \__|\__|      \________|\__|  \__|
❯ cargo install twitch-scraper # Install via cargo
    Updating crates.io index
  Downloaded twitch-scraper v0.1.3
  ...

❯ curl -OL https://github.com/norcalli/twitch-scraper/releases/download/v0.1.3/twitch-scraper \
	&& chmod +x twitch-scraper # Or download a prebuilt static binary

❯ twitch-scraper \
	-q \
	-c $TWITCH_CLIENT_ID \
	-d /videos/twitch/ \
	-o "%(channel)s/%(title)s-%(id)s.%(ext)s" \
	-x $PWD/stream-went-live.sh \
	ashkankiani naysayer88 demolition_d studio_trigger
	# Example session
[2019-09-12T16:26:47Z INFO  twitch_scraper] Watching ["demolition_d", "naysayer88", "ashkankiani", "studio_trigger"]
[2019-09-12T16:34:45Z INFO  twitch_scraper] Downloading stream 35633084336 from ashkankiani
[2019-09-12T16:36:21Z INFO  twitch_scraper] Finished downloading stream 35633084336
[2019-09-13T10:01:47Z INFO  twitch_scraper] Downloading stream 35628821840 from naysayer88
ERROR: ffmpeg exited with code 255
[2019-09-13T10:02:19Z ERROR twitch_scraper] Downloading of stream 35628821840 from naysayer88 failed with status ExitStatus(ExitStatus(256))
[2019-09-13T10:02:19Z INFO  twitch_scraper] Downloading stream 35628821840 from naysayer88
[2019-09-13T10:02:49Z ERROR twitch_scraper] Downloading of stream 35628821840 from naysayer88 failed with status ExitStatus(ExitStatus(15))
[2019-09-13T10:02:50Z INFO  twitch_scraper] Downloading stream 35628821840 from naysayer88
[2019-09-13T12:00:47Z INFO  twitch_scraper] Finished downloading stream 35628821840 from naysayer88

❯ cat example_scripts/stream-went-live.sh
#!/bin/sh
exec notify-send TWITCH "$TWITCH_CHANNEL_NAME went live at $TWITCH_STREAM_CREATED_AT!
Downloading $TWITCH_STREAM_ID"


❯ twitch-scraper --help
twitch-scraper 0.1.3
Program to poll twitch via its API and download streams from channels as they come live.

Fault tolerance:
 - It will retry requests to Twitch API with exponential jitter backoff up to 5s between retries.
 - It will handle API limit rating.
 - It will try to restart the download in case youtube-dl fails prematurely.

Sending SIGINT will kill all running downloads.
 Sending SIGTERM will keep the downloads running.

Use RUST_LOG to set logging level.
 e.g. export RUST_LOG='debug' or export RUST_LOG='twitch_scraper=info'

USAGE:
    twitch-scraper [FLAGS] [OPTIONS] --client-id <client-id> --directory <directory> [--] [channel-names]...

FLAGS:
    -h, --help       
            Prints help information

    -q, --quiet      
            Quiet output for youtube-dl.
            
            Shortcut for --additional_args=-q.
    -V, --version    
            Prints version information


OPTIONS:
    -a, --additional-args <additional-args>...     
            Extra args to pass to youtube-dl.
            
            Current arguments are: --write-info-json --hls-use-mpegts --no-part --netrc
    -c, --client-id <client-id>                    
            Twitch client id

        --delay-max <delay-max>                    
            Maximum milliseconds to wait before polling again [default: 3000]

        --delay-min <delay-min>                    
            Minimum milliseconds to wait before polling again [default: 100]

    -d, --directory <directory>                    
            Directory to save videos.

    -o, --filename-template <filename-template>    
            Directory to save videos.
            
            See `man youtube-dl` under OUTPUT TEMPLATE for variables to use.
            
            Useful variables:
             - %(uploader)s: channel name
             - %(uploader_id)s: channel name (lowercase)
             - %(description)s: channel status/title
             - %(timestamp)s
             - %(title)s: for a live stream, looks like 'ashkankiani 2019-09-06 14_19'
            
            Note that in the case of a temporary download failure, when youtube-dl is restarted the filename may change,
            meaning a separate file is created. This occurs with %(title)s, which uses the time of downloading as part
            of the filename.
            
            MPEGTS files can be concatenated together without any complex processing (e.g. with `cat`), so these streams
            can be trivially recomposed.
            
            I personally use "%(uploader)s/%(uploader)s-%(id)s-%(description)s.%(ext)s"
            
             [default: %(title)s-%(id)s.%(ext)s]
    -x, --script <script>                          
            A script to execute when the stream comes live.
            
            These environment variables will be set:
             - TWITCH_CHANNEL_NAME
             - TWITCH_CHANNEL_ID
             - TWITCH_CHANNEL_STATUS
             - TWITCH_STREAM_ID
             - TWITCH_STREAM_CREATED_AT
             - YOUTUBE_DL_PID: The pid of the child process launched with youtube-dl

ARGS:
    <channel-names>...    
            List of channel names to poll.

Planned features

  • Subcommand/companion tool for scraping chat logs.
  • Taking a file as the input list for channels to watch so that the list can be amended without restarting the tool. This will allow it to become a permanent daemon. Planned to be an optional CLI flag of -f for --file which will constitute a file to watch for channel names.

Thanks

Sweet ascii art http://www.patorjk.com/software/taag/