PhotoPipe

Photometry Pipeline for RATIR and RIMAS

PhotoPipe is a pipeline for automated reduction, photometry and astrometry, written in Python (2.7), and designed to process imaging data from the following instruments:

• RATIR: the Reionization and Transients Infrared/Optical Project.
• RIMAS: the Rapid infrared IMAger Spectrometer.

This project is based on the previous versions of the pipeline by cenko and vickitoy.
Built at the NASA Goddard Space Flight Center, in collaboration with the University of Maryland.

Full Documentation

See the Wiki for full documentation, examples, operational details and other information.

Prerequisites

The pipeline can be installed with very little additional software. However, depending on the installation, different libraries and software may be required. Please see the Installation section below, and the dedicated wiki page for a full list of prerequisites.

Installation

The easiest way to get up and running with the pipeline is to download the ready-to-use virtual machine box and run it with Vagrant.

Alternatively, it can be installed on Linux or macOS either with pip, or with the provided installation scripts. (WARNING: compiling the dependencies can take more than 6 hours.)

1) Download a pre-configured Virtual Machine (PhotoPipe-VM)

Please refer to the virtual machine repository.

2) Install on your machine from PyPI or git

• Run sudo -H pip install photopipe to install the latest stable version from PyPI.

• Or, clone from git:

$ git clone git@github.com:maxperry/photometrypipeline.git
$ cd photometrypipeline
$ sudo python setup.py install

NOTE (macOS): If the installation fails with sudo: port: command not found make sure that MacPorts is installed and /opt/local/bin is in the $PATH (e.g. export PATH=/opt/local/bin:/opt/local/sbin:$PATH).

3) Manual Installation

If you to run the pipeline from the Python enviroment rather than using the photopipe command as described in the Usage section, please follow the step by step instructions in the wiki pages below to install all the dependencies manually.

• Instructions for macOS
• Instructions for Linux-Debian

Usage

The following steps can be reproduced using the test data downloadable here, either from your host machine or from the virtual machine.

1) Create a new directory with the following structure:

NOTE for PhotoPipe-VM: Create the imdata directory in the VM's shared data dir (should be ./photopipe-vm/data)

imdata  
│
└───bias
│   │   20160628T032914C0b.fits
│   │   20160628T032914C1b.fits
│   │   ...
│   
└───dark
│   │   20160628T040211C0d.fits
│   │   20160628T040211C1d.fits
│   │   ...   
│
└───flat
│   │   20160628T024207C0f.fits
│   │   20160628T024207C1f.fits
│   │   ...    
│
└───science
│   │   20160628T043940C0o.fits
│   │   20160628T043940C1o.fits
│   │   ...    
│
└───science_selected
│
└───reduced

If running on your Host Machine: Open the terminal and cd ./imdata/reduced

If running on PhotoPipe-VM: Launch the VM first (see instructions), and cd /vagrant_data/imdata/reduced

NOTE: It's important to perform the steps below from the reduced folder, otherwise make sure to move the master frames there after running mkmaster (it saves to the current dir!).

2) Preprocessing

1. Enter Python environment: $ python
2. Run the following script:

from photopipe.reduction import preproc

# Bias frames calibration 

bias_calib = preproc.choose_calib( 'ratir', 
                                   'bias', 
                                   workdir='/vagrant_data/imdata/bias/', 
                                   cams=[0,1], 
                                   auto=True, 
                                   amin=0.0, amax=1.0, 
                                   reject_sat=False, 
                                   save_select=True, 
                                   noplot=False )


# Dark frames calibration

dark_calib = preproc.choose_calib( 'ratir', 
                                   'dark', 
                                   workdir='/vagrant_data/imdata/dark/', 
                                   cams=[0,1], 
                                   auto=True, 
                                   amin=0.0, amax=1.0, 
                                   reject_sat=False, 
                                   save_select=True, 
                                   noplot=False )

# Flat frames calibration 

flat_calib = preproc.choose_calib( 'ratir', 
                                   'flat', 
                                   workdir='/vagrant_data/imdata/flat/', 
                                   cams=[0,1,2,3], 
                                   auto=True, 
                                   amin=0.2, amax=0.8, 
                                   reject_sat=False, 
                                   save_select=True, 
                                   noplot=False )
                                  
# WARNING: RATIR flats are often bad even when 
# the median value is in the acceptable range. 
# Auto mode is only recommended for bias frame 
# selection.     


# Select science frames
# (selected frames will be copied to target_dir)

science_dict = preproc.choose_science( 'ratir', 
                                       workdir='/vagrant_data/imdata/science, 
                                       targetdir='/vagrant_data/imdata/science_selected', 
                                       cams=[0,1,2,3], 
                                       auto=True, 
                                       save_select=True, 
                                       calibrate=False, 
                                       noplot=False ) 

# WARNING: When auto is True, all science frames
# are selected. Since the telescope occasionally
# has tracking issues, it is recommended to check
# all frames.  


# Make master frames
# (saves to the current dir)

preproc.mkmaster('ratir', bias_calib, 'bias')

preproc.mkmaster('ratir', dark_calib, 'dark')

preproc.mkmaster('ratir', flat_calib, 'flat')  

See wiki page for preproc functions reference.

WARNING:

1. Always use absolute paths terminated by a / when setting dir parameters.
2. Make sure to set the correct number of cameras where required (e.g. cams[0,1,2,3).
3. amin/amax set the min and max saturation values. Make sure to select the appropriate values for each type of frame, only frames with median values in this range will be selected.

3) Reduction

1. Start a new python environment
2. Execute the script below:

 from photopipe.reduction.auto.autoproc import autoproc

 autoproc( datadir='/vagrant_data/imdata/science_selected/', 
            imdir='/vagrant_data/imdata/reduced/', 
            redo=1 )

3. Reduced frames will be saved to imdir using coadd as prefix in the filename.

See wiki page for autoproc function reference.

4) Photometry

1. Move the coadd files to a new dir called photometry, and cd into it
2. Start a new python environment, and execute the script below:

 from photopipe.photometry.autoredux import autoredux
 autoredux()

NOTE: It's important to start python from the photometry dir containing the coadd files.

autoredux will run the following scripts:

1. photom.py: Samples and crops all files, creates a multicolor image used to find all sources, then finds the aperture photometry for each of those sources (resampled) using sextractor (with corrected zeropoint).
   • Output: *.am (absolute magnitude) files for w/ RA and DEC identified by sextractor
2. plotphotom.py: Plots photometry results to a HTML page.

Bugs and Feedback

For bugs, questions and discussions please use the Github Issues.

License

This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.