{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Incorporating masks into calibrated science images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three ways of determining which pixels in a CCD image may need to be\n",
    "masked (this is in addition to whatever mask or bit fields the observatory at\n",
    "which you are taking images may provide).\n",
    "\n",
    "Two of them are the same for all of the science images:\n",
    "\n",
    "+ Hot pixels unlikely to be properly calibrated by subtracting dark current,\n",
    "discussed in [Identifying hot pixels](08-01-Identifying-hot-pixels.html).\n",
    "+ Bad pixels identified by `ccdproc.ccdmask` from flat field images, discussed\n",
    "in [Creating a mask with `ccdmask`](08-02-Creating-a-mask.html).\n",
    "\n",
    "The third, identifying cosmic rays, discussed in\n",
    "[Cosmic ray removal](08-03-Cosmic-ray-removal.html), will by its nature be different for each\n",
    "science image.\n",
    "\n",
    "The first two masks could be added to science images at the time the science\n",
    "images are calibrated, if desired. They are added to the science images here, as\n",
    "a separate step, because in many situations it is fine to omit masking entirely\n",
    "and there is no particular advantage to introducing it earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin, as usual, with a couple of imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from astropy import units as u\n",
    "from astropy.nddata import CCDData\n",
    "\n",
    "import ccdproc as ccdp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read masks that are the same for all of the science images\n",
    "\n",
    "[*Click here to comment on this section on GitHub (opens in new tab).*](https://github.com/mwcraig/ccd-reduction-and-photometry-guide/pull/197/files#diff-8ef287f1d9f06a3732bd30c19146ef93R60){:target=\"_blank\"}\n",
    "\n",
    "In previous notebooks we constructed a mask based on the dark current and a mask\n",
    "created by `ccdmask` from a flat image. Displaying the summary of the the\n",
    "information about the reduced images is a handy way to determine which files are\n",
    "the masks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>Table length=37</i>\n",
       "<table id=\"table140569328940984\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>file</th><th>imagetyp</th></tr></thead>\n",
       "<thead><tr><th>str31</th><th>str9</th></tr></thead>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-001.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-002.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-003.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-004.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-005.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-006.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-007.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-008.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-009.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>AutoFlat-PANoRot-r-Bin1-010.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>...</td><td>...</td></tr>\n",
       "<tr><td>Dark-S001-R001-C008-NoFilt.fit</td><td>DARK</td></tr>\n",
       "<tr><td>Dark-S001-R001-C009-NoFilt.fit</td><td>DARK</td></tr>\n",
       "<tr><td>Dark-S001-R001-C020-NoFilt.fit</td><td>DARK</td></tr>\n",
       "<tr><td>combined_bias.fit</td><td>BIAS</td></tr>\n",
       "<tr><td>combined_dark_90.000.fit</td><td>DARK</td></tr>\n",
       "<tr><td>combined_flat_filter_r.fit</td><td>FLAT</td></tr>\n",
       "<tr><td>kelt-16-b-S001-R001-C084-r.fit</td><td>LIGHT</td></tr>\n",
       "<tr><td>kelt-16-b-S001-R001-C125-r.fit</td><td>LIGHT</td></tr>\n",
       "<tr><td>mask_from_ccdmask.fits</td><td>flat mask</td></tr>\n",
       "<tr><td>mask_from_dark_current.fits</td><td>dark mask</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Table length=37>\n",
       "              file               imagetyp\n",
       "             str31                 str9  \n",
       "------------------------------- ---------\n",
       "AutoFlat-PANoRot-r-Bin1-001.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-002.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-003.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-004.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-005.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-006.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-007.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-008.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-009.fit      FLAT\n",
       "AutoFlat-PANoRot-r-Bin1-010.fit      FLAT\n",
       "                            ...       ...\n",
       " Dark-S001-R001-C008-NoFilt.fit      DARK\n",
       " Dark-S001-R001-C009-NoFilt.fit      DARK\n",
       " Dark-S001-R001-C020-NoFilt.fit      DARK\n",
       "              combined_bias.fit      BIAS\n",
       "       combined_dark_90.000.fit      DARK\n",
       "     combined_flat_filter_r.fit      FLAT\n",
       " kelt-16-b-S001-R001-C084-r.fit     LIGHT\n",
       " kelt-16-b-S001-R001-C125-r.fit     LIGHT\n",
       "         mask_from_ccdmask.fits flat mask\n",
       "    mask_from_dark_current.fits dark mask"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2_path = Path('example2-reduced')\n",
    "\n",
    "ifc = ccdp.ImageFileCollection(ex2_path)\n",
    "ifc.summary['file', 'imagetyp']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We read each of those in below, converting the mask to boolean after we read it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_ccdmask = CCDData.read(ex2_path / 'mask_from_ccdmask.fits', unit=u.dimensionless_unscaled)\n",
    "mask_ccdmask.data = mask_ccdmask.data.astype('bool')\n",
    "\n",
    "mask_hot_pix = CCDData.read(ex2_path / 'mask_from_dark_current.fits', unit=u.dimensionless_unscaled)\n",
    "mask_hot_pix.data = mask_hot_pix.data.astype('bool')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining the masks\n",
    "\n",
    "[*Click here to comment on this section on GitHub (opens in new tab).*](https://github.com/mwcraig/ccd-reduction-and-photometry-guide/pull/197/files#diff-8ef287f1d9f06a3732bd30c19146ef93R104){:target=\"_blank\"}\n",
    "\n",
    "We combine the masks using a logical \"OR\" since we want to mask out pixels that are\n",
    "bad for any reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_mask = mask_ccdmask.data | mask_hot_pix.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out we are masking roughly 0.056% of the pixels so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9418"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_mask.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect cosmic rays\n",
    "\n",
    "[*Click here to comment on this section on GitHub (opens in new tab).*](https://github.com/mwcraig/ccd-reduction-and-photometry-guide/pull/197/files#diff-8ef287f1d9f06a3732bd30c19146ef93R139){:target=\"_blank\"}\n",
    "\n",
    "Cosmic ray detection was discussed in detail in an\n",
    "[earlier section](08-03-Cosmic-ray-removal.html). Here we loop over all of the calibrated\n",
    "science images and:\n",
    "\n",
    "+ detect cosmic rays in them,\n",
    "+ combine the cosmic ray mask with the mask that applies to all images,\n",
    "+ set the mask of the image to the overall mask, and\n",
    "+ save the image, overwriting the calibrated science image without the mask.\n",
    "\n",
    "Since the cosmic ray detection takes a while, a status message is displayed\n",
    "before each image is processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system \n",
      "the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]\n",
      "WARNING:astropy:FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system \n",
      "the RADECSYS keyword is deprecated, use RADESYSa.\n",
      "WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATE-REF to '1858-11-17' from MJD-REF'. [astropy.wcs.wcs]\n",
      "WARNING:astropy:FITSFixedWarning: 'datfix' made the change 'Set DATE-REF to '1858-11-17' from MJD-REF'.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working on file kelt-16-b-S001-R001-C084-r.fit\n",
      "Starting 4 L.A.Cosmic iterations\n",
      "Iteration 1:\n",
      "199 cosmic pixels this iteration\n",
      "Iteration 2:\n",
      "108 cosmic pixels this iteration\n",
      "Iteration 3:\n",
      "112 cosmic pixels this iteration\n",
      "Iteration 4:\n",
      "106 cosmic pixels this iteration\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system \n",
      "the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]\n",
      "WARNING:astropy:FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system \n",
      "the RADECSYS keyword is deprecated, use RADESYSa.\n",
      "WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATE-REF to '1858-11-17' from MJD-REF'. [astropy.wcs.wcs]\n",
      "WARNING:astropy:FITSFixedWarning: 'datfix' made the change 'Set DATE-REF to '1858-11-17' from MJD-REF'.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working on file kelt-16-b-S001-R001-C125-r.fit\n",
      "Starting 4 L.A.Cosmic iterations\n",
      "Iteration 1:\n",
      "92 cosmic pixels this iteration\n",
      "Iteration 2:\n",
      "51 cosmic pixels this iteration\n",
      "Iteration 3:\n",
      "52 cosmic pixels this iteration\n",
      "Iteration 4:\n",
      "61 cosmic pixels this iteration\n"
     ]
    }
   ],
   "source": [
    "ifc.files_filtered()\n",
    "for ccd, file_name in ifc.ccds(imagetyp='light', return_fname=True):\n",
    "    print('Working on file {}'.format(file_name))\n",
    "    new_ccd = ccdp.cosmicray_lacosmic(ccd, readnoise=10, sigclip=8, verbose=True)\n",
    "    overall_mask = new_ccd.mask | combined_mask\n",
    "    # If there was already a mask, keep it.\n",
    "    if ccd.mask is not None:\n",
    "        ccd.mask = ccd.mask | overall_mask\n",
    "    else:\n",
    "        ccd.mask = overall_mask\n",
    "    # Files can be overwritten only with an explicit option\n",
    "    ccd.write(ifc.location / file_name, overwrite=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}