Spextool -------- ------------------------------------------------------------------------------ Quick and Dirty Manual for Running SPEXTOOL - a package for reducing SpeX data ------------------------------------------------------------------------------ (WDV and MCC, 13 Feb 2002 - Spextool Version 2.2) History: 13 Oct 2000 - v1.0 - original release 18 Oct 2000 - v1.1 - bug fixes, added sky base 20 Oct 2000 - v1.2 - added xcombspec, modified PS extraction; added description of xvspec 24 Oct 2000 - v1.3 - bugs fixed in xcombspec 18 Dec 2000 - v1.4 - Fixed bug where the spectra returned using the extended source extraction base WERE NOT in DN/s. The values are now returned as DN/s. - Added Combine base (see below). - Fixed a bug which made the spectra appear to have some sort of fringing when small apertures, < 1 arcsecond, were used. - Modified the way the arcs, flats and skys are stored. As a result, arcs, flats and skys made using earlier versions of Spextool have to modified as follows: At an IDL prompt, IDL> i = readfits('filename',hdr) IDL> writefits, 'filename',rotate(i,7), hdr The flats, arcs, and skys can now be used with version 1.4. 09 Mar 2001 - v1.5 - Improved wavelength calibration for the LXD1.9 and LXD2.3. See step 8. - Improved flat fielding (To be described later). - Prism mode can now be fully reduced. - Modified Path input. - XQTV can now perform image arithmetic (under file menu button). - xcombspec has been modified to combine apertures if requested. This is useful when point source data is combined before being extracted. In this case, Spextool writes both apertures to the same file. - Wavelength calibration now requires a user input which gives the pixel offset of an arc spectrum relative to a standard arc spectrum. See step 18. 17 Aug 2001 - v2.0 - Major revision. - Errors are now propagated through the extraction process. An error vector is now produced and appears as a third dimension in the FITS array or another column in the text array. - All output names (i.e. arcs, flats, etc) should NOT have the suffix .fits added now. Spextool will add it automatically. When loading images in using full names, the entire name, including the .fits, should be entered. - Xplotprofiles now shows the raw data and and has cursor tracking. - Xvspec can show the error spectra and compute the signal-to-noise in each spectrum. - Xvspec can now smooth the spectra (convolve with a Gaussian of FWHM = npixels) and propagate the error. - To launch a zoom window (xzspec) in xvspec, click on the order you want to examine. Xzspec is fully resizable. To learn the cursor commands, type 'h' in the plot window. - A new widget called xmkcals allows the user to construct the flats and arcs for the entire night. - The xcombspec widget has been completely redesigned to allow more user input. 2001-10-09 - v2.1 - Fixed bugs in xcombspec - Added xsmoothspec widget to smooth spectra by a Gaussian with FWHM equal to the slit width in pixels - Added a fits2text program to convert a SpeX FITS file to a text file - Improved bad pixel replacement - Added a data set to allow the user to practice using Spextool 2002-02-13 - v2.2 - Internal modifications to the structure of spextool. Not backwards compatible but extraction is the same. - Fixed wavelength calibration for the LongXD modes. - xzoomplot now has more zoom options. Type 'h' to see the cursor commands. - Xplotprofiles now has slider bars to allow better viewing of the super-profiles. The following is a brief description of how to run the Spextool package written by Mike Cushing and Bill Vacca for the reduction of SpeX data. A sample data set can also be downloaded so that users may practice using Spextool before their observing run. A more formal and complete manual will be available in the near future. Users should note that Spextool is still under construction. Some of the limitations of the code - which we plan to fix in future versions - are listed at the bottom of these instructions. We urge potential users to read these FIRST before attempting to use Spextool. Bugs in the code should be reported to cushing@ifa.hawaii.edu. Because of flexure and the variable nature of atmospheric absorption, it is recommended that each set of images of a given object be reduced with its own flats, arcs, and telluric standards. I. Setup 1. Spextool *REQUIRES* at least IDL v5.3 as well as the IDL Astronomy User's Library. The latter can be downloaded from the following URL: http://idlastro.gsfc.nasa.gov/homepage.html 2. Download the Spextool gzipped tar file, gunzip it, and unpack it with tar xvf Spextool.tar. This will create a directory called Spextool, which is the package directory, and a number of sub-directories. If the user wants to practice using Spextool, download the data SpeXdata.tar.gz as well. Gunzip it and unpack it with tar xvf SpeXdata.tar. This data should be placed in the raw directory (see step 5). 3. Modify your path to include the package directory. For example, on a UNIX machine, where the IDL path is defined in a .cshrc file, edit the .cshrc file to include the line setenv IDL_PATH +/packagedirectory:$IDL_PATH where /packagedirectory is the path to the package directory (e.g., /scr0/irtf/Spextool). If there is no IDL_PATH defined in the .cshrc file, then include the line setenv IDL_PATH +/packagedirectory 4. Start IDL and bring up Spextool by typing: xspextool This will bring up the main Spextool window, called the Main Base. In general, a "Base" is a panel or a set of panels containing fields and buttons that the user chooses to control procedures followed by the reduction programs. Please make certain that you are running the correct version of Spextool by looking at the title bar of the widget. 5. Click on the Paths button in the middle of the Main Base. This will bring up the Paths panel. Spextool will automatically load the paths last entered by the user. If the paths are correct, go to step 6. Spextool assumes (but does not require) that the raw data, processed data needed for calibrations (e.g., flats and arcs), and completely reduced data will be kept in separate directories. While this need not be the case, it makes the reductions easier to keep track of (for humans). If you wish to adopt this structure, create 3 subdirectories and put all of your raw data in one of them. Typically these directories will be called raw (for raw data), cal (for calibration files) and proc (for processed files). Type or (click) the paths for these three subdirectories in the fields listed in the Paths panel. Spextool will remember these directories, so that next time you start Spextool these fields will be loaded automatically. Finally, click the Load Paths button to load the paths into memory. 6. In the File Read panel, above the Paths panel, choose the type of naming convention used for your data. For example, if all your data files are called spec followed by sequential numbers for each file, choose Index and type in the prefix (here, spec) into the Input Prefix field. If you have different prefixes for different files, you will have to change this field each time the prefix changes. If your files all have different names, you will have to choose the Filename mode. Only a single file can be processed in this mode and the user must enter the full name of the input file and the output file. II. Flats, Arcs, and Skys 7. Make a flat. Click on the Flat button in the middle of the Main Base. This brings up the Flat Base. Type in the numbers of the files you will combine to make a flat (e..g., 386-390). (Remember to change the File Prefix in the File Read panel if the prefix is not 'flat'. Choose the statistic used for combining the flats. A typical choice might be median. Give the full name of the output flat in the Full Flat Name field (e.g., flat386-390, for a flat made from files 386-390). Click the Construct Flat button. Spextool will then construct a flat from the input files. This takes some time and requires a number of steps, including scaling and combining the images, finding the edges of the orders, fitting a 2-D surface to each order, fixing bad pixels, and normalizing the flat by the 2-D surface. Spextool will not allow the user to do anything while it processes the flat, but it will show you where it has found the individual orders in the images, and it display the final normalized flat when it is finished. ***NOTE - The scattered light is not removed in this process.*** 8. Make an arc. Click on the Arc button in the middle of the Main Base. This brings up the Arc Base. If the arcs are all "on" frames (ShortXD), choose A in the reduction mode. If the arcs have "on" and "off" frames (LongXD), then choose A-B in the reduction mode. Type in the numbers of the files you will combine to make an arc (e.g., 391-393 or 391,392,393; either notation is allowed). If you are NOT reducing LXD data, then skip to the next paragraph. Spextool v1.5 and later now requires a sky frame as well as the arcs for LXD wavelength calibration. Spextool uses the night sky lines in the lower orders of the LXD modes since they have either no argon lines or very weak argon lines. If your targets are extended sources, type (or click) the full name of the sky image in the Full Sky Name field in the second column (e.g., spc0467.b.fits). If your targets are point sources, type (or click) the full name of an A and B pair in the Full Sky Name field (e.g., spc0512.a.fits,spec0513.b.fits). (Two files can be selected at once if you click on the Full Sky Name button). This AB pair should be chosen to be near in time and position to the arc images. Spextool with then create a sky frame by sky = (A+B) - abs(A-B). Give the full name of the flat field image made above (needed for finding the edges of the orders). Choose the statistic used for combining the arc files. A typical choice might be mean. Choose whether the combination statistic should be weighted, by selecting Yes or No in the File Read Panel. The default is to weight the statistic. Give the full name of the output arc in the Full Arc Name field (e.g., arc391-393, for an arc made from files 391-393). Click the Construct Arc button. Spextool will then construct a final arc image from the input files, flat field the combination, fix bad pixels in the result, and write the final image to the cal path. 9. To construct the flats and arcs for the entire night, type xmkcals at the IDL prompt. Load the paths as in step 5. Select whether to create the arc, the combination statistic, and whether to normalize the flats. In the table, type the numbers of the flats and arcs for each cal set (386-393). Click Construct Calibration Images to make the flats and arcs. Xmkcals will inspect each cal set to determine if the observing mode is ShortXD or LongXD. If the cal is LongXD, the user will be prompted for the sky frame as in step 8. Xmkcals then constructs the flats and arcs. 10. Make a Super Sky frame. If the user has acquired data by moving the objects between two positions on the slit, this step is unnecessary. Similarly, if the user does not wish to subtract a sky frame, this step can be skipped. However, if the user acquired data by nodding to sky between object frames, then a "Super sky" frame should be created and subtracted from the object frames. Click on the Sky button in the middle of the Main Base. This will bring up the Sky base. Type in the numbers of the files you will use to make a sky frame (e.g., 21,23,25,27,29). Remember to change the File Prefix in the File Read panel if the prefix is different from that used for your object images.) Give the full name of the flat field image made above (needed for finding the edges of the orders). Choose the statistics used for combining the sky frames. A typical choices might be median. Give the full name of the output sky frame in the Full Sky Name field (e.g., sky21-29, for a sky made from files listed above). Click the Construct Super Sky button. Spextool will then scale and combine the sky frames, fix bad pixels, and write out the Super Sky frame. III. Extraction of Bright Objects 11. Prepare for extraction of your spectra by clicking either the Point Source or Extended Source button in the middle of the Main Base. This will bring up a set of panels (the Source Base) for the chosen source type. For the illustrative purposes of this manual, we will assume the observed objects are Point Sources. 12. In the File Read panel, type in the desired prefix for the output files (if the input files are in Index mode) or the full output file name (if the files are in Filename mode). Choose an output format. If you wish to use some of the other IDL packages we are developing for SpeX data, choose either FITS and/or Text. If you wish to use IRAF to process the output files further, choose Text. (The FITS files produced by Spextool have a slightly different format than that expected by IRAF. IRAF can in fact read and display these files (using splot, for example), but they won't appear to be wavelength calibrated. This is because the wavelength information is stored in "image band" 1 - in IRAF-speak - and the flux data are stored in bands 2-7 for spectral orders 3 (K) through 8 (I). See step 15 for the naming convention and wavelength ranges of the orders.) The FITS file contains a 3-D array of data that can be accessed as follows. If the user desires aperture Y in order X then lambda = array[*,0,( X - min(order number))*naps + Y], flux = array[*,1,( X - min(order number))*naps + Y], error = array[*,2,( X - min(order number))*naps + Y] The text files contain a FITS header followed by 3*NORDERS*NAPS columns, where NORDERS is the number of orders and NAPS is the number of apertures traced and extracted. The columns come in triplets and contain the wavelength in microns, the flux, and the error in the spectrum in the various extracted orders, arranged from lowest to highest order. For example, the first three columns might contain the wavelengths, fluxes and errors for the K band. The next three contain the data for the H band, etc. The text files can be converted to FITS files usable with IRAF by extracting the wavelength and flux columns associated with a particular order/aperture and then using the IRAF routine rspectext with the parameter dtype=nonlinear. 13. In the Reduction Mode panel, to the right of the File Read panel, select the type of observing mode used to acquire the data. If you do not wish to do any sky subtraction at all, choose A. If you made a "Super Sky", choose A-Sky, and give the name of the Super Sky frame. If the data were acquired in the standard IR method of moving the object between two positions on the slit (the pair method), choose A-B. Assuming the method is A-B, type in the numbers of the first two object frames (e.g, 376,377 or 376-377; either notation is allowed), type or click in the name of the flat field produced above (e.g., flat386-390.fits), and type or click in the full name of the arc image (e.g., arc391-393.fits). If you do not want to divide by the flat field, then turn off the flat fielding in the Other base. The flat field image is still required because its header contains information necessary for extraction process. If you do not want to wavelength calibrate the data, leave the Arc Field empty. Then click the Load Image button. Spextool will then subtract the pair of images and flatten the result. The resulting 2-D image will be displayed. 14. In the Find Aperture Positions panel, choose Manual and click on the Make Super Profiles button. This will construct spatial profiles of the spectra, summed along the wavelength direction in the various orders, and display them in a separate window, which can be resized to see the results. The white dots are the raw data and the red line is the smoothed version of the raw data. The Point Source version of this routine automatically attempts to identify and ignore columns of data with little or no flux from the object. Sometimes this can result in a poor spatial profile in some of the orders. If these orders are required, lower the Prune Thresh in the second column of the Other Base. A threshold of 1.0 will allow all the columns to be used in the super-profile, similar to the Extended Source Base. Choose the number of apertures to be found automatically in these profiles and click the Automatic button. For typical spectra obtained in pair, or A-B, mode, there will be two apertures, corresponding to the object in the A frame and its negative counterpart in the B frame. Click on the Find Positions button and Spextool will locate the centers of these profiles. If the user has a faint point source for which only a noisy super profile can be constructed, the Guess or Fix Positions option can be chosen. The positions in ARCSECONDS along the slit for the centers of the extraction apertures should be given in the Aperture Positions field. If Guess is selected, Spextool will use these positions as a guess in located the center of the profiles. If Fix is selected, the location of the apertures will be taken as given. If the user has extended objects, the positions in ARCSECONDS along the slit of each aperture to be extracted can be entered in the Positions field. Click on the Find Positions button and Spextool will locate the the specified positions. 15. Choose the spectral orders to be extracted. Normally the observer will want all spectral orders. However, if an order contains so little flux that the spectrum cannot be traced, the user should un-click that order. For the short cross-dispersed mode (SXD: 0.8-2.5 microns), the orders and the corresponding wavelength ranges and window labels are as follows: Order Wavelengths Window ----- ----------- ------ 8 0.83 - 0.93 "I" 7 0.83 - 1.06 "z" 6 1.01 - 1.24 "J1" 5 1.15 - 1.48 "J2" 4 1.43 - 1.86 "H" 3 1.93 - 2.45 "K" The wavelength range covered by order 8 completely overlaps that covered by order 7, but to avoid confusion we have adopted this naming convention. Note that the beginning and ending wavelengths of each window/order are approximate. For the long cross-dispersed mode (LXD: 2.0-5.5 microns), the orders, wavelength ranges, and window labels are as follows: Order Wavelengths Window ----- ----------- ------ 10 1.90 - 2.10 "K1" 9 2.10 - 2.35 "K2" 8 2.23 - 2.74 "K3" 7 2.55 - 3.14 "Kx" 6 2.98 - 3.67 "L1" 5 3.58 - 4.39 "L2" 4 4.47 - 5.47 "M" Note that not all of these orders can be obtained in a single grating setting in the LXD mode. In the "LXD1.9" setting, orders 5-10 can be recorded. In the "LXD2.3" setting, orders 4-8 can be recorded. 16. Now click on the Trace Objects button in the Trace Objects panel. Spextool will trace the centers of the apertures found above. If the extraction apertures haven't been fixed by the user, Spextool will fit a Gaussian to the spatial profile at every column, using the aperture centers determined above as the initial guess. The centers of the Gaussians are then fit with a polynomial to derive the trace position as a function of column. If the user has fixed the extraction aperture (e.g., for an extended source or a faint point source), Spextool will simply plot the specified positions on the image and use these as the traces. 17. The extraction apertures can now be defined in the Define Apertures panel. Choose the radius of the extraction aperture in ARCSECONDS, using the profiles found in step 14. Make sure the background subtraction button is turned on (BG Sub ON). Give the start radius in arcseconds for the background window, and the width of this background window in arcseconds. Specify the order of the polynomial used to fit the background in these windows and across the object. Order = 1 is a straight line. Now click on the Define Apertures button. Spextool will show the apertures used for both the background and the object on the Profiles plot. It will also show the object apertures on the background subtracted image. If the user has extended objects, the radii of each aperture defined in step 14 must be entered in the Ap Radius field (e.g., 1.5,1.5 for 2 apertures). The radii values should be separated by commas. The positions, in arcseconds along the slit, of the background windows must also be specified. An example might be 0-2, 13-15. Note that differential atmospheric refraction will cause peaks to be located at slightly different positions along the slit. This is automatically corrected for in the Point Source mode, because Spextool finds the peaks in each order independently. However, it is not corrected for in the Extended Source mode. The user should keep this in mind and make sure the extraction aperture widths are large enough to account for the variation in position. 18. The spectra will be extracted when the user clicks on the Extract button. Spextool will fix bad pixels in the pair-subtracted, flat-fielded image. It will then fit the background in each background window in each column and in each order, subtract it from each pixel in the aperture, and then sum the flux in the aperture. Once it has finished extracting the object spectra, it will extract the arc spectra at the same positions along the slit as occupied by the object. A widget called xgetoffset will then appear which displays a standard arc spectrum (stored on disk) and an extracted arc spectrum. The user must identify the same line in each spectrum by clicking on the lines with the left-most mouse button. The gaussian fit to the line will be displayed in the window next to each spectrum. (The purpose of this step is to determine the offset between the solution used to identify lines, determined from standard arc spectrum, and the extracted arc spectrum.) When the user is satisfied, click Accept and Spextool will then wavelength calibrate the spectra. 19. View the spectra. The wavelength calibrated spectra will then be displayed in a widget called xvspec (which can also be called separately from xspextool by typing xvspec at the IDL command line). Once the spectra are loaded, the ranges of the individual orders can be changed by clicking on the Range button and then selecting an order and a Min and Max value for the y axis of that order. Header information can be viewed by clicking on the Header button. The color used to plot the spectra can also be selected by clicking on the Color button. A postscript file can also be written out by clicking on the Write PS button and selecting various options. In addition, the error spectrum and single-to-noise can be displayed. To view a single spectrum, click on the order with the left-most mouse button. A widget called xzspec will appear with the spectrum plotted. The user can change the x and y range by hand using the fields at the bottom of the widget or interactively using the cursor. To learn about cursor commands type 'h' in the plot window. 20. Once the first pair of object spectra have been extracted and the user is happy with the extraction parameters, the full list of file numbers for the rest of the objects can be inserted in the Reduction Mode panel (e.g., 376-381). Then all of these images will be reduced automatically by clicking on the Do All Steps button in the panel to the right of the Reduction Mode panel. Note, the user will not have to determine a pixel offset because Spextool will use the pixel offset determined in step 18. IV. Extraction of Faint Objects 21. Extracting the spectra of faint objects can be difficult because Spextool may have problems accurately tracing the objects. In this case, the user should combine the 2-D images before extraction to increase the signal-to-noise of the faint orders. Move to the Combine Base. If the data were acquired by moving the object between two positions on the slit (the pair method), choose A-B; if not, choose A. Type the image numbers into the Image field, 376-381. The sky level may change between images. If the objects do not fill the slit completely in the spatial dimension, choose Yes for BG subtraction and give the full name of the flat field image made in step 7. Spextool will subtract the median value of the pixels in the slit from each column (in every order in each image) before combining the images. Choose the statistic used for combining the images. A typical choice might be median. Note that a weighted mean may not be an appropriate statistic because the assumptions underlying the weighted mean as the best estimate of the mean may not be satisfied due to guiding error and slit losses. This effect is most pronounced in short integration exposures. If Mean is chosen then the error of the final combined image is computed as sigma_mu = sigma/sqrt(N). If a median is chosen, then the error on the median is computed as sigma_med = MAD/sqrt(N) where the MAD (Median Absolute Deviation)=1.482*median(|data-median|). The constant 1.482 is set so the MAD = sigma for a Gaussian distribution. Finally, give the full name of the output file in the Output Name field (e.g., spc376-381.fits). Spextool will then combine the images and write the result to the proc directory. 22. The combined image may now be extracted by defining the Data Path in the Path Base to be the proc directory. The file read mode should then be set to Filename and the reduction mode is set to A. The user can then proceed as normal. Spextool will write both beams (apertures) to a single file. They can be combined in xcombspec (see step 23) by choosing the Combine Apertures options. V. Combining the Spectra 23. The individual reduced spectra (FITS files) may be combined with a separate IDL program called XCOMBSPEC. Bring up the XCOMBSPEC GUI within IDL by typing xcombspec. Choose the File Read Mode. If the files for any given object all have the same prefix, choose Index mode. Type the file prefix into the Input Prefix field and give the file numbers in the Spectra Files field (e.g., 376-385). Alternatively, the filename readmode can be selected and all the files can be selected at once by clicking on the Spectra Files button. Choose whether or not to combine the individual apertures. For example, if a faint point source is being reduced, then the images may have been combined before extraction. In this case, the output spectrum will have two apertures that need to be combined together. Click Load Spectra and the spectra will appear in the plot window to the right. To scale the spectra to the spectrum with the highest flux level, click on the button Scale to Max. This is recommended since the best estimate of the flux level of the star is the spectrum with the highest flux level. At times, a single spectrum within an order may be poor or perhaps an entire spectrum could be bad. To skip over this spectrum, click on the order with the bad spectrum. Another widget, xselectspec, will appear. Simply shut off the bad spectrum and click Accept. To eliminate the entire frame from the combination, click All Orders. Choose the statistic to be used for the combination process and whether it should be weighted. A typical choice would be Robust Mean or Median. If the statistic is either Robust Mean or Mean, choose whether the spectra should be weighted or not. Then select the output format and the name of the output file. Click on the Combine Spectra button. xcombspec will then combine the individual spectra using the chosen statistic, propagate the errors, display the result, and write it to the output file. If the spectra have units of pixels and flux, the spectra will simply be combined at each pixel. If the units are wavelength and flux, however, xcombspec chooses the first spectrum in the list as the template, and the remaining spectra are interpolated onto the same wavelength scale before being combined. VI. Misc. 24. The output files from Spextool v1.x will not work with the new xvspec and other routines because they do not contain and error spectrum. To convert v1.5 FITS files to v2.0 files, use the routine v1tov2.pro. At the IDL prompt type, v1tov2,inputfile,outputfile where inputfile is the string file name of the inputfile and outputfile is the string file name of the result. --- THE FOLLOWING ARE NOT YET AVAILABLE --- 25. We are currently writing IDL code to automatically correct object spectra for telluric absorption. This will be available later this year. 26. We are writing an IDL routine that will merge the spectra in the individual orders into a single spectrum. --------------------------------------------------------------------------- Limitations of Spextool (to be fixed in future versions): --------------------------------------------------------------------------- 1. Spextool will *NOT* run under versions of IDL earlier than 5.3 and there are no plans to make it compatible with these versions. Currently it works only under a Unix or Linux operating system. 2. At the moment, Spextool works only for the short cross-dispersed (SXD), long cross-dispersed (LXD) and Prism modes. Ability to reduce the Single Order modes will be incorporated in the next version of Spextool. 3. Spextool does not yet have the ability to perform "Optimal" extractions. 4. NOTE: The grating for the SXD mode was changed during the August downtime. This means that spectra acquired before August are located in slightly different positions relative to spectra acquired after August. Because of the present limitations of the wavelength calibration algorithm, this requires the user to copy the relevant wavelength calibration file into the file used by Spextool. The instructions on how to do this are as follows: For data acquired BEFORE August 2000: cd packagedirectory/data where packagedirectory is the directory containing the Spextool package (see #3 above) and "default" is the name of a Spextool subdirectory. cp ShortXD_wavecal_grat1.dat ShortXD_wavecal.dat For data acquired AFTER August 2000: cd packagedirectory/data where packagedirectory is the directory containing the Spextool package (see #3 above) and "default" is the name of a Spextool subdirectory. cp ShortXD_wavecal_grat2.dat ShortXD_wavecal.dat