Running brutifus

The spirit of brutifus is that each user can choose, depending on the object at hand & the quality of the data, what processing steps are warranted. These are governed by the procsteps_brutifus.yaml file. In parallel, the user can define all relevant parameters using the params_brutifus.yaml file. The two files (that rely on YAML syntax) are shipped as supplementary files with the code. The following high-level entry point will create local copies in your current location (assumed to be your favorite processing area). In a terminal:

cd ~/where/ever/you/want
brutifus --setup

Doing so will create a local copy of both files in the current directory, together with generic working directories.

Note

I very much suggest to run a brutifus --setup for each project, rather than using a single processing area for all projects.

Setting parameters with params_brutifus.yaml

All the brutifus parameters with a scientific impact can be specified inside this file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# === Very-high-level parameters ===
systemtex: True # Use the system LaTeX instead of the Python one ?

# === High-level parameters ===
target: 'SNR123'        # Short and sweet string, for the filenames
z_target: 0.0           # Redshift of the target
inst: 'MUSE'            # Instrument name
multiprocessing: True   # Use multi-cpus ? integer for upper limit, or True for all
verbose: True           # Print stuff that may be of interest ?

# === Data location ===
data_loc: './some/relative/path' # Where is the data ?
data_fn: 'some_cube.fits'        # What is the name of the datacube ?

# === Processing parameters ===
# --- Constructing SNR maps ---
snr_ranges:          # lam_min, lam_max, 'c'ontinuum, or 'e'mission
#   - [7400., 8500., 'c']
#   - [6560., 6570., 'e']

# --- Manual sky subtraction ---
sky_regions:         # x0,y0, radius, or x0,y0,dx,dy
#   - [265, 88, 3]
#   - [254, 51, 2]

# --- Correcting for the Galactic extinction ---
gal_curve: 'f99'     # Set to 'f99' to follow NED.
gal_rv: 3.1          # Set to 3.1 to follow NED.
Ab:                  # Ab (Landolt) from NED.
Av:                  # Av (Landolt) from NED.

# --- LOWESS Continuum fitting --- 
lowess_it: 10        # Number of iteration for sigma-clipping to get rid of outliers
lowess_frac: 0.05    # % of the array used for deriving each point. 0.05 = sweet spot?

Todo

A detailed list of what everything is/does would be nice. Until then, look at the comments inside the file for a clarification of what everything does.

Setting the processing steps with procsteps_brutifus.yaml

brutifus is designed to batch process all the individual spectra inside a given IFU datacube individually, exploiting multiple cpus (when available) to gain speed.

The different steps to be executed, in sequential order, are defined in procsteps_brutifus.py. Each step can be re-arranged as needed by the user. In practice, it is mostly the plotting steps that will be most duplicated to visualize the different intermediate products.

Each step contains a handful of common parameters, including a step name, whether to run it (True) or not (False), and the suffix added to the associated product file. Arguments (such as cut levels for images) are passed via the args container. Of note are the name_in and name_out values: these allow the user to give short reference tags to specific intermediate products, in order to easily feed them to subsequent steps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
# Each step below will be execute in order. Users should feel free to re-arrange/skip/
# delete/duplicate steps as needed. Note: step names are not optional !!!

# --------------------------------------------------------------------------------------------------
# The first two functions are used for plotting ... use them as often as needed to visualize
# the different data products !
# --- Make some RGB plots from the datacube ... ---
-  step: 'plot_RGB'
   run: True
   suffix: 'rgb1' 
   args:
      name_in: 'wcs-corr_cube' # What cube do I want to plot ?
      bands: [[6560,6570,7500,8750,4750,5500]]
      stretches: [['arcsinh','arcsinh','arcsinh']] # arcsinh, linear
      plims: [[1.,98.,1.,98.,1.,98.]]
      vlims: [[null, null, null, null, null, null]]
      gauss_blurs: [[null, null, null]]
      conts: [[null, null, null, null, null, null]]

# --- Make some BW plots ... ---
-  step: 'plot_BW'
   run: False
   suffix: 'bw1' 
   args:
      name_in: 'wcs-corr_cube'   # What cube do I want to plot ?
      bands: [[6556, 6579]]      # lam min and max ... add more pairs to plot more plots !
      stretches: ['linear']      # Image stretch ... add more to tweak different plots.
      plims: [[10, 99]]
      vlims: [[null, null]]      
      gauss_blurs: [null]
      conts: [[null, null]]

# --------------------------------------------------------------------------------------------------

# === Correct the WCS values to check with Gaia. ===
-  step: 'adjust_WCS' 
   run: False
   suffix: 's00'
   args:
      name_in: 'raw_cube'
      name_out: 'wcs-corr_cube'
   
# === First, compute some S/N maps for the continuum and emission lines of interest ... ===
-  step: 'crude_snr_maps' 
   run: False 
   suffix: 's01'
   args:
      name_in: 'raw_cube' # What cube do I want to process ?
      do_plot: True  # Do I want to save some pretty pictures ?
      zcorr_lams: False # True = correct wavelength ranges for redshift

# === Perform some sky subtraction, if needed ===
-  step: 'sky_sub' 
   run: False 
   suffix: 's03' 
   args:
      name_in: 'raw_cube'
      name_out: 'skysub_cube'
      
# === Correct for galactic reddening via the values given on NED ===
-  step: 'gal_dered' 
   run: False 
   suffix: 's04'
   args:
      name_in: 'skysub_cube'
      name_out: 'galdered_cube'
      do_plot: True
      
   
# === Continuum fitting ===
# First, fit the continuum using a LOWESS filter. Much more elegant than a polynomial!
-  step: fit_continuum 
   run: False 
   suffix: 's05'
   args:
      name_in: 'galdered_cube'
      start_row: 0 # Where to start the fitting ? None = 0
      end_row:  # Where to end the fitting ? None = max
      method: 'lowess' # What kind of fitting is this ?

# Gather the output in a single datacube
-  step: 'make_continuum_cube'
   run: False
   suffix: 's06'
   args:
      method:'lowess'


# Subtract the sky continuum
-  step: 'subtract_continuum'
   run: False
   suffix: 's07'
   args:
      name_in: 'galdered_cube'
      name_out: 'consub_cube'
      method: 'lowess'

Todo

A detailed list of what everything is/does would be nice. Until then, look at the comments inside the file for a clarification of what everything does.

Launching the post-processing

Having prepared both params_brutifus.yaml and procsteps_brutifus.yaml, you are ready to launch brutifus. This is done via the following high-level entry point:

>>> brutifus -e procsteps_brutifus.yaml params_brutifus.yaml

If all goes according to plan, brutifus will start processing your data, one step after the other, printing some info along the way. Some steps can take time (up to several hours) ! Just be patient - progress indicators will help you monitor each task, if you set 'verbose': True inside params_brutifus.yaml.