file: instruction.txt = hands-on demo SDO alignment 
init: Feb 16 2020  Rob Rutten  Deil
last: Apr 24 2021  Rob Rutten  Deil
note: # = comment
todo: tasks A-C below
site: parallel versions .txt, .html, .pdf (latter two with active links) at
  https://robrutten.nl/rrweb/sdo-demo/dircontent.html  
      (or Google "rob rutten webstek" > "Recipes for IDL" > SDO)

Introduction
------------
This instruction was written for a hands-on session at a planned
Solarnet school on solar space- and groundbased co-analysis at MSSL in
March 2020.  The school got covid-shifted to an on-line version in
January 2021.  Meanwhile I webposted this material for users not at
the school.

The tasks are:

  A Beforehand
    - set up my IDL, SSW, JSOC registration etc
    - download required data files
    
  B SDO cross-alignment
    - demo how to get and cross-align SDO cutout sequences
    - inspect alignments for the 2019-11-11 Mercury transit 

  C SST-SDO co-alignment
    - demo how to inspect SST/CRISP data my way 
    - demo how to get and co-align corresponding SDO cutout sequences

I welcome error logs, suggestions, improvements, and also science
discussions of what is seen or not seen in the co-aligned SST-SDO
data.  You are welcome to contact me (R.J.Rutten@uu.nl), especially if
you meet problems.  Skype may serve to share laptop screens; I am
robertjellerutten there but need prior email warning to switch on.

For the showex inspections below it may be better to run IDL not
remotely but in your local computer (maybe per remote license).  


A.  Beforehand (prior to the school)
------------------------------------

1. Get my SDO manual at:
  https://robrutten.nl/Recipes_IDL.html  
   
2. Do step 0 of the manual: get my rjrlib, install, add to IDL search
   path.  Renew if a newer date is given in the func line at the top
   of the manual.
   
3. You must also have SSW including gen, sdo (maybe also vobs, iris,
   optical).  Also David Fanning's coyotelib (see manual).  And add
   IDL search priorities (see manual).  And be registered at JSOC (see
   manual).

4. Download and unzip needed SST data (3G):
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/sst.zip  
   and also the small file:
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/align_sdo_sst.idl  
5. Optional data for further inspections (16M, 10M, 3.5G):
  https://robrutten.nl/rrweb/sdo-demo/2019-11-11/ingress.zip  
  https://robrutten.nl/rrweb/sdo-demo/2019-11-11/egress.zip  
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/sst2rotsdo.zip  
  
6. Quick IDL check:
     cd .../2014-06-24/sst
     IDL> showex,'wb.6563.corrected.aligned_cut.icube'   # granulation movie


---------------------------------------------------------
B. Get and cross-align SDO for the recent Mercury transit
---------------------------------------------------------

  mkdir 2019-11-11/midpoint
  go there, start IDL/SSW.

       IDL> sdo_getdata_rr,'2019.11.11_15:18',5,-13,70,xsize=60,ysize=50,$
            /addfires,name='xxx',email='yyy@zzz'

   # use straight-up single quotes (do not copy-paste the curvy latex
     quotes in the pdf version of this instruction).  Your name xxx and
     email address yyy@zzz must be registered at JOC.

   # it should start ordering data at JSOC.  Normally there will be
     waits to avoid JSOC-forbidden parallel ordering and also waits
     for data downloads if the orders are not yet completed.
     Completion then generates 8 JSOC emails (I taught Google to shunt
     them to spam).  However, if the same data were ordered earlier
     JSOC recognizes duplication and links to the already completed
     files.  The program then warns that the order sizes seem too
     small - this is fine, no waits and no emails to you.  The program
     then downloads and starts processing, flashing many aia_prep.pro
     messages that you can ignore.  It should end with
     "===== sdo_getdata_rr done" and the time it took.
   
   # at home (fast download) with my laptop the whole run takes
     about 25 min incuding the waits (only 16 min for a duplicate
     request).  

   # I show output graphs in appendices A and B of my recent
  https://robrutten.nl/rrweb/rjr-pubs/2020LingAstRep...1R.pdf  
     henceforth LAR-1, especially in figures 70-75 (last pdf pages)
     and their captions.  The "fire detector" defined in the caption
     of fig. 69 is now part of the pipeline and is called by the
     /addfires option above.
     
   # I found the (X,Y) location above with:

       IDL> sdo_featurelocator,'2019.11.11_15:23','continuum'
       
     and pulling out Mercury and clicking on it.  Try this.
     (SSW needs to know your email address, see manual.)
     
   When done inspect the cross-alignments (in your midpoint dir):
   
       IDL> showex,/allsdo,sdodirs=['center/cubes','center/cubesxal']
       
   which loads non-aligned and aligned disk center cubes.  They are
   all synchronised to the 12s-cadence 171 timings and follow solar
   rotation for the cutout center (also largely undoing the 1 px
   sawtooth jumps of the JSOC cutouts).  Blink between pairs of each
   sequence.

   Zoom in to Mercury by pulling out small areas (if this doesn't work
   anymore then you ran out of limited IDL LUNs and must restart IDL).
   Keep in mind that Mercury's shape gets deformed by temporal
   interpolation to the 171 timings (EUV 12s cadence, UV 24s, HMI
   45s).  Worse for HMI which uses 3 exposures to construct continuum
   and magnetogram images. 

   Also zoom in to field edges to observe the limbward increasing
   1600-1700 offsets shown and explained in appendix A of LAR-1.

   If you have interest in chromospheric and coronal heating then
   blink the cubesxal/aiafire and aia193 sequences (iwA=17, iwB=19).
   The ubiquitous grey patches in aiafire are heated chromosphere
   (with similar patterns in reversed Halpha intensity and ALMA), the
   white aiafire spots underlie "coronal bright points" in aia193.
   The diffuse aia193 background around these appears to be partly set
   by the chromosphere, partly leftovers from previous fires.  The
   smallest fires are "Solar Orbiter campfires".  Zoom in by drawing a
   small box around them and blink these cutouts.  Also wavs 12-17 in
   search of magnetic causes.  Return to the full-field display
   underneath with the "quit this" button.
   
   The cross-alignments are done in a sequence of SDO channel pairs
   with the resulting spline-fitted corrections shown as ps plots in
   dir driftscenter.  Inspect these.  Explanation under LAR-1 figure
   75.  The check-xal plot is the final check, comparing two at the
   ends of pairwise chains and so showing combined errors.  Should
   remain within 1 px (AIA 0.6 arcsec).
   
   Inspect the small target cutouts with:

       IDL> showex,/allsdo,sdodirs='target/cubes'

   # sliding the time to it=16 has Mercury on a "fire" as reference
   # hitting the sqrt button may improve greyscale
   
   I did the same as above for the Mercury ingress and egress with:

       IDL> sdo_getdata_rr,'2019.11.11_12:39',20,-936,52.5,xsize=100,ysize=50
       IDL> sdo_getdata_rr,'2019.11.11_17:54',13,940,89.7,xsize=100,ysize=50

      You find the resulting results under
  https://robrutten.nl/rrweb/sdo-demo/2019-11-11/dircontent.html  
      For comparison pull over the zips and inspect target/cubes as above.  


-------------------------------------------
C.  Get SDO for SST/CRISP data and co-align
-------------------------------------------

  This practical uses a brief segment from a high-quality 75-min
  SST/CRISP data set pointing at very quiet Sun under a disk-center
  coronal hole.  The observations were done in June 2014 by Tiago
  Pereira and are property of the Institutt for Teoretisk Astrofysikk
  at Oslo.  If you want to use them further you should obtain
  permission from Luc Rouppe van der Voort
  <l.r.van.der.voort@astro.uio.no> as well as the full-duration data.

  These data are in
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/sst.zip  
  which you should have and unzipped.   Go to dir sst and list.
  
  The three sst/CRISP sequences ("crispex" 6563, 8542, 6302) resulted
  from the usual CRISPRED pipeline: MOMFBD, undoing of the
  time-dependent SST field rotation, and co-alignment putting 6563 and
  6302 on 8542. (I think Luc did this.)

  I cut out a 10-minute-only segment to gain download and processing
  speed in this demo.  The seeing improved during the observation as
  evident in sst/rmsplot.ps showing photospheric contrast rms along
  the full 75-min duration. I initially selected it=218-270 for 10 min
  of consistently very good seeing, but in December 2020 I switched to
  it=60-112 because this less good segment contains some coyly-called
  "Solar Orbiter campfires", the topic of my LAR-1 based on SDO data;
  this dataset adds higher-resolution SST data.


C-1  Inspect the SST data
-------------------------
       (from dir .../2014-06-24)
       cd sst
       IDL> showex,/full

  This assoc-opens 51 parallel movies.  My laptop needs 5s for
  greyscale setup and then opens three screens.
  
  Shift the stream A wavelength slider to iw=6 (wide band 6563) and
  appreciate the overall quality by sliding the time slider (topmost).
  During this segment the seeing varied from mostly good to sometimes
  excellent ("good to excellent" at the SST is "excellent to superb"
  at other telescopes).  Zoom in to (pull out) some small magnetic
  network patch and see how the "flower" morphs with time just as in
  MURaM and Bifrost simulations.  Hit "quit this" on top to return to
  the full-field display.

  The overall seeing was better than in the two SST data sets of
  Rutten++ 2019A\&A...632A..96R
  https://robrutten.nl/rrweb/rjr-pubs/2019AAp...632A..96R.pdf  
  which made much use of "Strous" scatter contour plots as the one now
  on your screen.  Its format is explained in Section 2 and demoed in
  Fig. 5 of that article.  You may mimic that here by moving the
  stream A slider to iw=10 (Halpha-0.8 AA), stream B to iw=18
  (Halpha+0.8 AA).  Hit blink.  The mountain summit represents all
  grey internetwork.  The downward mountain ridge represents RREs,
  tne wider leftward ridge more numerous RBEs, the small upper-right
  ridge the rather scarce magnetic bright points.  This separation of
  different features is more informative than the overall Pearson
  correlation coefficient in the lower-left corner.
  
  Now move B to iw=15 (+0.2 AA), blink and cursor-increment the
  time-delay slider slowly to the right while inspecting the scatter
  diagram per blink.  The contour mountain tilts down from no
  dark-dark correlation at simultaneous sampling to an extended
  left-down promontory of darkest-darkest correlations around delta_t
  = 20 (4 minutes, the cadence is 11.5s).  The blink shows matches
  between RBEs and subsequent core fibrils.  Hit dt=0 to jump back to
  simultaneous showing without correlation (perpendicular dashed
  moment curves).  You see similar time-averaged downtilt when hitting
  the mean(t) button.  Even clearer at Ha+0.4 (iw_B=16).  These
  correlations are evidence that dark downdrafting Halpha core fibrils
  tend to follow on repeating dark RBEs (spicules-II).  We might well
  have added these better-seeing data to our article.

  These data also have CaII 8542 and so repeat the quiet-sun dual-line
  analysis of Cauzzi++ 2009A\&A...503..577C
  https://robrutten.nl/rrweb/rjr-pubs/2009AAp...503..577C.pdf  
>  which used lower-quality DST/IBIS data.  A major result there was
  that Halpha core width and CaII 8452 core minimum intensity both
  sample temperature, clearest in temporal averages.  In that article
  Kevin Reardon cleverly defined ways to measure these quantities
  isolating chromospheric cores.  My program mwseq2special.pro
  implements them.  It is slow by fitting profiles per pixel per
  timestep, so I ran it for you and added the results into the sst
  directory you now have.  They are the first six A and B "xxx_kr"
  "wavelengths" of the showex display.  You can check the reported
  "bright-bright" Halpha-width and CaII 8542 minimum intensity
  correlations by selecting these (iwA=1, iwB=4), inspect the scatter
  plot while blinking and time-sliding, and then check their temporal
  average correlation by hitting mean(t).  In the blinking image
  display you may study how they come about - which features make
  them.  For example, iw=4 versus iw=10 shows that RBEs are hot
  (bright in Halpha width).  In mean(t) its good bright-dark
  correlation survives, showing that most hot Halpha features here
  were recurrent spicules-II.
  
  There is more to see but let's add corresponding SDO data.


C-2  Get and co-align SDO
-------------------------

  Reread my SDO manual.  First now is Step 1: find where and when.
  The SST turret logs provide pointing data but imprecise.
       cd sst
       IDL> sst_findlocation,$
         'wb.6563.corrected.aligned_cut.icube',$
         '2014.06.24','tseries.6563_cut.sav',$
         x_sst,y_sst,angle_sst,px_sst
  which puts SST precisely on SDO in (slow) "Metcalf" iterations
  (I have green light from LMSAL to use "Metcalving" as verb).
  I got:
     itsample_sst = 22 as best SST seeing moment
     after rotation excellent blinks between mucked SST and HMI granulation
     result: (X,Y) = -114.9 -112.3  angle_stx = 62.02  px_stx = 0.0567
             at it=0 (specify in getting SDO): (X,Y) = -115.5 -112.3
  Copy your results in your log file for insertion below.
   
  What SDO timing is needed?
       cd sst (or you are still there)
       IDL> restore,'tseries.6563_cut.sav'
       IDL> help,time       ; nt=53 
       IDL> print,time[0]   ; 08:38:43.402
       IDL> print,time[52]  ; 08:48:38.956  # 10 minutes duration (my cutout)

  Now Step 2: get co-aligned SDO now knowing (X,Y) and timing return
       to top directory = 2014-06-24, mkdir sdo, go there
       IDL> sdo_getdata_rr,'2014.06.24_08:37',12,-115.5,-112.3,/addfires
       In my laptop this took 31 min.
       The check-xal plot was good (explanation under last figure LAR-1).

  Now Step 3: co-align SST and SDO.  I suggest sst2rotsdo output mode
  in which the SST data are co-aligned with the SDO data after
  rotating the latter to the large SST field angle.  This maximizes
  the field within figures (no awkward grey triangles as for (X,Y)
  orientation) and undoes the drifts of the SST guiding (which tracks
  features near field center going their own erratic way).  The SDO
  data strictly follow standard differential rotation, but with a
  0.1-AIA-px wobble left from the JSOC full-px im_patch cutting.
  This I also remove by separating it from the slower drift (two
  spline fits in the resulting align plot opening on your screen).
  This rotation-following output mode is best suited to temporal
  averaging, Fourier or wavelet frequency analysis, time-averaged
  scatter analysis etc., by undoing both AIA wobble and SST drift.

  Step 3 is done by running sdo_stx_align.pro which is a complex
  program needing an elaborate choice of input parameters.  I
  collected my choices in IDL main align_sdo_sst.idl in:
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/dircontent.html  
  Inspect these choices, adapt the path specified at its top to your
  directory, and run it with
    .r align_sdo_sst.idl
  on your IDL command line.  It skips the slow Metcalving because this
  was done already by sst_findlocation.pro, by entering its results
  above.  For setting show=1 you get intermediate showex inspections
  of "forward" = SDO on SST and "reverse" = SST on SDO results, first
  for the best-seeing image pair, later for the final cubefiles of the
  two alignment channels (usually both continuum but your choice).  In
  each showex hit "quit this" to continue the program.  It finally
  transforms and writes all co-aligned data into dir sst2rotsdo, first
  SDO, then SST.

  In my laptop this program took 9 min for these relatively small
  files.  The initial check blink was good thanks to
  sst_findlocation.pro (otherwise Metcalving is needed).  The two
  showex blinkers with the mucked align-images were good.  The
  driftplot showed a drift in y of 5 SST px (10x smaller than AIA px!)
  from the SST tracker plus 1-px SDO leftover wobble; both were
  spline-fitted and removed.  The final showex sequence blinkers
  showed good forward and reverse alignments (slide the time slider
  for check blinking at different times).

  If this co-alignment fails for you or takes to long you can instead
  use my results (5.4G) in:
  https://robrutten.nl/rrweb/sdo-demo/2014-06-24/sst2rotsdo.zip  
C-3  Time for science
---------------------

  Step 4: inspection of the co-aligned data.
       cd 2014-06-24
       IDL> showex,/full,montydirs='sst2rotsdo',spectdirs='sst'

  This assoc-opens 64 parallel movies; in my laptop the greyscale
  setup takes 10s.  The spectdirs serve to add Halpha and 8542
  wavelength labels and for Dopplergram mode.
  
  First blink iwA=1 and iwB=63 (last) = magnetograms to check SDO-SST
  alignment and admire quality difference.  Zoom in and check that it=22
  is best.

  Select the 304x131 ("aia3013", iw=5) fire detector (the next
  "aiafire" is nearly the same due to lack of activity above the greyscale
  threshold and shows only few pixels above the whiten-fire threshold).
  At mid-sequence showex startup there is a "Solar Orbiter campfire"
  within the coronal hole.  Check it out in 171 and 193.  Shift the
  time slider to find some more and wonder whether they come in pairs.
  Blink with the Halpha and 8542 samplings to search for cause or
  effect underneath, also at time delays.  Let me know when you find
  such! (I call them "St. Elmo's fires", just as coyly, for being
  harmless little electric flames.)
  
  Compare RBEs and RREs in the blue and red wings of Halpha and 8542.
  Blink Halpha Doppler at iwA=30 ("black=blue") against Halpha core
  width (iwB=17) and identify heating spicules-II versus subsequent
  cooling return fibrils.  Can you identify tips of spicules-II in
  hotter AIAs as in 2016ApJ...820..124H?

  The upper-left quadrant samples internetwork that is unusually
  non-veiled by Halpha fibrils in this very quiet area so that you can
  actually discern reversed granulation (cut it out, blink iwA=24 and
  iwB=60).  Try to identify acoustic grains in 8542 ("K2V
  Carlsson-Stein shocks") and study time-delay Halpha response.  What
  are the thin swirling Halpha iw=30 Doppler strands there?  Are there
  granular swirls (iw=7)?  Any swirls or swirl heating in AIA?

  All yours!  This rich multi-diagnostic display is the motivation to
  precisely co-align ground- and spacebased data sets.  Adding IRIS
  would be yet better but that day had no SST-IRIS co-pointing.

  Showex also offers an option (upper-left pull-down menu) to show
  periodograms per pixel for <UK> wave types.  I aim to add running
  differences, unsharp masking, line-core integration, and more
  <sometime>.


D  Write your article
---------------------

  Naturally I offer recipes:
  https://robrutten.nl/Recipes_latex.html  
  https://robrutten.nl/Recipes_publications.html  
  You might acknowledge me, but if you ask me as co-author the warning is
  that I first make you redo every figure and then muck with every comma. 


Acknowledgements
----------------
  Gherardo Valori diagnosed and solved various problems.
  SDO is the most important facility of solar physics but only rarely
  mentioned in acknowledgments specifying lesser instruments with
  mandatory (often irritating) formulations.  Likewise, Alan Title
  and Phil Scherrer should be the most acknowledged solar physicists
  whereas lesser facilities may even require coauthorship.
  Harry Warren, Neil Sheeley and Peter Young gave key suggestions
  during my start of this pipeline.  Since then I enjoy frequent help
  from Arthur Amezcua, Greg Slater, John Serafin, Marc DeRosa, Mark
  Cheung, Phil Scherrer, Sam Freeland.  Various users reported deficiencies.