KAPPA/POL2MAP bug causes incorrect variances and error values

A bug was introduced into the cmult and cdiv commands in the Starlink KAPPA package on 21st May 2021. This bug caused incorrect Variance values to be stored in the output and was fixed on 12th July 2021. Consequently, anyone who uses these commands and who last rsynced the nightly Starlink build from EAO between these dates should consider rsyncing the current nightly build, which includes the bug fix.

POL2 users should be aware that the cmult command is used within the pol2map script when converting values from pW to mJy/beam. This means that vector catalogues produced by pol2map will contain incorrect DI/DQ/DU values if starlink was rsynced between the two dates mentioned above.  Again, the solution is to rsync the current nightly build, which includes the bug fix.

Bug in POL2MAP

A  recent change in the Starlink KAPPA package causes the SMURF pol2map command to crash . The main symptom is a message at the end of the pol2map logfile that looks something like this:

+ -50
. could not convert string to float: '+ -50'

The change that causes this problem occurred on 30th April 2021 so will affect Starlink installations copied from EAO on or after that date. The problem has been fixed today (19th May) so anyone affected by this problem should copy starlink again after completion of the next nightly build.

Thanks to Jeremy Yang for reporting this problem.

New FITS Header indicating makemap convergence

David Berry recently added a new feature to SMURF’s makemap. There is now a FITS header in the output maps which will let you know if makemap successfully converged.

The new header is NCONTNCV – the Number of CONTiguous chunks that did Not ConVerge. This should be zero if everything went well. If you are reducing data yourself, you can also check the makemap output or the ORAC-DR log for more information.

You will have access to this feature if you are using an rsynced Starlink build from after the 19th of June 2019. Observations in the archive reduced from 19th June 2019 onwards should also have this FITS header present.

You can check the fitsheaders in the output file with the KAPPA commands fitslist or fitsval, or if you are downloading a reduced file from the archive in .FITS format you can use any of your favourite FITS header viewers.

Fix for skyloop convergence problem

A bug in the smurf:skyloop command has recently been found and fixed. This bug could cause negative or zero values to be included in the extinction  (“EXT”) model. This in turn could cause effectively random behaviour in the other models, resulting in very poor convergence and some spurious values being introduced into the final map (if convergence does eventually happen). This bug is triggered by one or more observations having some time slices for which no extinction values are available (indicated by the presence of the string ” EXT:” in the skyloop log file). If such holes in the extinction data extend over only a few time slices, skyloop interpolates across them using the extinction values on either side of the hole. However,  there was an error in this interpolation that led to the holes being filled with negative or zero values. This bug has now been fixed (as of 22nd May 2019).  This fix affects both direct use of skyloop and indirect use via the pol2map command.


A new dimmconfig that uses PCA

The $STARLINK_DIR/share/smurf directory includes several “dimmconfig” files that package up commonly used groups of configuration parameter values for use by the makemap command.  A new file called dimmconfig_pca.lis has recently been added, which can be combined with  other dimmconfig files to tell makemap to include a PCA model in its iterative algorithm (the PCA model removes noise signals that are correlated between multiple bolometer time-streams). For instance, to use a PCA model when creating a map of a bright extended source, you could run makemap as follows:

% more conf
% makemap config=^conf

To process compact sources, change “bright_extended” above to “bright_compact“.

Using a PCA model can help to reduce the spurious extended structures that often appear in SCUBA-2 maps (although this benefit is bought at the cost of a much extended run time). For instance, below are four 850 um maps of DR 21 – the top row shows the maps made with the basic “bright extended” dimmconfig, and the bottom shows eh results of adding in the new PCA dimmconfig:

Below are the mosaics of the four observations, with the difference map shown in between:

As another example, the following panes show similar maps for three observations of the Serpens South field:

Faster PCA

By default, the SMURF makemap command identifies and removes a single correlated background signal – called the common-mode – from all bolometer time-streams. However, in some cases there is clear evidence that there is more than one correlated background signal present in the bolometer time streams. This is particularly evident in POL2 data, where the varying instrumental polarisation causes different parts of the focal plane to see different levels of sky polarisation. For POL2, better maps are created if multiple correlated background signals are identified and removed . This is achieved within makemap using a process called Principal Component Analysis (PCA). The down side to using PCA is that it is very very slow – it is just about practical in the case of POL2 data because of the very low scan speed and consequent very low sample rate. 

A change introduced into SMURF on 1st April  2019 should speed up PCA by a factor of 2 or 3, making POL2 reductions quicker and maybe making PCA background removal practical for non-POL2 data. Maps created using the new PCA system will not be identical to maps made with the old system, but the differences should be well within the noise levels (the pixel variances remain largely unaffected).

To use PCA within a normal run of makemap it is recommended to add the following to your config file:

modelorder = (com,gai,pca,ext,flt,ast,noi) 
pca.pcathresh = -50

It is usually a good idea to mask out the source when calculating the PCA model on the first few iterations, since this seems to aid convergence. The same sort of mask should be used with PCA as is used with AST, but it should only be applied for a few iterations. So for instance for a point source you could add:

pca.zero_circle = 0.01667
pca.zero_niter = 2

This masks the PCA model on the first two iterations using a circle of radius 60 arc-seconds (0.01667 degrees).

Using PCA usually causes makemap to converge more slowly, but often produces maps with lower levels of artificial structures. If pca.pcathresh is negative, the absolute value indicates the number of correlated signals to remove as the background in each bolometer. Smaller numbers result in a lower level of noise reduction in the final map, but faster convergence. Larger numbers result in a higher level of noise reduction in the final map, but slower convergence. The default value is 50, which usually seems to be a reasonable compromise.

The left map below was made with a default common-mode background (no PCA)  and the centre map was made with PCA background removal as shown above. Each observation took about 6 minutes to create without PCA and about 25 minutes with the new faster PCA. The right map shows the difference between the other two maps. All three use the same scaling.