{"id":11250,"date":"2020-06-01T05:46:39","date_gmt":"2020-06-01T15:46:39","guid":{"rendered":"http:\/\/www.eaobservatory.org\/jcmt\/?p=11250"},"modified":"2020-12-29T10:55:29","modified_gmt":"2020-12-29T20:55:29","slug":"re-modelling-the-error-estimates-in-a-pol2-vector-catalogue","status":"publish","type":"post","link":"https:\/\/www.eaobservatory.org\/jcmt\/2020\/06\/re-modelling-the-error-estimates-in-a-pol2-vector-catalogue\/","title":{"rendered":"Re-modelling the error estimates in a POL2 vector catalogue"},"content":{"rendered":"<p>If the <span style=\"font-family: 'andale mono', monospace\">MAPVAR=YES<\/span> option is used when running the SMURF <span style=\"font-family: 'andale mono', monospace\">pol2map<\/span>\u00a0 script, the I, Q and U error estimates in the resulting vector catalogue will be based on the spread of pixel values at each point on the sky in the I, Q and U maps made from individual observations. Thus if 20 observations are processed by <span style=\"font-family: 'andale mono', monospace\">pol2map<\/span> to create a vector catalogue, then each I, Q or U error estimate in the vector catalogue will be based on the spread of 20 independent measurements of I, Q or U.\u00a0 Even though 20 observations is a lot of POL2 data, 20 is still a fairly small number from which to produce an accurate estimate of the error. Consequently, it is usual to see a large level of random &#8220;noise&#8221; on the error estimates, as in the following example which shows the total intensity (I) error estimates taken from a 12&#8243; vector catalogue near Ophiuchus L 1688 (the noise level increases towards the edge of the map due to there being fewer bolometer samples per pixel near the edge):<\/p>\n<p><a href=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error.png\"><img loading=\"lazy\" class=\"size-medium wp-image-11251 aligncenter\" src=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error-297x300.png\" alt=\"\" width=\"297\" height=\"300\" srcset=\"https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error-297x300.png 297w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error-768x775.png 768w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error-149x150.png 149w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/noise_on_error.png 939w\" sizes=\"(max-width: 297px) 100vw, 297px\" \/><\/a><\/p>\n<p>The uncertainty on the error estimate can cause some vectors that are clearly wrong (e.g. because they are very different to nearby vectors) to have anomalously low error estimates and so to be included in the set of &#8220;good&#8221; vectors (i.e. vectors that pass some suitable selection criterion based on the noise estimates).<\/p>\n<p>One simple solution to this could be to apply some spatial smoothing to the error estimates. This would be a reasonable thing to do if there were no compact sources in the map. The errors close to a compact source are generally higher than those in a background region because of things like pointing errors, calibration errors, etc. These things cause a compact source to look slightly different in each observation and so cause higher error estimates in the vector catalogue. The above error estimates map shows this effect in the higher values at the very centre. Simply smoothing this map would spread that central feature out, artificially decreasing the peak error and increasing the errors in the neighbouring background pixels.<\/p>\n<p>An alternative to smoothing is to split the total noise up into several different components, create a model of each component , and then add the models together. The <span style=\"font-family: 'andale mono', monospace\">pol2noise<\/span> script in SMURF has been modified to include\u00a0 a facility to re-model the noise estimates in a vector catalogue using such an approach. This facility is used by setting <span style=\"font-family: 'andale mono', monospace\">MODE=REMODEL<\/span>\u00a0on the <span style=\"font-family: 'andale mono', monospace\">pol2noise<\/span> command line:<\/p>\n<pre>% pol2noise mycat.FIT mode=remodel out=newcat.FIT exptime=iext debiastype=mas<\/pre>\n<p>This\u00a0 creates an output catalogue (newcat.FIT) holding a copy of the input catalogue (mycat.FIT), and then calculates new values for all the error columns in the output catalogue. The new I, Q and U error values are first derived from a three component model of the noise in each quantity, and then errors for the derived quantities (PI, P and ANG) are found. New values of PI and P are also found using the specified de-biasing algorithm. <span style=\"font-family: 'andale mono', monospace\"><span style=\"font-family: georgia, palatino, serif\">The file <span style=\"font-family: 'andale mono', monospace\">iext.sdf<\/span> holds the total intensity coadd map created by <span style=\"font-family: 'andale mono', monospace\">pol2map <span style=\"font-family: georgia, palatino, serif\">and is used to define the total exposure time in each pixel<\/span><\/span>. The re-modelled total intensity error estimates look like this :<\/span><\/span><\/p>\n<p><a href=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod.png\"><img loading=\"lazy\" class=\"size-medium wp-image-11254 aligncenter\" src=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod-297x300.png\" alt=\"\" width=\"297\" height=\"300\" srcset=\"https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod-297x300.png 297w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod-768x775.png 768w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod-149x150.png 149w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/remod.png 915w\" sizes=\"(max-width: 297px) 100vw, 297px\" \/><\/a><\/p>\n<p>Most of the noise has gone without reducing the resolution. The script displays the original and re-modelled error estimates for each Stokes parameter (I, Q and U), the residuals between the two and a scatter plot. The best fitting straight line through the scatter plot is also displayed:<\/p>\n<p><a href=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise.png\"><img loading=\"lazy\" class=\"alignnone wp-image-11255\" src=\"http:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-300x222.png\" alt=\"\" width=\"755\" height=\"559\" srcset=\"https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-300x222.png 300w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-1024x759.png 1024w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-768x569.png 768w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-202x150.png 202w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise-150x111.png 150w, https:\/\/www.eaobservatory.org\/jcmt\/wp-content\/uploads\/sites\/2\/2020\/06\/pol2noise.png 1448w\" sizes=\"(max-width: 755px) 100vw, 755px\" \/><\/a><\/p>\n<p>The three components used to model the error on each Stokes parameter (I, Q or U) are described below:<\/p>\n<ul>\n<li>The <strong>background component<\/strong>: This is derived from an exposure time map \u00a0(obtained from <span style=\"font-family: 'andale mono', monospace\">iext.sdf<\/span> in the above example). The background component is equal to A.t<sup>B<\/sup> , where &#8220;t&#8221; is the exposure time at each pixel and A and B are constants determined by doing a linear fit between the log of the noise estimate in the catalogue (DQ, DU or DI) and the log of the exposure time (in practice, B is usually close to -0.5). The fit excludes bright source areas, but also excludes a thin rim around the edge of the map where the original noise estimates are subject to large inaccuracies. Since the exposure time map is usually very much smoother than the original noise estimates, the background component is also much smoother.<\/li>\n<li>The <strong>source component<\/strong>: This represents the extra noise found in and around compact sources caused by pointing errors, calibration errors, etc. The background component is first subtracted from the catalogue noise estimates and the residual noise values are then modelled using a collection of Gaussians. This modeling is done using the GaussClumps algorithm provided by the <span style=\"font-family: 'andale mono', monospace\">findclumps<\/span> command in the Starlink CUPID package. The noise residuals are first divided into a number of &#8220;islands&#8221;, each island being a collection of contiguous pixels with\u00a0 noise residual significantly higher than zero (this is done using the FellWalker algorithm in CUPID). The GaussClumps algorithm is then used to model the noise residuals in each island. The resulting model is smoothed lightly using a Gaussian kernel of FWHM 1.2 pixels.<\/li>\n<li>The <strong>residual component<\/strong>: This represents any noise not accounted for by the other two models. The noise residuals are first found by subtracting the other two components from the original catalogue noise estimates. Any strong outlier values are removed and the results are smoothed more heavily using a Gaussian kernel of FWHM 4 pixels.<\/li>\n<\/ul>\n<p>The final model is the sum of the above three components. The new DI, DQ and DU values are found independently using the above method. The errors for the derived quantities (DPI, DP and DANG) are then found from DQ, DU and DI using the usual error popagation formulae. Finally new P and PI values are found using a specified form of de-biasing (see parameter DEBIASTYPE).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If the MAPVAR=YES option is used when running the SMURF pol2map\u00a0 script, the I, Q and U error estimates in the resulting vector catalogue will be based on the spread of pixel values at each point on the sky in the I, Q and U maps made from individual observations.\u2026 <a class=\"continue-reading-link\" href=\"https:\/\/www.eaobservatory.org\/jcmt\/2020\/06\/re-modelling-the-error-estimates-in-a-pol2-vector-catalogue\/\">Continue reading<\/a><\/p>\n","protected":false},"author":13,"featured_media":11254,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[36],"tags":[37],"_links":{"self":[{"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/posts\/11250"}],"collection":[{"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/comments?post=11250"}],"version-history":[{"count":9,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/posts\/11250\/revisions"}],"predecessor-version":[{"id":11263,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/posts\/11250\/revisions\/11263"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/media\/11254"}],"wp:attachment":[{"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/media?parent=11250"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/categories?post=11250"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.eaobservatory.org\/jcmt\/wp-json\/wp\/v2\/tags?post=11250"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}