Data files created at the JCMT are usually distributed in the Starlink NDF format (see “Learning from 25 years of the extensible N-Dimensional Data Format“). An NDF usually contains several arrays, each with a well defined purpose (data values, variances, quality flags, etc). However, each such array has in the past been limited in size to no more than 2,147,483,647 pixels (this is because the software used to access NDFs uses 32 bit signed integers to store pixel counts, etc). Recent work at the EAO has removed this limitation by changing the relevant libraries to use 64 bit integers instead of 32 bit integers. This is a necessary first step towards supporting future instruments that will produce much larger data arrays.
One consequence of this work is a small change to the way NDFs are stored on disk – the integer values that store the pixel origin of each array are now stored as 64 bit integers (type “_INT64”) rather than 32 bit integers (type “_INTEGER”). New NDFs created with these _INT64 values will be readable by software in the “2018A” Starlink release (July 2018) and subsequent nightly builds, but will not be readable by older versions of Starlink. However, NDFs in the new format can be converted to the old format by doing:
% setenv NDF_SHORTORIGIN 1
% kappa
% ndfcopy newndf.sdf oldndf.sdf
It should be noted that the changes described above do not mean that existing Starlink commands in packages such as Kappa or SMURF can now be used to create or process huge NDFs. It just means that the NDF format itself, together with the infrastructure libraries needed to access NDFs, are ready for the next stage in the process, which will be the modification of Kappa, SMURF, etc, to use the new facilities. The one exception is that the CUPID package has now been updated to use the new facilities and so should now be capable of handling huge NDFs.
The new features described above are present in the Starlink nightly build on or after 27th November 2019.
As a final note, this is all new stuff so please be on the look-out for any strange behaviour in the new version of Starlink and report it to EAO.