Learning from FITS: Limitations in use in modern astronomical research
| dc.contributor.author | Pascual Ramírez, Sergio | |
| dc.contributor.author | otros, ... | |
| dc.date.accessioned | 2023-06-18T06:48:22Z | |
| dc.date.available | 2023-06-18T06:48:22Z | |
| dc.date.issued | 2015-09 | |
| dc.description | Artículo firmado por 38 autores. ©Elsevier Science BV. The authors wish to thank Bill Pence and Rob Seaman for many insightful discussions. We thank Bob Hanisch and Michael Wise for providing helpful comments that improved the contents of this paper. This research has made use of NASA’s Astrophysics Data System. | |
| dc.description.abstract | The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them. (C) 2015 Elsevier B.V. All rights reserved. | |
| dc.description.department | Depto. de Física de la Tierra y Astrofísica | |
| dc.description.faculty | Fac. de Ciencias Físicas | |
| dc.description.refereed | TRUE | |
| dc.description.status | pub | |
| dc.eprint.id | https://eprints.ucm.es/id/eprint/34699 | |
| dc.identifier.doi | 10.1016/j.ascom.2015.01.009 | |
| dc.identifier.issn | 2213-1337 | |
| dc.identifier.officialurl | http://dx.doi.org/10.1016/j.ascom.2015.01.009 | |
| dc.identifier.relatedurl | http://arxiv.org/abs/1502.00996 | |
| dc.identifier.relatedurl | http://www.sciencedirect.com/ | |
| dc.identifier.uri | https://hdl.handle.net/20.500.14352/24251 | |
| dc.journal.title | Astronomy and computing | |
| dc.language.iso | eng | |
| dc.page.final | 145 | |
| dc.page.initial | 133 | |
| dc.publisher | Elsevier Science BV | |
| dc.rights.accessRights | open access | |
| dc.subject.cdu | 52 | |
| dc.subject.keyword | Open provenance model | |
| dc.subject.keyword | Image compression | |
| dc.subject.keyword | System | |
| dc.subject.keyword | Representations | |
| dc.subject.keyword | Lofar | |
| dc.subject.keyword | Pathfinder | |
| dc.subject.keyword | Telescope | |
| dc.subject.keyword | Extension | |
| dc.subject.keyword | Standard | |
| dc.subject.keyword | Storage | |
| dc.subject.ucm | Astrofísica | |
| dc.subject.ucm | Astronomía (Física) | |
| dc.title | Learning from FITS: Limitations in use in modern astronomical research | |
| dc.type | journal article | |
| dc.volume.number | 12 | |
| dcterms.references | Alexov, A., et al., 2012. Status of LOFAR Data in HDF5 Format, in: Ballester, P., Egret, D., Lorente, N.P.F. (Eds.), Astronomical Data Analysis Software and Systems XXI, volume 461 of ASP Conf. Ser. p. 283. Allegrezza, S., 2012. Flexible Image Transport System: a new standard file format for long-term preservation projects?, in: European Week of Astronomy and Space Science, 1-6 July 2012, Rome. Anderson, K., Alexov, A., Bähren, L., Grießmeier, J.M., Wise, M., Renting, G.A., 2011. LOFAR and HDF5: Toward a New Radio Data Standard, in: Evans, I.N., Accomazzi, A., Mink, D.J., Rots, A.H. (Eds.), Astronomical Data Analysis Software and Systems XX, volume 442 of ASP Conf. Ser. p. 53. Astropy Collaboration, 2013. Astropy: A community Python package for astronomy. A&A 558, A33. doi:10.1051/0004-6361/201322068, arXiv:1307.6212. Barnes, D.G., 1998. Realtime, Object-oriented Reduction of Parkes Multibeam Data using AIPS++, in: Albrecht, R., Hook, R.N., Bushouse, H.A. (Eds.), Astronomical Data Analysis Software and Systems VII, volume 145 of ASP Conf. Ser. p. 32. Berry, D.S., Jenness, T., 2012. New Features in AST: A WCS Management and Manipulation Library, in: Ballester, P., Egret, D., Lorente, N.P.F. (Eds.), Astronomical Data Analysis Software and Systems XXI, volume 461 of ASP Conf. Ser. p. 825. Bertin, E., 2006. Automatic Astrometric and Photometric Calibration with SCAMP, in: Gabriel, C., Arviset, C., Ponz, D., Enrique, S. (Eds.), Astronomical Data Analysis Software and Systems XV, volume 351 of ASP Conf. Ser. p. 112. Calabretta, M.R., Greisen, E.W., 2002. Representations of celestial coordinates in FITS. A&A 395, 1077–1122. doi:10.1051/0004-6361:20021327. Calabretta, M.R., Valdes, F.G., Greisen, E.W., Allen, S.L., in preparation. Representations of distortions in FITS world coordinate systems. A&A http://fits.gsfc.nasa.gov/wcs/dcs_20040422.pdf. Cornwell, T., Humphreys, B., 2010. SKA exascale software challenges. SKA Memo Series 128. Cotton, W.D., Tody, D., Pence, W.D., 1995. Binary table extension to FITS. A&AS 113, 159. Currie, M.J., 1988. Standard data formats. Starlink Bulletin 2, 11. Currie, M.J., 1997. Data-format conversion. Starlink Bulletin 19, 14. Currie, M.J., Privett, G.J., Chipperfield, A.J., Berry, D.S., Davenhall, A.C., 2012. CONVERT – A Format-conversion Package. Starlink User Note 55. DeBoer D. R. et al., 2009. Australian SKA Pathfinder: A High-Dynamic Range Wide-Field of View Survey Telescope. IEEE Proceedings 97, 1507–1521. doi:10.1109/JPROC.2009.2016516. Demleitner, M., Derriere, S., Gray, N., Louys, M., Ochsenbein, F., 2014. Units in the VO. http://www.ivoa.net/documents/VOUnits/. Disney, M.J., Wallace, P.T., 1982. STARLINK. QJRAS 23, 485. Dowler, P., 2012. CAOM-2.0: The Inevitable Evolution of a Data Model, in: Ballester, P., Egret, D., Lorente, N.P.F. (Eds.), Astronomical Data Analysis Software and Systems XXI, volume 461 of ASP Conf. Ser. p. 339. Economou F. et al., 2015. Observatory/data centre partnerships and the VOcentric archive: The JCMT Science Archive experience. Astron. Comp. in press. doi:10.1016/j.ascom.2014.12.005, arXiv:1412.4399. Economou, F., Jenness, T., Currie, M.J., Berry, D.S., 2014. Advantages of extensible self-described data formats: Lessons learned from NDF, in: Manset, N., Forshay, P. (Eds.), Astronomical Data Analysis Software and Systems XXIII, volume 485 of ASP Conf. Ser. p. 355. Garwood, R.W., 2000. SDFITS: A Standard for Storage and Interchange of Single Dish Data, in: Manset, N., Veillet, C., Crabtree, D. (Eds.), Astronomical Data Analysis Software and Systems IX, volume 216 of ASP Conf. Ser. p. 243. Gaudet, S., Dowler, P., Goliath, S., Redman, R., 2008. The JCMT Science Archive and the Virtual Observatory, in: Argyle, R.W., Bunclark, P.S., Lewis, J.R. (Eds.), Astronomical Data Analysis Software and Systems XVII, volume 394 of ASP Conf. Ser. p. 135. George, I.M., Angelini, L., 1995. Specification of Physical Units within OGIP FITS files. http://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/docs/general/ogip_93_001/ogip_93_001.html. Giaretta, D., Taylor, M., Draper, P., Gray, N., McIlwrath, B., 2003. HDX Data Model: FITS, NDF and XML Implementation, in: Payne, H.E., Jedrzejewski, R.I., Hook, R.N. (Eds.), Astronomical Data Analysis Software and Systems XII, volume 295 of ASP Conf. Ser. p. 221. Gom, B., Naylor, D., 2010. Testing results and current status of FTS-2, an imaging Fourier transform spectrometer for SCUBA-2, in: Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy V, volume 7741 of Proc. SPIE. p. 77412E. doi:10.1117/12.857667. Greisen, E.W., 2003. FITS: A Remarkable Achievement in Information Exchange. Information Handling in Astronomy - Historical Vistas 285, 71. doi:10.1007/0-306-48080-8_5. Greisen, E.W., 2011. AIPS Memo 114r: The FITS Interferometry Data Interchange Convention. Greisen, E.W., Calabretta, M.R., 2002. Representations of world coordinates in FITS. A&A 395, 1061–1075. doi:10.1051/0004-6361:20021326. Greisen, E.W., Calabretta, M.R., Valdes, F.G., Allen, S.L., 2006. Representations of spectral coordinates in FITS. A&A 446, 747–771. doi:10.1051/0004-6361:20053818. Greisen, E.W., Harten, R.H., 1981. An Extension of FITS for Groups of Small Arrays of Data. A&AS 44, 371. Greisen, E.W., Wells, D.C., Harten, R.H., 1980. The FITS Tape Formats: Flexible Image Transport Systems, in: Elliott, D.A. (Ed.), Applications of Digital Image Processing to Astronomy, volume 264 of Proc. SPIE. p. 298. doi:10.1117/12.959819. Grosbol, P., Harten, R.H., Greisen, E.W., Wells, D.C., 1988. Generalized extensions and blocking factors for FITS. A&AS 73, 359–364. Groth, P., Deelman, E., Juve, G., Mehta, G., Berriman, B., 2009. PipelineCentric Provenance Model, in: Deelman, E., Taylor, I. (Eds.), Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, ACM. WORKS ’09. p. 4. doi:10.1145/1645164.1645168, arXiv:1005.4457. Hack, W.J., Dencheva, N., Fruchter, A.S., 2013. DrizzlePac: Managing Multicomponent WCS solutions for HST data, in: Friedel, D.N. (Ed.), Astronomical Data Analysis Software and Systems XXII, volume 475 of ASP Conf. Ser. p. 49. Hanisch, R.J., Farris, A., Greisen, E.W., Pence, W.D., Schlesinger, B.M., Teuben, P.J., Thompson, R.W., Warnock, III, A., 2001. Definition of the Flexible Image Transport System (FITS). A&A 376, 359–380. doi:10.1051/0004-6361:20010923. Hanisch, R.J., Schlesinger, B.M., Brotzman, L.E., Kemper, E., Teuben, P.J., Steenbert, M.E.V., Warren Jr., W.H., White, R.A., 1993. Definition of the Flexible Image Transport System (FITS). NOST 100-1.0, NASA/- Science Office of Standards and Technology, http://fits.gsfc.nasa.gov/standard10/fits_standard10.pdf. Harten, R.H., Grosbol, P., Greisen, E.W., Wells, D.C., 1988. The FITS tables extension. A&AS 73, 365–372. HEASARC FITS Working Group, 2007. The CONTINUE Long String Keyword Convention. http://fits.gsfc.nasa.gov/registry/continue/continue.pdf. Herschel Team, 2008. Herschel Data Processing: Architecture and Design. http://herschel.esac.esa.int/hcss-doc-11.0/load/hcss_drm/ia/doc/add/index.html. Holland, W.S., et al., 2013. SCUBA-2: the 10,000 pixel bolometer camera on the James Clerk Maxwell Telescope. MNRAS 430, 2513–2533. doi:10.1093/mnras/sts612. Hotan, A.W., van Straten, W., Manchester, R.N., 2004. PSRCHIVE and PSRFITS: An Open Approach to Radio Pulsar Data Storage and Analysis. PASA 21, 302–309. doi:10.1071/AS04022, arXiv:astro-ph/0404549. Jenness, T., 2015. Reimplementing the Hierarchical Data System using HDF5. Astron. Comp. in press. Jenness, T., et al., 2015. Learning from 25 years of the extensible NDimensional Data Format. Astron. Comp. in press. doi:10.1016/j.ascom.2014.11.001, arXiv:1410.7513. Jenness, T., Berry, D.S., Cavanagh, B., Currie, M.J., Draper, P.W., Economou, F., 2009. Developments in the Starlink Software Collection, in: Bohlender, D.A., Durand, D., Dowler, P. (Eds.), Astronomical Data Analysis Software and Systems XVIII, volume 411 of ASP Conf. Ser. p. 418. Jennings, D.G., Pence, W.D., Folk, M., 1995. Convert: Bridging the Scientific Data Format Chasm, in: Shaw, R.A., Payne, H.E., Hayes, J.J.E. (Eds.), Astronomical Data Analysis Software and Systems IV, volume 77 of ASP Conf. Ser. p. 229. Jennings, D.G., Pence, W.D., Folk, M., Schlesinger, B., 2007. A Hierarchical Grouping Convention for FITS. http://fits.gsfc.nasa.gov/registry/grouping/grouping.pdf. Kitaeff, V.V., Cannon, A., Wicenec, A., Taubman, D., 2015. Astronomical imagery: Considerations for a contemporary approach with JPEG2000. Astron. Comp. in press. doi:10.1016/j.ascom.2014.06.002, arXiv:1403.2801. Library of Congress, 2012. Flexible Image Transport System (FITS), Version 3.0. http://www.digitalpreservation.gov/formats/fdd/fdd000317.shtml. McCann, M., Gomes, K., 2008. Oceanographic Data Provenance Tracking with the Shore Side Data System, in: Freire, J., Koop, D., Moreau, L. (Eds.), Provenance and Annotation of Data and Processes. Springer-Verlag, pp. 309–322. doi:10.1007/978-3-540-89965-5_30. Meyerdierks, H., 1991. Error handling applications. Starlink Bulletin 8, 19. Moreau L. et al., 2011. The Open Provenance Model core specification (v1.1). Future Generation Computer Systems 27, 743–756. doi:10.1016/j.future.2010.07.005. Moreau, L., Freire, J., Futrelle, J., Mcgrath, R.E., Myers, J., Paulson, P., 2008. The Open Provenance Model: An Overview, in: Freire, J., Koop, D., Moreau, L. (Eds.), Provenance and Annotation of Data and Processes. Springer-Verlag, pp. 323–326. doi:10.1007/978-3-540-89965-5_31. Muders D. et al., 2006. APECS - the Atacama pathfinder experiment control system. A&A 454, L25–L28. doi:10.1051/0004-6361:20065359. Ochsenbein, F., 2000. Standard for Documentation of Astronomical Catalogues. http://vizier.u-strasbg.fr/doc/catstd.htx. Ott, S., 2010. The Herschel Data Processing System – HIPE and Pipelines – Up and Running Since the Start of the Mission, in: Mizumoto, Y., Morita, K.I., Ohishi, M. (Eds.), Astronomical Data Analysis Software and Systems XIX, volume 434 of ASP Conf. Ser. p. 139. arXiv:1011.1209. Pence, W.D., Chiappetti, L., Page, C.G., Shaw, R.A., Stobie, E., 2010. Definition of the Flexible Image Transport System (FITS), version 3.0. A&A 524, A42. doi:10.1051/0004-6361/201015362. Pence, W.D., Seaman, R., White, R.L., 2009. Lossless Astronomical Image Compression and the Effects of Noise. PASP 121, 414–427. doi:10.1086/599023, arXiv:0903.2140. Peters, S.M., Kitaeff, V.V., 2014. The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery. Astron. Comp. 6, 41. doi:10.1016/j.ascom.2014.06.003, arXiv:1401.7433. Petry, D., CASA Development Team, 2012. Analysing ALMA Data with CASA, in: Ballester, P., Egret, D., Lorente, N.P.F. (Eds.), Astronomical Data Analysis Software and Systems XXI, volume 461 of ASP Conf. Ser. p. 849. arXiv:1201.3454. Rajasekar, A., Moore, R., Vernon, F., 2007. iRODS: A Distributed Data Management Cyberinfrastructure for Observatories. AGU Fall Meeting Abstracts, B1214. Redman, R.O., Dowler, P., 2013. Implementing a Common Database Architecture at the CADC using CAOM-2, in: Friedel, D.N. (Ed.), Astronomical Data Analysis Software and Systems XXII, volume 475 of ASP Conf. Ser. p. 159. Seaman, R., Pence, W., White, R., Dickinson, M., Valdes, F., Zárate, N., 2007. Astronomical Tiled Image Compression: How and Why, in: Shaw, R.A., Hill, F., Bell, D.J. (Eds.), Astronomical Data Analysis Software and Systems XVI, volume 376 of ASP Conf. Ser. p. 483. Shaya, E., Thomas, B., Cheung, C., 2001. Specifics on a XML Data Format for Scientific Data, in: Harnden, Jr., F.R., Primini, F.A., Payne, H.E. (Eds.), Astronomical Data Analysis Software and Systems X, volume 238 of ASP Conf. Ser. p. 217. Shupe, D.L., Laher, R.R., Storrie-Lombardi, L., Surace, J., Grillmair, C., Levitan, D., Sesar, B., 2012. More flexibility in representing geometric distortion in astronomical images, in: Software and Cyberinfrastructure for Astronomy II, volume 8451 of Proc. SPIE. p. 84511M. doi:10.1117/12.925460. Thomas B. et al., 2014. Significant problems in FITS limit its use in modern astronomical research, in: Manset, N., Forshay, P. (Eds.), Astronomical Data Analysis Software and Systems XXIII, volume 485 of ASP Conf. Ser. p. 351. Thomas, B., Shaya, E., Cheung, C., 2001. Converting FITS into XML: Methods and Advantages, in: Harnden, Jr., F.R., Primini, F.A., Payne, H.E. (Eds.), Astronomical Data Analysis Software and Systems X, volume 238 of ASP Conf. Ser. p. 487. Thureau, N.D., Ireland, M., Monnier, J.D., Pedretti, E., 2006. Software tools for optical interferometry, in: Advances in Stellar Interferometry, volume 6268 of Proc. SPIE. p. 62683C. doi:10.1117/12.670560. Tilmes, C., Fleig, A.J., 2008. Provenance Tracking in an Earth Science Data Processing System, in: Freire, J., Koop, D., Moreau, L. (Eds.), Provenance and Annotation of Data and Processes. Springer-Verlag, pp. 221–228. doi:10.1007/978-3-540-89965-5_23. Tingay S. J. et al., 2013. The Murchison Widefield Array: The Square Kilometre Array Precursor at Low Radio Frequencies. PASA 30, 7. doi:10.1017/pasa.2012.007, arXiv:1206.6945. Valdes, F., 1993. The IRAF/NOAO Spectral World Coordinate Systems, in: Hanisch, R.J., Brissenden, R.J.V., Barnes, J. (Eds.), Astronomical Data Analysis Software and Systems II, volume 52 of ASP Conf. Ser. p. 467. van Haarlem M. P. et al., 2013. LOFAR: The LOw-Frequency ARray. A&A 556, A2. doi:10.1051/0004-6361/201220873, arXiv:1305.3550. Warren-Smith, R.F., Berry, D.S., 1998. World Coordinate Systems as Objects, in: Albrecht, R., Hook, R.N., Bushouse, H.A. (Eds.), Astronomical Data Analysis Software and Systems VII, volume 145 of ASP Conf. Ser. p. 41. Warren-Smith, R.F., Wallace, P.T., 1993. The STARLINK Software Collection, in: Hanisch, R.J., Brissenden, R.J.V., Barnes, J. (Eds.), Astronomical Data Analysis Software and Systems II, volume 52 of ASP Conf. Ser. p. 229. Wells, D.C., 1997. Speculations on the Future of FITS, in: Hunt, G., Payne, H. (Eds.), Astronomical Data Analysis Software and Systems VI, volume 125 of ASP Conf. Ser. p. 257. Wells, D.C., Greisen, E.W., 1979. FITS: a flexible image transport system, in: Sedmak, G., Capaccioli, M., Allen, R.J. (Eds.), Image Processing in Astronomy, p. 445. Wells, D.C., Greisen, E.W., Harten, R.H., 1981. FITS: a flexible image transport system. A&AS 44, 363–370. West, J.L., Cameron, I.D., 2006. Using the Medical Image-Processing Package, ImageJ, for Astronomy. JRASC 100, 242. Wicenec, A., Grosbol, P., Pence, W.D., 2009. The ESO HIERARCH Keyword Convention. http://fits.gsfc.nasa.gov/registry/hierarch_keyword.html. | |
| dspace.entity.type | Publication | |
| relation.isAuthorOfPublication | 67ec03af-ce83-4a1a-8dd6-c4b4bc6cc3bc | |
| relation.isAuthorOfPublication.latestForDiscovery | 67ec03af-ce83-4a1a-8dd6-c4b4bc6cc3bc |
Download
Original bundle
1 - 1 of 1
Loading...
- Name:
- pascualramirez35preprint.pdf
- Size:
- 319.51 KB
- Format:
- Adobe Portable Document Format
