Just Solve the File Format Problem - User contributions [en]

Stata

2012-11-05T16:42:18Z

Mwoollard: Created page with "Some details [http://faculty.smu.edu/millimet/classes/eco7377/lectures/StataIntro.pdf here]"

Some details [http://faculty.smu.edu/millimet/classes/eco7377/lectures/StataIntro.pdf here]

Scientific Data formats

2012-11-05T16:38:20Z

Mwoollard:

{|
|[[File Formats]]
| >
|[[Electronic File Formats]]
| >
|Scientific Data formats
|}

== General ==
* [[cdf]] (Common Data Format)
* [[EAS3]] (binary file format for structured data)
* [[hdf]] (Hierarchical Data Format, from NASA)
* [[NetCDF]] (Network Common Data Format)
* [[SDF]] (Simple Data Format, a platform-independent, precision-preserving binary data I/O format capable of handling large, multi-dimensional arrays)
* [[SDXF]] (Structured Data Exchange Format)
* [[Silo]] (a storage format for visualization developed at Lawrence Livermore National Laboratory)* [[XDF]] (eXtensible Data Format)
* [[XSIL]] (Extensible Scientific Interchange Language)

== Astronomical and Space ==
* [[FITS]] (Flexible Image Transport System)
* [[PDS/ODL]] (Planetary Data System)

== Biological ==

* [[AB1]] (Chromatogram files used by DNA sequencing instruments from Applied Biosystems)
* [[ACE]] (Sequence assembly format)
* [[BAM]] (Binary compressed SAM format)
* [[BED]] (Browser extensible display format describing genes and other features of DNA sequences)
* [[CAF]] (Common Assembly Format for sequence assembly)
* [[EMBL]] (Flatfile format used by the EMBL for nucleotide and peptide sequences)
* [[FASTA and FASTQ]] (File format for sequence data, FASTQ with quality).
* [[GenBank]] (Flatfile format used by NCBI for nucleotide and peptide sequences)
* [[GFF]] (General feature format for describing genes and other features of DNA, RNA and protein sequences)
* [[GTF]] (Gene transfer format holds information about gene structure)
* [[NEXUS]] (Encodes mixed information about genetic sequence data in a block structured format)
* [[PDB]] (Structures of biomolecules deposited in Protein Data Bank)
* [[PHD]] (Output from the basecalling software Phred)
* [[SAM]] (Sequence Alignment/Map format)
* [[SCF]] (Staden chromatogram files used to store data from DNA sequencing)
* [[SBML]] (Systems Biology Markup Language used to store biochemical network computational models)
* [[Stockholm]] (Representing multiple sequence alignments)
* [[Swiss-Prot]] (Flatfile format used for protein sequences from the Swiss-Prot database)
* [[VCF]] (Variant Call Format)

== Biomedical signals (time series) ==

* [[ACQ]] (AcqKnowledge)
* [[BCI2000]] (The BCI2000 project)
* [[BDF]] (BioSemi data format0
* [[BKR]] (EEG data format)
* [[CFWB]] (Chart Data File Format)
* [[ DICOM-Waveform]] (An extension of Dicom for storing waveform data)
* [[ecgML]] (A markup language for electrocardiogram data acquisition and analysis)
* [[EDF/EDF+]] (European Data Format)
* [[FEF]] (File Exchange Format for Vital signs, CEN TS 14271)
* [[GDF v1.x]] (General Data Format for biomedical signals - Version 1.x)
* [[ GDF v2.x]] (The General Data Format for biomedical signals - Version 2.x)
* [[ HL7aECG]] (Health Level 7 v3 annotated ECG)
* [[OpenXDF]] (Open Exchange Data Format)
* [[SCP-ECG]] (Standard Communication Protocol for Computer assisted electrocardiography)
* [[SIGIF]] (A digital SIGnal Interchange Format)
* [[WFDB]] (Format of Physiobank)

== Chemical ==
* [[CCP4]] (X-ray crystallography voxels (electron density))
* [[CHM]] (ChemDraw file format)
* [[CIF]] (Crystallographic Information File, standardised by IUCr)
* [[CML]] (Chemical markup language)
* [[CTab]] (Chemical table file .mol, .sd, .sdf)
* [[HITRAN]] (spectroscopic data with one optical/infrared transition per line in the ASCII file (.hit))
* [[JCAMP]] (Joint Committee on Atomic and Molecular Physical Data, .dx, .jdx)
* [[MOL]] (MDL Molfile)
* [[MOP]] (MOPAC format)
* [[MRC]] (voxels in cryo-electron microscopy)
* [[PDB]] (Protein Data Bank)
* [[SDF]] (Structure Data File)
* [[SMILES]] (Simplified molecular input line entry specification, .smi)
* [[SPC]] (spectroscopic data)

Chemical data may be distinguished in various ways, including [http://www.ch.ic.ac.uk/chemime/ Chemical MIME] types.

== Ecological ==
* [[Darwin Core]] (Standard for sharing information about biological diversity)
* [[EML]] (Ecological Metadata Language)

== Geographic and Geospatial ==

See also [[Geospatial]]

* [[DEM]] (Digital Elevation Model)
* [[DOQ]] (Digital Orthophotos)
* [[e00]] (ESRI ArcInfo Interchange File)
* [[FGDC]] (Content Standard for Digital Geospatial Metadata??)
* [[GeoTIFF]] (Geospatial extensions to TIFF)
* [[GML]] (Geography Markup Language)
* [[HDFEOS, HD2, HD4]] (Hierarchical Data Format-Earth Observing System)
* [[KML]] (KML (formerly Keyhole Markup Language), Version 2.2)
* [[NDF]] (National Landsat Archive Production System (NLAPS) Data Format)
* [[SAIF]] (Spatial Archive and Interchange Format, Canadian)
* [[SDTS]] (Spatial Data Transfer Standard)
* [[shp and shx]] (ESRI Shaepfile must have components; other optional components as well, see entry)
* [[SID]] (MrSID- Multi-resolution Seamless Image Database)
* [[TAB]] (MapInfo dataset format, must have component)

== Mathematical ==
* [[graph6, sparse6]] (ASCII encoding of Adjacency matrices (.g6, .s6))
* [[M]] (Mathematica package file)
* [[MathML]]
* [[WP2]] WinPlot

== Medical Imaging ==
* [[AFNI]] (data, meta-data (.BRIK,.HEAD))
* [[MGH]] (uncompressed)
* [[MGZ]] (zip-compressed)
* [[Analyze data, meta-data]] (.img,.hdr)
* [[DICOM]] (Digital Imaging and Communications in Medicine (.dcm))
* [[MINC]] (Medical Imaging NetCDF format; since version 2.0, based on HDF5 (.mnc))
* [[OME-TIFF]] (Open Microscopy Imaging format)
* [[OME-XML]] (Open Microscopy Imaging format)
* [[OST (Open Spatio-Temporal)]] (extensible, open alternative for microscope images)
* [[nii]] (Neuroimaging Informatics Technology Initiative (NIfTI) single-file (combined data and meta-data))
* [[gii]] (NIfTI offspring for brain surface data, single-file (combined data and meta-data) style)
* [[.img,.hdr]] (NIfTI offspring for brain surface data, dual-file (separate data and meta-data, respectively) style)
* [[SDM]] (Signed Differential Mapping- brain maps(.sdm))

== Oceanographic, Atmospheric and Meteorological ==

* [[GRIB]] (Grid in Binary)
* [[BUFR]] (Binary Universal Format Representation)
* [[IOAPI]] (netCDF augmented with metadata from the I/O API)
* [[PP]] (UK Met Office format for weather model data)

== Physics ==

* [[CGNS]] (Computational Fluid Dynamics General Notation System)
* [[NeXuS]] (Common data format for neutron, x-ray and muon science)
* [[QCDml]] (Lattice QCD gauge configuration markup language)

== Scientific Signal data ==

* [[ACQ]] (AcqKnowledge File Format for Windows)
* [[BKR]] (EEG data format)
* [[BDF]] (BioSemi data format)
* [[CFWB]] (Chart Data File Format)
* [[EDF]] (European data format)
* [[FEF]] (File Exchange Format for Vital signs)
* [[GDF]] (General data formats for biomedical signals)
* [[GMS]] (Gesture And Motion Signal format)
* [[IROCK]] (intelliRock Sensor Data File Format)
* [[MFER]] (Medical waveform Format Encoding Rules)
* [[SCP-ECG]] (Standard Communication Protocol for Computer assisted electrocardiography)
* [[SEG Y]] (Reflection seismology data format)
* [[SIGIF]] (SIGnal Interchange Format)

== Social Sciences ==

* [[DDI]] (Data Documentation Initiative)
* [[SAS]] (Statistical package)
* [[SPSS]] (Statistical package)
* [[Stata]] (Statistical package)

Sources

2012-11-05T16:37:01Z

Mwoollard:

__NOTOC__
If you're looking for places to pull either file format information, or to verify stuff you've found, or to go that extra mile to track down variations, here's some similar endeavors and items.

==List of Places Keeping Track of Formats==

* [http://www.forensicswiki.org/wiki/Category:File_Formats Forensics Wiki]
* http://archivematica.org/preservation
* http://www.fileformat.info/
* http://www.ace.net.nz/tech/TechFileFormat.html
* http://wotsit.org/ - their download page is broken. need to ping webmaster about that.
* Wikipedia: http://en.wikipedia.org/wiki/List_of_file_formats and http://en.wikipedia.org/wiki/List_of_file_formats_%28alphabetical%29
** Also [[wikipedia:Template:Infobox file format]] ([http://en.wikipedia.org/w/index.php?title=Special%3AWhatLinksHere&limit=500&target=Template%3AInfobox+file+format&namespace=0 pages using this template])
* http://unarchiver.c3.cx/formats
* [[Sources/PRONOM|The PRONOM format registry]].
* http://wiki.multimedia.cx/
* http://archive.org/download/file-format-encyclopedia/fileformat.zip/
* http://www.digitalpreservation.gov/formats/fdd/descriptions.shtml
* http://www.udfr.org/, see also http://gdfr.info/.
* http://gitorious.org/re-lab Tools & specs from re-lab- graphics formats, and office formats
* http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/index.html - Documentation, details, specs, and samples for a handful of audio formats
* http://wiki.xentax.com/index.php/Game_File_Format_Central This wiki is the home of the most game (archive) file format knowledge in the world.
* http://rewiki.regengedanken.de has some more game related reverse engineering
* http://mark0.net/soft-trid-e.html Home to TrID, which can identify a vast number of file formats using an updateable database
* http://mark0.net/soft-trid-deflist.html Online searchable version of the TrID database file
* http://file-extension.net/seeker/ Metasearch engine for file extensions - Find info about unknown file extensions or filetypes
* http://www.vgmpf.com/Wiki/index.php?title=Main_Page - the Videogame Music Preservation Foundation Wiki has information on a number of formats used to play back videogame music
* http://hackipedia.org/File%20formats/ - loads of info on various formats here
* http://wiki.mobileread.com/wiki/E-book_formats - detailed information on eBook formats.
* http://mediapedia.nla.gov.au/ - "The Mediapedia resource is intended to enable the identification of various physical media carrier types for assisting with collection planning, assessment, documentation, infrastructure and preservation planning for the content they hold. These could include media across various genres such as cine, video, photo, audio, data, paper carriers, microfilm, etc."
* http://wiki.opf-labs.org/display/TR/Formats - Small list of formats with detailed descriptions of some known access/preservation problems.
* [http://registry.dcc.ac.uk:8080/RegistryWeb/Registry/ CASPAR/DCC Representation Information Repository]
* [http://theswo.sourceforge.net/ The Software Ontology]
* [http://keep-totem.co.uk/ KEEP Totem] also covers software, hardware etc.
* [http://cultureandcommunication.org/deadmedia/index.php/Main_Page Dead Media Archives]
* [http://www.forensicinnovations.com/formats.html The >4000 formats known to FI Tools], [http://www.forensicinnovations.com/formats-ext1.html by extension], [http://www.forensicinnovations.com/formats-mime.html by MIME type]
* [http://hg.python.org/cpython/file/2.7/Lib/mimetypes.py#l402 Python sourcecode for mimetypes module]
* /etc/mimetypes on *nix systems defines a number of file formats and [[mimetypes]]
* http://www.iasa-web.org/tc04/key-digital-principles - Recommended Audio formats by Audiovisual Archives and Institutions
* http://www.gpsbabel.org/capabilities.html - List of formats used by GPS devices
* http://www.microsoft.com/openspecifications/en/us/programs/osp/office-file-formats/default.aspx Microsoft's Open Document Specifications
* http://mamedev.org/source/src/lib/formats/index.html http://mamedev.org/source/src/mess/formats/index.html - format handlers for disk images, cassette images, and memory snapshots/quickloads in the MESS source code
* http://minidisc.org alot of minidisc related info
* http://cool.conservation-us.org/ Conservation Online
* http://filext.com/ File Extensions with Mime types

==Emulation==

* http://nonmess.retrogames.com/ - Emulator listing for various computer formats, including MESS and single-machine emulators.
* [http://emuframework.sourceforge.net/ The Emulation Framework] is software developed by the international KEEP project, co-funded by the European Union's 7th Framework Programme.

==Books==

* [http://www.amazon.com/Compressed-Image-File-Formats-JPEG/dp/0201604434/ Compressed Image File Formats: JPEG, PNG, GIF, XBM, BMP] by John Miano (1999)
* [http://www.amazon.com/Encyclopedia-Graphics-File-Formats-Reference/dp/1565921615/ Encyclopedia of Graphics File Formats: The Complete Reference on CD-ROM with Links to Internet Resources] by James D. Murray and William VanRyper (1996) '''Copy Acquired!'''
* The CD-ROM for the above book is located [http://archive.org/details/EncyclopediaOfGraphicsFileFormatsCompanionCd-rom here].
* [http://www.amazon.com/Graphics-File-Formats-David-Kay/dp/0070340250/ Graphics File Formats] by David C. Kay and John R. Levine (1994)
* [http://www.amazon.com/Graphics-File-Formats-Reference-Guide/dp/0133034054/ Graphics File Formats: Reference and Guide] by C. Wayne Brown, Barry J. Shepherd (1994)
* [http://www.amazon.com/Formats-Popular-Personal-Computer-Software/dp/0471836710/ File Formats for Popular Personal Computer Software: A Programmer's Reference (Self-teaching Guides)] by Jeff Walden (1986) '''Copy Acquired!'''
* [http://www.amazon.com/File-Formats-Internet-Guide-Users/dp/0871114410 File Formats on the Internet: A Guide for PC Users] by Allison B. Zhang (1996) '''Copy Acquired!'''
* [http://www.amazon.com/Graphics-File-Formats-Programmers-Reference/dp/0201488353/ 3D Graphics File Formats: A Programmer's Reference] by Keith Rule (1996)
* [http://www.amazon.com/More-File-Formats-Popular-Software/dp/0471850772/ More File Formats for Popular PC Software: A Programmer's Reference] by Jeff Walden (1987) '''Copy Acquired!'''
* [http://www.amazon.com/Lotus-File-Formats-1-2-3-Symphony/dp/0201168243/ Lotus File Formats for 1-2-3 Symphony & Jazz: File Structure Descriptions for Developers] from Lotus, Inc. (1986)
* [http://www.amazon.com/Inside-Windows-File-Formats-Swan/dp/0672303388/ Inside Windows File Formats] by Tom Swan (Nov 1993)
* [http://www.amazon.com/Internet-File-Formats-Complete-Receiving/dp/188357756X/ Internet File Formats: Your Complete Resource for Sending, Receiving, and Using Internet Files] by Tim Kientzle (Oct 15, 1995)
* [http://www.amazon.com/File-Formats-Popular-PC-Software/dp/0471528064/ File Formats for Popular PC Software] by Jeff Walden (Nov 1992)
* [http://www.amazon.com/File-Formats-Conversions-Book-Developers/dp/155755059X/ PC File Formats and Conversions/Book and Disk (Developers Series)] by Ralf Kussmann (Mar 1990)
* [http://www.amazon.com/Windows-Undocumented-Formats-Working-Inside/dp/0879304375/ Windows Undocumented File Formats; Working Inside 16- and 32- bit Windows] by Pete Davis (Jan 23, 1997)
* [http://www.amazon.com/Graphics-File-Formats-Complete-Reference/dp/0070375321/ Graphics File Formats: The Complete Reference] by John R. Levine and David C. Kay (May 1992)
* [http://www.amazon.com/Using-Pcx-Graphics-Files-Programmers/dp/0879304324/ Using Pcx Graphics Files: The Programmer's Definitive Guide to Pcx File Formats] by Roger T. Stevens (Apr 1, 1996)
* [http://www.amazon.com/File-Formats-Handbook-Gunter-Born/dp/1850321175/ The File Formats Handbook] by Gunter Born (Sep 1995)
* [http://www.amazon.com/Window-System-Formats-Application-Conventions/dp/1872630154/ X Window System File Formats and Application Conventions (X11R4)] by The Open Group (Jul 17, 1991)
* [http://www.iasa-web.org/audio-preservation-tc04 Guidelines on the Production and Preservation of Digital Audio Objects] by IASA (2009)

==Other Useful Materials and Services==

* http://www.nsrl.nist.gov/ National Software Reference Library (see also http://blogs.loc.gov/digitalpreservation/2012/05/life-saving-the-national-software-reference-library/)
* http://www.openplanetsfoundation.org/software/fido Format Identification for Digital Objects
* https://github.com/usnationalarchives/File-Analyzer File Analyzer
* http://sk1project.org/modules.php?name=products&product=uniconvertor UniConvertor is a universal vector graphics translator. It is a command line tool which uses sK1 object model to convert one format to another
* http://www.textfiles.com/programming/FORMATS/
* http://wiki.opf-labs.org/display/SPR/Digital+Preservation+Tools Wiki list of digital preservation tool lists
* http://wiki.opf-labs.org/display/REQ/Digital+Preservation+and+Data+Curation+Requirements+and+Solutions Digital Preservation and Data Curation Requirements and Solutions. Not classified by formats, but by datasets, issues and solutions.
* http://fileformats.wordpress.com/ - The File Formats Blog
* http://www.archive.org/details/cdbbsarchive - The Internet Archive has over 1,500 CD-ROMs loaded with shareware, and with that comes converters, documentation and other vital works related to file formats.
* [http://wiki.opf-labs.org/display/SPR/Crowd+sourced+Representation+Information+for+Supporting+Preservation+(cRIsp) Crowd sourced Representation Information for Supporting Preservation (cRIsp)]
* [http://www.nsrl.nist.gov/ National Software Reference Library]
* [http://mediatypes.appspot.com/ Linked Data URIs for MIME Types]
* http://darwinsys.com/file/ UNIX file(1) format guessing utility