Data File Types and Formats

File formats are the method by which data is organized and stored in a computer file. A file format standard is an agreed upon method for organizing data within a file. Standard formats exist for many data types, including images, audio and video files, remote sensing, CAD and GIS data, Using an agreed upon standard can simplify the exchange of information as tools or applications can be developed to use the common format. For example JPEG image files can be viewed and even edited with the correct software on just about any device. Standards can also slow adoption of newer technologies, as both JPEG 2000 and HEIC image files have improved compression, metadata handling and security over the older JPEG standard but are still not as widely adopted making their use more cumbersome.

File storage types

Here we discuss that major file storage types. These are not specific formats, so do not control how data is structured in the file, but instead discuss the primary method used to store and interpret data on a computer system. Specifically, storing data as plain text, binary, hybrid or compressed.

Plain Text Files

Everything on a computer is at its core a binary number, since computers do everything with bits that represent 0 and 1. In order to have a file that is "plain text", so human readable with minimal intervention, binary values must be mapped to specific letters. This process is typically called encoding. Almost everything is now encoded using UNICODE, but there are still some older files and documentation that refer to older standards, especially ASCII which was dominant in the United States and therefore throughout the world for many decades. The most common UNICODE system, UTF-8 is backwards compatible with ASCII, meaning that many people will still refer to ASCII encoding even when a file is UTF-8.

Here are some additional details on plain text encoding systems:

American Standard Code for Information Interchange (ASCII)

• An ASCII text file is one in which each byte represents one character according to the ASCII code.
• ASCII files are human readable and are sometimes called plain text files though in reality they are binary files with a standard interpretation.
• For an example of the ASCII table definition, refer to this version of the ASCII table, which illustrates how binary numbers are assigned to standard English language characters.
• ASCII files are relatively simple to structure and do not require special tools to read so they are commonly used for data storage.
• Data corruption is often easier to identify and recover from since the file should be readable using many common text editing and viewing tools.
• ASCII files have low entropy - information stored in an ASCII file typically occupies more storage than is strictly necessary.

UNICODE

The Unicode Consortium is the standards body for the internationalization of software and services. Before the Unicode standard was developed, there were many different systems, called character encodings (such as ASCII), for assigning these numbers. These earlier character encodings were limited and did not cover characters for all the world’s languages. Even for a single language like English, no single encoding covered all the letters, punctuation, and technical symbols in common use. Pictographic languages, such as Japanese, were a challenge to support with these earlier encoding standards.

Early character encodings also conflicted with one another. That is, two encodings could use the same number for two different characters, or use different numbers for the same character. Any given computer might have to support many different encodings. However, when data is passed between computers and different encodings it increased the risk of data corruption or errors.

The two primary Unicode encodings you are likely to work with are UTF-8 and UTF-16:

• UTF-8 (Unicode Transformation Format – 8-bit). UTF-8 is capable of encoding all 1,112,064 valid Unicode scalar values using a variable-width encoding of one to four one-byte (8-bit) code units. It was designed for backward compatibility with ASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file. Most software designed for any extended ASCII can read and write UTF-8 (including on Microsoft Windows) and this results in fewer internationalization issues than any alternative text encoding.

• UTF-16 (16-bit Unicode Transformation Format). UTF-16 encoding is variable-length as code points are encoded with one or two 16-bit code units and is also capable of encoding all Unicode scalar values. UTF-16 is used by the Windows API, and therefore by many programming environments such as the Java programming language and JavaScript/ECMAScript. It is also sometimes used for plain text and word-processing data files. UTF-16 is the only encoding (still) allowed on the web that is incompatible with 8-bit ASCII, but its adoption has been very limited and it is considered less secure than UTF-8 for web applications.

Binary Files

All computers store information in binary form, specifically as bits (binary digits). Eight bits make up one byte. As everything stored on a computer is technically a file containing binary data, we will use the term "binary file" to mean a file that is mostly computer but not human readable. Many files contain a combination of binary and text formats. Such files are usually considered to be binary. For example, files that have been formatted with a word processor may encode written characters using a plain text format (UTF-8 or UTF-16 now), but encode additional information such as formatting instructions and embedded images using binary coding. Such mixed files should be treated as binary to preserve the original data when, for example, transmitting the files between computer systems using FTP or other file transfer protocols.

Binary formats are most commonly used when space or computer efficiency are most valued. Since they are are stored in a way convenient for the machine, read and write operations are very efficient. Binary files are also the only way to store exact values from the computer, since writing a binary number using decimal digits will require the loss of precision for many floating point numbers. Binary files are also typically more compact than plain text files as information (specifically numbers) occupy storage more in line with what they actually require.

Hybrid files

Many data file types are more of a hybrid, with both plain text and binary components. One example of this that you are likely familiar with are Adobe PDF files. The PDF (Portable Document File) is often a combination of vector graphics, text, and bitmap graphics. The basic types of content in a PDF are:

• Typeset text stored as content streams (i.e., not encoded in plain text);
• Vector graphics for illustrations and designs that consist of shapes and lines;
• Raster graphics for photographs and other types of images; and
• Other multimedia objects.

The PDF is organized using primarily plain text, though some objects may have binary content. For example, typeset text and vector graphics along with document metadata are typically encoded using plain text, while raster images and other multimedia objects are more likely encoded using embedded binary information.

Plain text is generally easier to port between systems, since since single bytes do not suffer from issues of endianness and are more easily read and interpreted by software packages as nothing is hidden. Binary data provides efficiencies in loading data and in reducing file sizes. The hybrid approach tries to maximize the benefits of both without taking on too many of the negatives.

Compressed and archive files

Data compression is the process of encoding information using fewer bits than required by the original representation. Archiving usually involves the bundling of multiple files and folders into a single file that is easier to store and share. Compression and archiving files are presented here as a file type because they typically have to be uncompressed or unarchived before the data can be access. Compression and archive files are an intermediate point between data files that start and end as one of the previously describe file types.

A device or code that performs data compression is referred to as an encoder, and one that performs the reversal of the process (decompression) as a decoder. Data compression algorithms present a space–time complexity trade-off between the bytes needed to store or transmit information, and the computational resources needed to perform the encoding and decoding. The design of data compression schemes involves balancing the degree of compression, the amount of distortion introduced (when using lossy data compression), and the computational resources or time required to compress and decompress the data.

Compression is either lossy or lossless.

• Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression.
• Lossy compression reduces bits by removing unnecessary or less important information.

Lossless compression is more likely to be used for large data files were loss of data is deemed unacceptable. The performance of lossless compression algorithms will vary a lot based on the statistical similarity / redundancy of the source dataset. A dataset with a lot of repeating or redundant information, such as a lot of No Data values, will be compressed much smaller than a dataset that is more densely filled with non-repeating data.

Lossy data is most often used where the impact of lost data is least likely to be noticed. For example, in digital photos or audio. There the compression algorithm is based on the idea that human senses will compensate for missing data, so compression of a photo can reduce the size of the photo file by reducing the details within the photo, while still capturing the primary sense of the image. For example, saving the details within a brick in a brick wall is less important than preserving a shapes of the bricks as the sense of there being a brick wall will be preserved. Lossy compression will result in similar reductions in file size based on the compression level and not the data contents. However, high levels of compression will become visible in the rendering as too much detail is lost or distorted. Additionally, as data is lost in the compression process, every subsequent decompression will lose more data.

An archive files can be compressed or uncompressed. The Tape ARchive or "tar" file is an early and still relevant archive file format. it will bundle any number of files and folders into a single object. A tar file is uncompressed, but can be compressed during creation, say by running the process through the gzip encoder to create a "tgz" file. Or by compressing the resulting tar file using a separate encoder in a two step process.

Many modern encoders can also compress directories and their contents, which blurs the line between archiving and compressing data files.

Major types of file formats

These lists pull from multiple sources on the internet, including Geeks for Geeks List of File Formats, File Format Basics from the Ohio State University, various AI generated summaries, and are then curated by Dr. Cherkauer to focus on file formats you are most likely to run into as part of your research. It is not designed to be comprehensive, and is likely already out of date.

Plain text file formats

Here are some examples of plain text file formats that you might encounter.

Image files

Image file formats determine how pictures are saved and shown. Choosing the right format affects the file size, quality, and how well it works with different devices, especially for photography, websites, and design. Images can be stored as raster, or grids of pixels with specific properties, or as vectors where information is encoded as to how objects within the image should be drawn. Raster formats are most common for photos but suffer from degradation as they are enlarged. Vector formats are preferred when images are sparse but require sharp edges, such as plots and schematics. As the relative size of fonts and drawing objects scale with the figure and are redrawn before printing, vector graphics can be resized without much concern for losing content.

Raster image formats

Vector image formats

Audio files

Video files

Program files

GIS files

Vector Formats

Raster Formats

Meta data and other Formats

Geodatabases (Hybrid/Comprehensive)

Remote sensing data files

Remote sensing imagery will use many of the standard GIS data formats, but here are some additional formats used primarily in the remote sensing community and not as broadly in spatial sciences.

Imagery or raster files

Other remote sensing data

Metadata files

Scientific data formats

Descriptions of Common Scientific Data Formats. Note that these are generally self-describing (so metadata is contained within the file), with some caveats. Additionally, there are typically tools developed to work with and explore data in these formats developed for specific communities (climate, weather, oceanography, remote sensing), which are not explored here but may be worth an extra look if you are going to make regular use of data in these formats.

• NASA Earth Observation data formats is a little heavy on satellite and radar specific forms, but does cover a lot of the basics.
• Climate Data Analysis Tools and Methods provides a summary overview of tools and formats used for climate analysis supported by NSF and NCAR
  • Here is a more direct link to the Common Climate Data Formats: Overview page.
• This page from the University of Chicago gets into Choosing a Data Format, Naming Data Files and Versioning.
• Wikipedia's Take on Data Formats is mostly a very comprehensive list with links to descriptive pages,

Links to Specific Data Formats

• NetCDF home page: http://www.unidata.ucar.edu/software/netcdf/
• HDF home page: http://hdf.ncsa.uiuc.edu/
• GRIB (GRIdded Binary or General Regularly-distributed Information in Binary form): Wikipedia overview

Tutorials for Working with Common Scientific Formats in Python

• General guidance for reading multiple data file format types with Python
• Flat ASCII files
  • Working with Built-in File Methods
  • Official pandas documentation for working with ASCII files
• NetCDF files
  • Official SciPy documentation on NetCDF files
• HDF5 files (HDF and HDF5 files are not the same thing)
  • A quick primer for working with HDF5 files using the pandas module in Python
  • Official pandas documentation for HDF functions