Archive formats and utilities

An archive, is a collection of something in a container. In our use the things are files and their associated meta data. The container can be punch tape, floppy disk, tape, or another file.

An archive tool manipulates the container, while the format specifies how data is stored in the container.

The two key early program were CPIO (CoPy In Out) and TAR. CPIO is mostly used in a pipeline. Where it is fed a list of file names on its input, reads their meta data and contents from storage, and outputs them as an archive to its output.

cat MyListOfFile | cpio -ocB > /dev/rmt0

Tar on the other hand, usually takes the name of a file in which to create the archive and a list of directories to include on its command line.

tar -c -f MyArchive.tar ./book1/ ./book2/

CPIO also provided a copy mode where the target is a directory rather than a device, and the list of files presented on its input is copied to the new directory. In this case no archive is created so archive format restrictions do not apply. Limits are those of the system on which it is run.

cat MyListOfFile | cpio -pdum ../NewDirectory 

One major issue these days is format limits. The limit on the size of files to be include, the size of the archive to be created, and what can be included as names. The effective limits are always the lesser of the format limits, the operating system limits, and the resource limits of the devices involved.

Back in the mid 1970s with only a small number of computers counting their total disk storage in Megabytes, many had only 160 KB stored on a floppy. An archive format that allowed for files 2,000 times the size of the then available storage devices was never seen as a constraint.

As the techniques used to make silicon chips were applied to drive heads, the head size shrank, and data density rose. Some of the increase in data density was traded off for smaller drives.

• In 1990 the commonest drive was 5.25 inch 5 MB.
• In 2000 the commonest drive was 3.5 inch 5 GB.
• In 2010 the commonest drive was 2.5 inch 32 GB.
• In 2020 the commonest drive was 2.5 inch 1 TB.

At the data centre end of the market a server in a rack with 100 8 TB drives is perfectly practical.

Archive formats, and the meta data they can store has had to change.

Side note on prefixes indicating size

Computing mostly operated in binary and 2¹⁰ = 1024, so in the early days people working with computers tended to ignore the 2.4% difference between 1000 and 1024, and borrow the SI prefixes.

Système International d'Unités - SI - International system of units

Back in 1960 the scientific community standardized a consistent set of units, for various physical properties, and a set of prefixes for when you were dealing with large numbers of, or small fractions of the unit.

• https://en.wikipedia.org/wiki/International_System_of_Units
• http://physics.nist.gov/cuu/Units/prefixes.html
• http://physics.nist.gov/cuu/Units/rules.html
• http://www.npl.co.uk/reference/measurement-units/si-base-units/

IEC

By the time you reach Gigabyte quantities the difference between 10⁹, and 2³⁰, reaches 7.4%. Drive manufacturers were using GB and TB to mean 10⁹, and 10¹². While memory manufacturers were using them for 2³⁰, and 2⁴⁰

IEC responded to the growing issue by standardising a new set of prefixes

History of archive standards.

The POSIX 2 standard in 1992 replaced cpio with pax. Pax keeps the pipeline and copy modes of cpio, but also support ACL's and extended attributes. It introduced its own archive format, with support for much larger files, and a large enough number of user ids, for a system with multiple use accounts for everyone in the world. For backwards compatibility and portability it could read and write in selected cpio and tar formats.

History of standardization attempts.

When the developers wrote there first archive program in 1977, they thought ahead. In 1977 a large computer could have a whole MB of storage, on a platter about the size of an LP. They designed a format that would work on a system with 65,000 users, creating files 2,000 times the size of the largest disk then available.

Tar was developed independently, a couple of years later with broadly similar capabilities. Give a problem to several independent sets of developers, and they will generally come up with similar solutions.

It was mostly the bits that had not been envisaged, that cause issues over the following 20 years.

• longer file names
• multi level directories
• Extended character sets and non ASCII file names
• Access Control Lists
• Xattr (eXtended attributes)
• Security context
• ...

By the late 1990s we had several dozen propitiatory modifications, for various systems that were mostly incompatible. Archive programs would by default generate archives in the propitiatory format. Which caused lots of problems when you want to access the archive from a different computer to the one that created it. Most had the ability to read, and some times write, one or more of,

• ASCII CPIO standard from 1980
• POSIX CPIO standard from 1988
• POSIX ustar standard from 1988

When doing this any thing that did not fit the target standard was dropped, or the file creation failed.

The revised POSIX standard in 2001 introduced PAX. As an extensible archive format. The originator may add records to the archive describing addition attributes that had not been thought of in 2001, The receiver can process those attributes it recognises and ignore the rest.

Summary format list. Note codes are used as column headings in the next table. As a general complication, different programs, or in some cases different versions of the same program, use the same format code, for different formats. :-(

OS tool Compatibility and support table.

This is not the whole story. With regard to some formats notable the POSIX standard from 2001, the standard provided an extension mechanism for additional attributes or meta data. Currently this includes access control lists, extended attributes, SeLinux contexts, ...

Just because a particular tool can read or write a format does not mean it supports, all of the extensions allowed in the format. When creating an archive a tool can be asked to write those attributes it knows about to the archive. When restoring they mainly just ignore any attributes recorded in the archive that they do not understand.

For transferring or archiving data files, extended attributes are unlikely to matter very much. However for backing up an operating system, that you expect to be able to restore and run they do.

Applications

Gnu tar

yum install tar

Feature set, option names, and archive format have changed a lot over the years.

1999 version 1.13 new archive format

2003 version 1.14 new archive formats, support for sparce files

2004 version 1.15 new archive format, supporting acl, xattr, selinux

2007 version 1.17 rework option names....

star (ustar)

The package star provides the commands /usr/bin/star, and /usr/bin/ustar

yum install star

The star command suports creating and reading a wide range of archive formats. In all cases the restrictions of the format applies.

When invoked as ustar the default output format is ustar, the restrictions of that format apply. So maximum file size 8 GB, no ACLs, or extended attributes.

When invoked as star the default output format is xstar. So maximum file size 8 GB, no ACLs, or extended attributes.

Specifying pax as the output format gives a file that complies with tha standard with out any extended headers.

Sparse files are not supported in tar, ustar, suntar, pax, or any cpio variant. I think that leaves star, gnutar, xstart, xustar, and exustar.

ACL access control lists are only supported for the format exustar.

Extended attributes are only supported when the format is exustar.

spax (pax)

The package spax provides the commands /usr/bin/spax, and /usr/bin/pax

yum install star

opax

Provides the commands /usr/bin/opax, and /usr/bin/pax.

yum install pax

Confusingly this version of the pax command does to support the 2001 stadardized Portable Archive eXchange format (PAX). It does however support additional formats

• bcpio The old binary cpio format. The default blocksize for this format is 5120 bytes. This format is not very portable and should not be used when other formats are available. Inode and device information about a file (used for detecting file hard links by this format) which may be truncated by this format is detected by opax and is repaired.
• sv4cpio The System V release 4 cpio. The default blocksize for this format is 5120 bytes. Inode and device information about a file (used for detecting file hard links by this format) which may be truncated by this format is detected by opax and is repaired.
• sv4crc The System V release 4 cpio with file crc checksums. The default blocksize for this format is 5120 bytes. Inode and device information about a file (used for detecting file hard links by this format) which may be truncated by this format is detected by opax and is repaired.
• tar The old BSD tar format as found in BSD4.3. The default blocksize for this format is 10240 bytes. Pathnames stored by this format must be 100 characters or less in length (including the trailing character, which means that filenames can have a maximum length of 99 characters). Only regular files, hard links, soft links, and directories will be archived (other file system types are not supported). For backwards compatibility with even older tar formats, a -o option can be used when writing an archive to omit the storage of directories. This option takes the form: -o write_opt=nodir

Port from BSD to SuSE release 2001 as version 3.0, 3.4 came in 2005

bsdtar

yum install bsdtar