File: coreutils.info,  Node: dd invocation,  Next: install invocation,  Prev: cp invocation,  Up: Basic operations

11.2 ‘dd’: Convert and copy a file
==================================

‘dd’ copies input to output with a changeable I/O block size, while
optionally performing conversions on the data.  Synopses:

     dd [OPERAND]...
     dd OPTION

   The only options are ‘--help’ and ‘--version’.  *Note Common
options::.

   By default, ‘dd’ copies standard input to standard output.  To copy,
‘dd’ repeatedly does the following steps in order:

  1. Read an input block.

  2. If converting via ‘sync’, pad as needed to meet the input block
     size.  Pad with spaces if converting via ‘block’ or ‘unblock’, NUL
     bytes otherwise.

  3. If ‘bs=’ is given and no conversion mentioned in steps (4) or (5)
     is given, output the data as a single block and skip all remaining
     steps.

  4. If the ‘swab’ conversion is given, swap each pair of input bytes.
     If the input data length is odd, preserve the last input byte
     (since there is nothing to swap it with).

  5. If any of the conversions ‘swab’, ‘block’, ‘unblock’, ‘lcase’,
     ‘ucase’, ‘ascii’, ‘ebcdic’ and ‘ibm’ are given, do these
     conversions.  These conversions operate independently of input
     blocking, and might deal with records that span block boundaries.

  6. Aggregate the resulting data into output blocks of the specified
     size, and output each output block in turn.  Do not pad the last
     output block; it can be shorter than usual.

   ‘dd’ accepts the following operands, whose syntax was inspired by the
DD (data definition) statement of OS/360 JCL.

‘if=FILE’
     Read from FILE instead of standard input.

‘of=FILE’
     Write to FILE instead of standard output.  Unless ‘conv=notrunc’ is
     given, truncate FILE before writing it.

‘ibs=BYTES’
     Set the input block size to BYTES.  This makes ‘dd’ read BYTES per
     block.  The default is 512 bytes.

‘obs=BYTES’
     Set the output block size to BYTES.  This makes ‘dd’ write BYTES
     per block.  The default is 512 bytes.

‘bs=BYTES’
     Set both input and output block sizes to BYTES.  This makes ‘dd’
     read and write BYTES per block, overriding any ‘ibs’ and ‘obs’
     settings.  In addition, if no data-transforming ‘conv’ operand is
     specified, input is copied to the output as soon as it’s read, even
     if it is smaller than the block size.

‘cbs=BYTES’
     Set the conversion block size to BYTES.  When converting
     variable-length records to fixed-length ones (‘conv=block’) or the
     reverse (‘conv=unblock’), use BYTES as the fixed record length.

‘skip=N’
‘iseek=N’
     Skip N ‘ibs’-byte blocks in the input file before copying.  If N
     ends in the letter ‘B’, interpret N as a byte count rather than a
     block count.  (‘B’ and the ‘iseek=’ spelling are GNU extensions to
     POSIX.)

‘seek=N’
‘oseek=N’
     Skip N ‘obs’-byte blocks in the output file before truncating or
     copying.  If N ends in the letter ‘B’, interpret N as a byte count
     rather than a block count.  (‘B’ and the ‘oseek=’ spelling are GNU
     extensions to POSIX.)

‘count=N’
     Copy N ‘ibs’-byte blocks from the input file, instead of everything
     until the end of the file.  If N ends in the letter ‘B’, interpret
     N as a byte count rather than a block count; this is a GNU
     extension to POSIX. If short reads occur, as could be the case when
     reading from a pipe for example, ‘iflag=fullblock’ ensures that
     ‘count=’ counts complete input blocks rather than input read
     operations.  As an extension to POSIX, ‘count=0’ copies zero blocks
     instead of copying all blocks.

‘status=LEVEL’
     Specify the amount of information printed.  If this operand is
     given multiple times, the last one takes precedence.  The LEVEL
     value can be one of the following:

     ‘none’
          Do not print any informational or warning messages to standard
          error.  Error messages are output as normal.

     ‘noxfer’
          Do not print the final transfer rate and volume statistics
          that normally make up the last status line.

     ‘progress’
          Print the transfer rate and volume statistics on standard
          error, when processing each input block.  Statistics are
          output on a single line at most once every second, but updates
          can be delayed when waiting on I/O.

     Transfer information is normally output to standard error upon
     receipt of the ‘INFO’ signal or when ‘dd’ exits, and defaults to
     the following form in the C locale:

          7287+1 records in
          116608+0 records out
          59703296 bytes (60 MB, 57 MiB) copied, 0.0427974 s, 1.4 GB/s

     The notation ‘W+P’ stands for W whole blocks and P partial blocks.
     A partial block occurs when a read or write operation succeeds but
     transfers less data than the block size.  An additional line like
     ‘1 truncated record’ or ‘10 truncated records’ is output after the
     ‘records out’ line if ‘conv=block’ processing truncated one or more
     input records.

     The ‘status=’ operand is a GNU extension to POSIX.

‘conv=CONVERSION[,CONVERSION]...’
     Convert the file as specified by the CONVERSION argument(s).  (No
     spaces around any comma(s).)

     Conversions:

     ‘ascii’
          Convert EBCDIC to ASCII, using the conversion table specified
          by POSIX.  This provides a 1:1 translation for all 256 bytes.
          This implies ‘conv=unblock’; input is converted to ASCII
          before trailing spaces are deleted.

     ‘ebcdic’
          Convert ASCII to EBCDIC.  This is the inverse of the ‘ascii’
          conversion.  This implies ‘conv=block’; trailing spaces are
          added before being converted to EBCDIC.

     ‘ibm’
          This acts like ‘conv=ebcdic’, except it uses the alternate
          conversion table specified by POSIX.  This is not a 1:1
          translation, but reflects common historical practice for ‘~’,
          ‘[’, and ‘]’.

          The ‘ascii’, ‘ebcdic’, and ‘ibm’ conversions are mutually
          exclusive.  If you use any of these conversions, you should
          also use the ‘cbs=’ operand.

     ‘block’
          For each line in the input, output ‘cbs’ bytes, replacing the
          input newline with a space and truncating or padding input
          lines with spaces as necessary.

     ‘unblock’
          Remove any trailing spaces in each ‘cbs’-sized input block,
          and append a newline.

          The ‘block’ and ‘unblock’ conversions are mutually exclusive.
          If you use either of these conversions, you should also use
          the ‘cbs=’ operand.

     ‘lcase’
          Change uppercase letters to lowercase.

     ‘ucase’
          Change lowercase letters to uppercase.

          The ‘lcase’ and ‘ucase’ conversions are mutually exclusive.

     ‘sparse’
          Try to seek rather than write NUL output blocks.  On a file
          system that supports sparse files, this will create sparse
          output when extending the output file.  Be careful when using
          this conversion in conjunction with ‘conv=notrunc’ or
          ‘oflag=append’.  With ‘conv=notrunc’, existing data in the
          output file corresponding to NUL blocks from the input, will
          be untouched.  With ‘oflag=append’ the seeks performed will be
          ineffective.  Similarly, when the output is a device rather
          than a file, NUL input blocks are not copied, and therefore
          this conversion is most useful with virtual or pre zeroed
          devices.

          The ‘sparse’ conversion is a GNU extension to POSIX.

     ‘swab’
          Swap every pair of input bytes.

     ‘sync’
          Pad every input block to size of ‘ibs’ with trailing zero
          bytes.  When used with ‘block’ or ‘unblock’, pad with spaces
          instead of zero bytes.

     The following “conversions” are really file flags and don’t affect
     internal processing:

     ‘excl’
          Fail if the output file already exists; ‘dd’ must create the
          output file itself.

     ‘nocreat’
          Do not create the output file; the output file must already
          exist.

          The ‘excl’ and ‘nocreat’ conversions are mutually exclusive,
          and are GNU extensions to POSIX.

     ‘notrunc’
          Do not truncate the output file.

     ‘noerror’
          Continue after read errors.

     ‘fdatasync’
          Synchronize output data just before finishing, even if there
          were write errors.  This forces a physical write of output
          data, so that even if power is lost the output data will be
          preserved.  If neither this nor ‘fsync’ are specified, output
          is treated as usual with file systems, i.e., output data and
          metadata may be cached in primary memory for some time before
          the operating system physically writes it, and thus output
          data and metadata may be lost if power is lost.  *Note sync
          invocation::.  This conversion is a GNU extension to POSIX.

     ‘fsync’
          Synchronize output data and metadata just before finishing,
          even if there were write errors.  This acts like ‘fdatasync’
          except it also preserves output metadata, such as the
          last-modified time of the output file; for this reason it may
          be a bit slower than ‘fdatasync’ although the performance
          difference is typically insignificant for ‘dd’.  This
          conversion is a GNU extension to POSIX.

‘iflag=FLAG[,FLAG]...’
     Access the input file using the flags specified by the FLAG
     argument(s).  (No spaces around any comma(s).)

‘oflag=FLAG[,FLAG]...’
     Access the output file using the flags specified by the FLAG
     argument(s).  (No spaces around any comma(s).)

     Here are the flags.

     ‘append’
          Write in append mode, so that even if some other process is
          writing to this file, every ‘dd’ write will append to the
          current contents of the file.  This flag makes sense only for
          output.  If you combine this flag with the ‘of=FILE’ operand,
          you should also specify ‘conv=notrunc’ unless you want the
          output file to be truncated before being appended to.

     ‘cio’
          Use concurrent I/O mode for data.  This mode performs direct
          I/O and drops the POSIX requirement to serialize all I/O to
          the same file.  A file cannot be opened in CIO mode and with a
          standard open at the same time.

     ‘direct’
          Use direct I/O for data, avoiding the buffer cache.  Note that
          the kernel may impose restrictions on read or write buffer
          sizes.  For example, with an ext4 destination file system and
          a Linux-based kernel, using ‘oflag=direct’ will cause writes
          to fail with ‘EINVAL’ if the output buffer size is not a
          multiple of 512.

     ‘directory’

          Fail unless the file is a directory.  Most operating systems
          do not allow I/O to a directory, so this flag has limited
          utility.

     ‘dsync’
          Use synchronized I/O for data.  For the output file, this
          forces a physical write of output data on each write.  For the
          input file, this flag can matter when reading from a remote
          file that has been written to synchronously by some other
          process.  Metadata (e.g., last-access and last-modified time)
          is not necessarily synchronized.

     ‘sync’
          Use synchronized I/O for both data and metadata.

     ‘nocache’
          Request to discard the system data cache for a file.  When
          count=0 all cached data for the file is specified, otherwise
          the cache is dropped for the processed portion of the file.
          Also when count=0, failure to discard the cache is diagnosed
          and reflected in the exit status.

          Note data that is not already persisted to storage will not be
          discarded from cache, so note the use of the ‘sync’
          conversions in the examples below, which are used to maximize
          the effectiveness of the ‘nocache’ flag.

          Here are some usage examples:

               # Advise to drop cache for whole file
               dd if=ifile iflag=nocache count=0

               # Ensure drop cache for the whole file
               dd of=ofile oflag=nocache conv=notrunc,fdatasync count=0

               # Advise to drop cache for part of file
               # Note the kernel will only consider complete and
               # already persisted pages.
               dd if=ifile iflag=nocache skip=10 count=10 of=/dev/null

               # Stream data using just the read-ahead cache.
               # See also the ‘direct’ flag.
               dd if=ifile of=ofile iflag=nocache oflag=nocache,sync

     ‘nonblock’
          Use non-blocking I/O.

     ‘noatime’
          Do not update the file’s access timestamp.  *Note File
          timestamps::.  Some older file systems silently ignore this
          flag, so it is a good idea to test it on your files before
          relying on it.

     ‘noctty’
          Do not assign the file to be a controlling terminal for ‘dd’.
          This has no effect when the file is not a terminal.  On many
          hosts (e.g., GNU/Linux hosts), this flag has no effect at all.

     ‘nofollow’
          Do not follow symbolic links.

     ‘nolinks’
          Fail if the file has multiple hard links.

     ‘binary’
          Use binary I/O.  This flag has an effect only on nonstandard
          platforms that distinguish binary from text I/O.

     ‘text’
          Use text I/O.  Like ‘binary’, this flag has no effect on
          standard platforms.

     ‘fullblock’
          Accumulate full blocks from input.  The ‘read’ system call may
          return early if a full block is not available.  When that
          happens, continue calling ‘read’ to fill the remainder of the
          block.  This flag can be used only with ‘iflag’.  This flag is
          useful with pipes for example as they may return short reads.
          In that case, this flag is needed to ensure that a ‘count=’
          argument is interpreted as a block count rather than a count
          of read operations.

     These flags are all GNU extensions to POSIX. They are not supported
     on all systems, and ‘dd’ rejects attempts to use them when they are
     not supported.  When reading from standard input or writing to
     standard output, the ‘nofollow’ and ‘noctty’ flags should not be
     specified, and the other flags (e.g., ‘nonblock’) can affect how
     other processes behave with the affected file descriptors, even
     after ‘dd’ exits.

   The behavior of ‘dd’ is unspecified if operands other than ‘conv=’,
‘iflag=’, ‘oflag=’, and ‘status=’ are specified more than once.

   The numeric-valued strings above (N and BYTES) are unsigned decimal
integers that can be followed by a multiplier: ‘b’=512, ‘c’=1, ‘w’=2,
‘xM’=M, or any of the standard block size suffixes like ‘k’=1024 (*note
Block size::).  These multipliers are GNU extensions to POSIX, except
that POSIX allows BYTES to be followed by ‘k’, ‘b’, and ‘xM’.  Block
sizes (i.e., specified by BYTES strings) must be nonzero.

   Any block size you specify via ‘bs=’, ‘ibs=’, ‘obs=’, ‘cbs=’ should
not be too large – values larger than a few megabytes are generally
wasteful or (as in the gigabyte..exabyte case) downright
counterproductive or error-inducing.

   To process data with offset or size that is not a multiple of the I/O
block size, you can use a numeric string N that ends in the letter ‘B’.
For example, the following shell commands copy data in 1 MiB blocks
between a flash drive and a tape, but do not save or restore a 512-byte
area at the start of the flash drive:

     flash=/dev/sda
     tape=/dev/st0

     # Copy all but the initial 512 bytes from flash to tape.
     dd if=$flash iseek=512B bs=1MiB of=$tape

     # Copy from tape back to flash, leaving initial 512 bytes alone.
     dd if=$tape bs=1MiB of=$flash oseek=512B

   For failing storage devices, other tools come with a great variety of
extra functionality to ease the saving of as much data as possible
before the device finally dies, e.g.  GNU ‘ddrescue’
(https://www.gnu.org/software/ddrescue/).  However, in some cases such a
tool is not available or the administrator feels more comfortable with
the handling of ‘dd’.  As a simple rescue method, call ‘dd’ as shown in
the following example: the operand ‘conv=noerror,sync’ is used to
continue after read errors and to pad out bad reads with NULs, while
‘iflag=fullblock’ caters for short reads (which traditionally never
occur on flash or similar devices):

     # Rescue data from an (unmounted!) partition of a failing device.
     dd conv=noerror,sync iflag=fullblock  /mnt/rescue.img

   Sending an ‘INFO’ signal (or ‘USR1’ signal where that is unavailable)
to a running ‘dd’ process makes it print I/O statistics to standard
error and then resume copying.  In the example below, ‘dd’ is run in the
background to copy 5GB of data.  The ‘kill’ command makes it output
intermediate I/O statistics, and when ‘dd’ completes normally or is
killed by the ‘SIGINT’ signal, it outputs the final statistics.

     # Ignore the signal so we never inadvertently terminate the dd child.
     # Note this is not needed when SIGINFO is available.
     trap '' USR1

     # Run dd with the fullblock iflag to avoid short reads
     # which can be triggered by reception of signals.
     dd iflag=fullblock if=/dev/zero of=/dev/null count=5000000 bs=1000 & pid=$!

     # Output stats every second.
     while kill -s USR1 $pid 2>/dev/null; do sleep 1; done

   The above script will output in the following format:

     3441325+0 records in
     3441325+0 records out
     3441325000 bytes (3.4 GB, 3.2 GiB) copied, 1.00036 s, 3.4 GB/s
     5000000+0 records in
     5000000+0 records out
     5000000000 bytes (5.0 GB, 4.7 GiB) copied, 1.44433 s, 3.5 GB/s

   The ‘status=progress’ operand periodically updates the last line of
the transfer statistics above.

   On systems lacking the ‘INFO’ signal ‘dd’ responds to the ‘USR1’
signal instead, unless the ‘POSIXLY_CORRECT’ environment variable is
set.

   An exit status of zero indicates success, and a nonzero value
indicates failure.