![]()
Storage Media Output FormatGeneralThis document describes the media format written by the Storage daemon. The Storage daemon reads and writes in units of blocks. Blocks contain records. Each block has a block header followed by records, and each record has a record header followed by record data.This chapter is intended to be a technical discussion of the Media Format and as such is not targeted at end users but rather at developers and system administrators that want or need to know more of the working details of Bacula. Definitions
Storage Daemon File Output FormatThe file storage and tape storage formats are identical except that tape records are by default blocked into blocks of 64,512 bytes, except for the last block, which is the actual number of bytes written rounded up to a multiple of 1024 whereas the last record of file storage is not rounded up. The default block size of 64,512 bytes may be overridden by the user (some older tape drives only support block sizes of 32K). Each Session written to tape is terminated with an End of File mark (this will be removed later). Sessions written to file are simply appended to the end of the file.Overall FormatA Bacula output file consists of Blocks of data. Each block contains a block header followed by records. Each record consists of a record header followed by the record data. The first record on a tape will always be the Volume Label Record. Wherever possible, the last record on the tape will be an End of Volume Label Record (not yet implemented -- version 1.26) . However this is not guaranteed as some tapes do not permit writing data after the End of Tape condition is signaled. Bacula will never attempt to read past an End of Volume Label.No Record Header will be split across Bacula blocks. However, Record Data may be split across any number of Bacula blocks. Obviously this will not be the case for the Volume Label which will always be smaller than the Bacula Block size. To simplify reading tapes, the Start of Session (SOS) and End of Session (EOS) records are never split across blocks. If this is about to happen, Bacula will write a short block before writing the session record (actually, the SOS record should always be the first record in a block, excepting perhaps the Volume label). Due to hardware limitations, the last block written to the tape may not be fully written. *NOT YET IMPLEMENTED! In this case, Bacula will attempt to write an additional block containing only the End of Volume Label followed by an End of File mark. Due to hardware limitations, the End of Volume Label and the End of File mark may not be written. When a new tape is mounted Bacula will write the full contents of the partially written block to the new tape ensuring that there is no loss of data. When reading a tape, Bacula will discard any block that is not totally written, thus ensuring that there is no duplication of data. In addition, since Bacula blocks are sequentially numbered within a Job, it is easy to ensure that no block is missing. SerializationAll Block Headers, Record Headers, and Label Records are written using Bacula's serialization routines. These routines guarantee that the data is written to the output volume in a machine independent format.Block HeaderThe format of the Block Header (version 1.27 and later) is:uint32_t CheckSum; /* Block check sum */ uint32_t BlockSize; /* Block byte size including the header */ uint32_t BlockNumber; /* Block number */ char ID[4] = "BB02"; /* Identification and block level */ uint32_t VolSessionId; /* Session Id for Job */ uint32_t VolSessionTime; /* Session Time for Job */The format of the Block Header (version 1.26 and earlier) is: uint32_t CheckSum; /* Block check sum */ uint32_t BlockSize; /* Block byte size including the header */ uint32_t BlockNumber; /* Block number */ char ID[4] = "BB01"; /* Identification and block level */The Block header is a fixed length and fixed format and is followed by Record Headers and Record Data. The CheckSum field is a 32 bit checksum of the block data and the block header but not including the CheckSum field. The Block Header is always immediately followed by a Record Header. If the tape is damaged, a Bacula utility will be able to recover as much information as possible from the tape by recovering blocks which are valid. The Block header is written using the Bacula serialization routines and thus is guaranteed to be in machine independent format. See below for version 2 of the block header. Record HeaderEach binary data record is preceded by a Record Header. The Record Header is fixed length and fixed format, whereas the binary data record is of variable length. The Record Header is written using the Bacula serialization routines and thus is guaranteed to be in machine independent format.The format of the Record Header (version 1.27 or later) is: int32_t FileIndex; /* File index supplied by File daemon */ int32_t Stream; /* Stream number supplied by File daemon */ uint32_t DataSize; /* size of following data record in bytes */The format of the Record Header (version 1.26 or earlier) is: uint32_t VolSessionId; /* Unique ID for this session */ uint32_t VolSessionTime; /* Start time/date of session */ int32_t FileIndex; /* File index supplied by File daemon */ int32_t Stream; /* Stream number supplied by File daemon */ uint32_t DataSize; /* size of following data record in bytes */This record is followed by the binary Stream data of DataSize bytes, followed by another Record Header record and the binary stream data. For the definitive definition of this record, see record.h in the src/stored directory. Additional notes on the above:
Version BB02 Block HeaderThe existing block header BB01 was designed to hold records from multiple sessions. However, it is simpler (and probably more efficient) for each session (Job) to have its own private block. As a consequence, the SessionId and SessionTime can be written once in each Block Header and not in the Record Header. So, the second version of the Block Header is:uint32_t CheckSum; /* Block check sum */ uint32_t BlockSize; /* Block byte size including the header */ uint32_t BlockNumber; /* Block number */ char ID[4] = "BB02"; /* Identification and block level */ uint32_t VolSessionId; /* Applies to all records */ uint32_t VolSessionTime; /* contained in this block */As with the previous version, the BB02 Block header is a fixed length and fixed format and is followed by Record Headers and Record Data. The CheckSum field is a 32 bit checksum of the block data and the block header but not including the CheckSum field. The Block Header is always immediately followed by a Record Header. If the tape is damaged, a Bacula utility will be able to recover as much information as possible from the tape by recovering blocks which are valid. The Block header is written using the Bacula serialization routines and thus is guaranteed to be in machine independent format.
Version 2 Record HeaderVersion 2 Record Header is written to the medium when using Version BB02 Block Headers. The memory representation of the record is identical to the BB01 Record Header, but on the storage medium, the first two fields, namely VolSessionId and VolSessionTime will not be written. The Block Header will be filled with these values when the First user record is written (i.e. non label record) so that when the block is written, it will have the current and unique VolSessionId and VolSessionTime. On reading each record from the Block, the VolSessionId and VolSessionTime will be filled in the Record Header from the Block Header.Volume Label FormatTape volume labels are created by the Storage daemon in response to a label command given to the Console program, or alternatively by the btape program. created. Each volume is labeled with the following information using the Bacula serialization routines, which guarantee machine byte order independence.For Bacula versions 1.27 and later, the Volume Label Format is: char Id[32]; /* Bacula 1.0 Immortal\n */ uint32_t VerNum; /* Label version number */ /* VerNum 11 and greater Bacula 1.27 and later */ btime_t label_btime; /* Time/date tape labeled */ btime_t write_btime; /* Time/date tape first written */ /* The following are 0 in VerNum 11 and greater */ float64_t write_date; /* Date this label written */ float64_t write_time; /* Time this label written */ char VolName[128]; /* Volume name */ char PrevVolName[128]; /* Previous Volume Name */ char PoolName[128]; /* Pool name */ char PoolType[128]; /* Pool type */ char MediaType[128]; /* Type of this media */ char HostName[128]; /* Host name of writing computer */ char LabelProg[32]; /* Label program name */ char ProgVersion[32]; /* Program version */ char ProgDate[32]; /* Program build date/time */For Bacula versions 1.26 and earlier, the Volume Label is: char Id[32]; /* Bacula 0.9 mortal\n */ uint32_t VerNum; /* Label version number */ float64_t label_date; /* Date tape labeled */ float64_t label_time; /* Time tape labeled */ float64_t write_date; /* Date this label written */ float64_t write_time; /* Time this label written */ char VolName[128]; /* Volume name */ char PrevVolName[128]; /* Previous Volume Name */ char PoolName[128]; /* Pool name */ char PoolType[128]; /* Pool type */ char MediaType[128]; /* Type of this media */ char HostName[128]; /* Host name of writing computer */ char LabelProg[32]; /* Label program name */ char ProgVersion[32]; /* Program version */ char ProgDate[32]; /* Program build date/time */Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label, ...) is stored in the record FileIndex field of the Record Header and does not appear in the data part of the record. Session LabelThe Session Label is written at the beginning and end of each session as well as the last record on the physical medium. It has the following binary format:char Id[32]; /* Bacula Immortal ... */ uint32_t VerNum; /* Label version number */ uint32_t JobId; /* Job id */ uint32_t VolumeIndex; /* sequence no of vol */ /* Prior to VerNum 11 */ float64_t write_date; /* Date this label written */ /* VerNum 11 and greater */ btime_t write_btime; /* time/date record written */ /* The following is zero VerNum 11 and greater */ float64_t write_time; /* Time this label written */ char PoolName[128]; /* Pool name */ char PoolType[128]; /* Pool type */ char JobName[128]; /* base Job name */ char ClientName[128]; /* Added in VerNum 10 */ char Job[128]; /* Unique Job name */ char FileSetName[128]; /* FileSet name */ uint32_t JobType; uint32_t JobLevel;In addition, the EOS label contains: /* The remainder are part of EOS label only */ uint32_t JobFiles; uint64_t JobBytes; uint32_t start_block; uint32_t end_block; uint32_t start_file; uint32_t end_file; uint32_t JobErrors;In addition, for VerNum greater than 10, the EOS label contains (in addition to the above): uint32_t JobStatus /* Job termination code */: Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label, ...) is stored in the record FileIndex field and does not appear in the data part of the record. Also, the Stream field of the Record Header contains the JobId. This permits quick filtering without actually reading all the session data in many cases. Overall Storage FormatCurrent Bacula Tape Format 6 June 2001 Version BB02 added 28 September 2002 A Bacula tape is composed of tape Blocks. Each block has a Block header followed by the block data. Block Data consists of Records. Records consist of Record Headers followed by Record Data. :=======================================================: | | | Block Header (16 bytes) | | (24 bytes version BB02) | |-------------------------------------------------------| | | | Record Header (20 bytes) | | (12 bytes version BB02) | |-------------------------------------------------------| | | | Record Data | | | |-------------------------------------------------------| | | | Record Header (20 bytes) | | (12 bytes version BB02) | |-------------------------------------------------------| | | | ... | Block Header: the first item in each block. The format is shown below. Partial Data block: occurs if the data from a previous block spills over to this block (the normal case except for the first block on a tape). However, this partial data block is always preceded by a record header. Record Header: identifies the Volume Session, the Stream and the following Record Data size. See below for format. Record data: arbitrary binary data. Block Header Format BB01 :=======================================================: | CheckSum (uint32_t) | |-------------------------------------------------------| | BlockSize (uint32_t) | |-------------------------------------------------------| | BlockNumber (uint32_t) | |-------------------------------------------------------| | "BB01" (char [4]) | :=======================================================: BBO1: Serves to identify the block as a Bacula block and also servers as a block format identifier should we ever need to change the format. Block Header Format BB02 :=======================================================: | CheckSum (uint32_t) | |-------------------------------------------------------| | BlockSize (uint32_t) | |-------------------------------------------------------| | BlockNumber (uint32_t) | |-------------------------------------------------------| | "BB02" (char [4]) | |-------------------------------------------------------| | VolSessionId (uint32_t) | |-------------------------------------------------------| | VolSessionTime (uint32_t) | :=======================================================: BBO2: Serves to identify the block as a Bacula block and also servers as a block format identifier should we ever need to change the format. BlockSize: is the size in bytes of the block. When reading back a block, if the BlockSize does not agree with the actual size read, Bacula discards the block. CheckSum: a checksum for the Block. BlockNumber: is the sequential block number on the tape. VolSessionId: a unique sequential number that is assigned by the Storage Daemon to a particular Job. This number is sequential since the start of execution of the daemon. VolSessionTime: the time/date that the current execution of the Storage Daemon started. It assures that the combination of VolSessionId and VolSessionTime is unique for all jobs written to the tape, even if there was a machine crash between two writes. Record Header Format BB01 :=======================================================: | VolSessionId (uint32_t) | |-------------------------------------------------------| | VolSessionTime (uint32_t) | |-------------------------------------------------------| | FileIndex (int32_t) | |-------------------------------------------------------| | Stream (int32_t) | |-------------------------------------------------------| | DataSize (uint32_t) | :=======================================================: Record Header Format BB02 :=======================================================: | FileIndex (int32_t) | |-------------------------------------------------------| | Stream (int32_t) | |-------------------------------------------------------| | DataSize (uint32_t) | :=======================================================: VolSessionId: a unique sequential number that is assigned by the Storage Daemon to a particular Job. This number is sequential since the start of execution of the daemon. VolSessionTime: the time/date that the current execution of the Storage Daemon started. It assures that the combination of VolSessionId and VolSessionTime is unique for all jobs written to the tape, even if there was a machine crash between two writes. FileIndex: a sequential file number within a job. The Storage daemon enforces this index to be greater than zero and sequential. Note, however, that the File daemon may send multiple Streams for the same FileIndex. The Storage Daemon uses negative FileIndices to identify Session Start and End labels as well as the End of Volume labels. Stream: defined by the File daemon and is intended to be used to identify separate parts of the data saved for each file (attributes, file data, ...). The Storage Daemon has no idea of what a Stream is or what it contains. DataSize: the size in bytes of the binary data record that follows the Session Record header. The Storage Daemon has no idea of the actual contents of the binary data record. For standard Unix files, the data record typically contains the file attributes or the file data. For a sparse file (not yet implemented), the binary data record my have a sub-header which indicates the storage address (probably 64 bits) for the data block. Volume Label :=======================================================: | Id (32 bytes) | |-------------------------------------------------------| | VerNum (uint32_t) | |-------------------------------------------------------| | label_date (float64_t) | | label_btime (btime_t VerNum 11 | |-------------------------------------------------------| | label_time (float64_t) | | write_btime (btime_t VerNum 11 | |-------------------------------------------------------| | write_date (float64_t) | | 0 (float64_t) VerNum 11 | |-------------------------------------------------------| | write_time (float64_t) | | 0 (float64_t) VerNum 11 | |-------------------------------------------------------| | VolName (128 bytes) | |-------------------------------------------------------| | PrevVolName (128 bytes) | |-------------------------------------------------------| | PoolName (128 bytes) | |-------------------------------------------------------| | PoolType (128 bytes) | |-------------------------------------------------------| | MediaType (128 bytes) | |-------------------------------------------------------| | HostName (128 bytes) | |-------------------------------------------------------| | LabelProg (32 bytes) | |-------------------------------------------------------| | ProgVersion (32 bytes) | |-------------------------------------------------------| | ProgDate (32 bytes) | |-------------------------------------------------------| :=======================================================: Id: 32 byte Bacula identifier "Bacula 1.0 immortal\n" (old version also recognized:) Id: 32 byte Bacula identifier "Bacula 0.9 mortal\n" LabelType (Saved in the FileIndex of the Header record). PRE_LABEL -1 Volume label on unwritten tape VOL_LABEL -2 Volume label after tape written EOM_LABEL -3 Label at EOM (not currently implemented) SOS_LABEL -4 Start of Session label (format given below) EOS_LABEL -5 End of Session label (format given below) VerNum: 11 label_date: Julian day tape labeled label_time: Julian time tape labeled write_date: Julian date tape first used (data written) write_time: Julian time tape first used (data written) VolName: "Physical" Volume name PrevVolName: The VolName of the previous tape (if this tape is a continuation of the previous one). PoolName: Pool Name PoolType: Pool Type MediaType: Media Type HostName: Name of host that is first writing the tape LabelProg: Name of the program that labeled the tape ProgVersion: Version of the label program ProgDate: Date Label program built Session Label :=======================================================: | Id (32 bytes) | |-------------------------------------------------------| | VerNum (uint32_t) | |-------------------------------------------------------| | JobId (uint32_t) | |-------------------------------------------------------| | write_btime (btime_t) VerNum 11 | | *write_date (float64_t) VerNum 10 | |-------------------------------------------------------| | 0 (float64_t) VerNum 11 | | *write_time (float64_t) VerNum 10 | |-------------------------------------------------------| | PoolName (128 bytes) | |-------------------------------------------------------| | PoolType (128 bytes) | |-------------------------------------------------------| | JobName (128 bytes) | |-------------------------------------------------------| | ClientName (128 bytes) | |-------------------------------------------------------| | Job (128 bytes) | |-------------------------------------------------------| | FileSetName (128 bytes) | |-------------------------------------------------------| | JobType (uint32_t) | |-------------------------------------------------------| | JobLevel (uint32_t) | |-------------------------------------------------------| | FileSetMD5 (50 bytes) VerNum 11 | |-------------------------------------------------------| Additional fields in End Of Session Label |-------------------------------------------------------| | JobFiles (uint32_t) | |-------------------------------------------------------| | JobBytes (uint32_t) | |-------------------------------------------------------| | start_block (uint32_t) | |-------------------------------------------------------| | end_block (uint32_t) | |-------------------------------------------------------| | start_file (uint32_t) | |-------------------------------------------------------| | end_file (uint32_t) | |-------------------------------------------------------| | JobErrors (uint32_t) | |-------------------------------------------------------| | JobStatus (uint32_t) VerNum 11 | :=======================================================: * => fields deprecated Id: 32 byte Bacula Identifier "Bacula 1.0 immortal\n" LabelType (in FileIndex field of Header): EOM_LABEL -3 Label at EOM SOS_LABEL -4 Start of Session label EOS_LABEL -5 End of Session label VerNum: 11 JobId: JobId write_btime: Bacula time/date this tape record written write_date: Julian date tape this record written - deprecated write_time: Julian time tape this record written - deprecated. PoolName: Pool Name PoolType: Pool Type MediaType: Media Type ClientName: Name of File daemon or Client writing this session Not used for EOM_LABEL. Unix File AttributesThe Unix File Attributes packet consists of the following: <File-Index> <Type> <Filename>@<File-Attributes>@<Link> @<Extended-Attributes@> where
The File-attributes consist of the following:
|