Formats for storing Hi-C pairs

.pairs

.pairs is a simple tabular format for storing DNA contacts detected in a Hi-C experiment. The detailed .pairs specification is defined by the 4DN Consortium.

The body of a .pairs contains a table with a variable number of fields separated by a “\t” character (a horizontal tab). The .pairs specification fixes the content and the order of the first seven columns:

index name description
1 read_id the ID of the read as defined in fastq files
2 chrom1 the chromosome of the alignment on side 1
3 pos1 the 1-based genomic position of the outer-most (5’) mapped bp on side 1
4 chrom2 the chromosome of the alignment on side 2
5 pos2 the 1-based genomic position of the outer-most (5’) mapped bp on side 2
6 strand1 the strand of the alignment on side 1
7 strand2 the strand of the alignment on side 2

A .pairs file starts with a header, an arbitrary number of lines starting with a “#” character. By convention, the header lines have a format of “#field_name: field_value”. The .pairs specification mandates a few standard header lines (e.g., column names, chromosome order, sorting order, etc), all of which are automatically filled in by pairtools.

The entries of a .pairs file can be flipped and sorted. “Flipping” means that the sides 1 and 2 do not correspond to side1 and side2 in sequencing data. Instead, side1 is defined as the side with the alignment with a lower sorting index (using the lexographic order for chromosome names, followed by the numeric order for positions and the lexicographic order for pair types). This particular order of “flipping” is defined as “upper-triangular flipping”, or “triu-flipping”. Finally, pairs are typically block-sorted: i.e. first lexicographically by chrom1 and chrom2, then numerically by pos1 and pos2.

Pairtools’ flavor of .pairs

.pairs files produced by pairtools extend .pairs format in a few ways.

  1. pairtools store null, unmapped, ambiguous (multiply mapped) and chimeric (if not parsed by parse2 or –walks-policy all of parse) alignments as chrom=’!’, pos=0, strand=’-‘.
  2. pairtools store the header of the source .sam files in the ‘#samheader:’ fields of the pairs header. When multiple .pairs files are merged, the respective ‘#samheader:’ fields are checked for consistency and merged.
  3. Each pairtool applied to .pairs leaves a record in the ‘#samheader’ fields (using a @PG sam tag), thus preserving the full history of data processing.
  4. pairtools append an extra column describing the type of a Hi-C pair:
index name description
8 pair_type the type of a Hi-C pair

Pair types

pairtools use a simple two-character notation to define all possible pair types, according to the quality of alignment of the two sides. The type of a pair can be defined unambiguously using the table below. To use this table, identify which side has an alignment of a “poorer” quality (unmapped < multimapped < unique alignment) and which side has a “better” alignment and find the corresponding row in the table.

. . . Less informative alignment More informative alignment .
Pair type Code >2 alignments Mapped Unique Mapped Unique Sidedness
walk-walk WW 0 [1]
null NN     0
corrupt XX     0 [2]_
null-multi NM   0
null-rescued NR   1 [3]
null-unique NU   1
multi-multi MM 0
multi-rescued MR 1 [3]
multi-unique MU 1
rescued-unique RU 2 [3]
unique-rescued UR 2 [3]
unique-unique UU 2
duplicate DD 2 [4]_
[1]“walks”, or, C-walks are Hi-C molecules formed via multiple ligation events which cannot be reported as a single pair.
[2]“corrupt” pairs are those with technical issues - e.g. missing a FASTQ sequence/SAM entry from one side of the molecule.
[2]“rescued” pairs have two non-overlapping alignments on one of the sides (referred below as the chimeric side/read), but the inner (3’-) one extends the only alignment on the other side (referred as the non-chimeric side/read). Such pairs form when one of the two ligated DNA fragments is shorter than the read length. In this case, one of the reads contains this short fragment entirely, together with the ligation junction and a chunk of the other DNA fragment (thus, this read ends up having two non-overlapping alignments). Following the procedure introduced in HiC-Pro and Juicer, pairtools parse rescues such Hi-C molecules, reports the position of the 5’ alignment on the chimeric side, and tags them as “NU”, “MU”, “UR” or “RU” pair type, depending on the type of the 5’ alignment on the chimeric side. Such molecules can and should be used in downstream analysis. Read more on the rescue procedure in the section on parsing.
[3](1, 2, 3, 4) pairtools dedup detects molecules that could be formed via PCR duplication and tags them as “DD” pair type. These pairs should be excluded from downstream analyses.

.pairsam

pairtools also define .pairsam, a valid extension of the .pairs format. On top of the pairtools’ flavor of .pairs, .pairsam format adds two extra columns containing the alignments from which the Hi-C pair was extracted:

index name description
9 sam1 the sam alignment(s) on side 1; separate supplemental alignments by NEXT_SAM
10 sam2 the sam alignment(s) on side 2; separate supplemental alignments by NEXT_SAM

Note that, normally, the fields of a sam alignment are separated by a horizontal tab character (\t), which we already use to separate .pairs columns. To avoid confusion, we replace the tab character in sam entries stored in sam1 and sam2 columns with a UNIT SEPARATOR character (\031).

Finally, sam1 and sam2 can store multiple .sam alignments, separated by a string ‘\031NEXT_SAM\031’

Extra columns

pairtools can operate on .pairs/.pairsam with extra columns. Extra columns are specified in the order defined by the order their addition by various tools. Column names can be checked in the header of .pairs/.pairsam file. We provide pairtools header utilities for manipulating and verifying compatibility of headers and their columns.

The list of additional columns used throughout pairtools modules:

extra column generating module format how to add it description
mapq1, mapq2 parse/parse2 number from 0 to 255 pairtools parse –add-columns mapq Mapping quality, as reported in .sam/.bam, $-10 log_{10}(P_{error})$
pos51, pos52 parse/parse2 genomic coordinate pairtools parse –add-columns pos5 5’ position of alignment (closer to read start)
pos31, pos32 parse/parse2 genomic coordinate pairtools parse –add-columns pos3 3’ position of alignment (further from read start)
cigar1, cigar2 parse/parse2 string pairtools parse –add-columns cigar CIGAR, or Compact Idiosyncratic Gapped Alignment Report of alignment, as reported in .sam/.bam
read_len1, read_len2 parse/parse2 number pairtools parse –add-columns read_len read length
matched_bp1, matched_bp2 parse/parse2 number pairtools parse –add-columns matched_bp number of matched alignment basepairs to the reference
algn_ref_span1, algn_ref_span2 parse/parse2 number pairtools parse –add-columns algn_ref_span basepairs of reference covered by alignment
algn_read_span1, algn_read_span2 parse/parse2 number pairtools parse –add-columns algn_read_span basepairs of read covered by alignment
dist_to_51, dist_to_52 parse/parse2 number pairtools parse –add-columns dist_to_5 distance to 5’-end of read
dist_to_31, dist_to_32 parse/parse2 number pairtools parse –add-columns dist_to_3 distance to 3’-end of read
seq1, seq2 parse/parse2 string pairtools parse –add-columns seq sequence of alignment
mismatches1, mismatches2 parse/parse2 string pairtools parse –add-columns mismatches comma-separated list of mismatches relative to the reference, “{ref_letter}:{mut_letter}:{phred}:{ref_position}:{read_position}”
XB1/2,AS1/2,XS1/2 or any sam tag parse/parse2   pairtools parse –add-columns XA,XB,NM format depends on tag specification
walk_pair_type parse/parse2 string pairtools parse2 –add-pair-index Type of the pair relative to R1 and R2 reads of paired-end sequencing, see pasring docs
walk_pair_index parse/parse2 number pairtools parse2 –add-pair-index Order of the pair in the complex walk, starting from 5’-end of left read, see pasring docs
phase phase 0, 1 or “.” pairtools phase Phase of alignment (haplotype 1, 2, on unphased), see phasing walkthrough
rfrag1, rfrag2 restrict number pairtools restrict Unique index of the restriction fragment after annotating pairs positions, see restriction walkthrough
rfrag_start1, rfrag_start2 restrict number pairtools restrict Coordinate of the start of restriction fragment
rfrag_end1, rfrag_end2 restrict number pairtools restrict Coordinate of the end of restriction fragment