The Archon Genomics X Prize, presented by Express Scripts, is a 10 million dollar competition to establish highly accurate clinical grade sequencing and variation detection methods. Our group at Harvard School of Public Health works with the EdgeBio team on developing the infrastructure for the competition: identify variations in the grading genomes and provide software to compare these reference variation sets against a competitor's list of variations.

The exciting aspect of the Genomics X Prize is that it enables open comparisons between sequencing technologies and variant calling methodologies. Sequencing genomes to the high degree of accuracy sufficient for clinical usage is a difficult, open, problem. Here I'll present detailed numbers comparing variants called by different sequencing technologies and variant callers.

The public phase of the Genomics X Prize starts today, August 15th. The goal of this six month period is to have an open dialog with everyone working in the sequencing and variant calling communities. We want to refine our methods to provide the most accurate and fair variant calling for the reference genomes. To start the discussion we've prepared:

• Variant calls for a HapMap individual: NA19239, a Yoruba male from Ibadan, Nigeria. We sequenced haploid fosmids from NA19239 with two technologies: Illumina and SOLiD; and called variants with three different methods: GATK Unified Genotyper, FreeBayes, and SAMtools mpileup. We combined these calls into a unified final call set, NA19239 version 0.1, that I discuss in detail below.
• Fully documented methods, access to all data files used, and a public scoring site. The X Prize Validation wiki contains detailed information about sequencing, variant calling, validation and scoring. The validationprotocol.org website provides a simple way for anyone to compare their variant calls against the public reference genomes. It encourages submission and analysis in public tools like Galaxy through transparent interoperability with GenomeSpace.
• An automated variant analysis infrastructure built on top of the Broad's Genome Analysis Toolkit (GATK) that performs comparisons as well as variant unification. This is a generally useful toolkit of functionality to manipulate variants, and we presented an overview at the Bioinformatics Open Source Conference last month. This is an open-source community developed project, and has received great contributions from the Genome in a Bottle Consortium at the National Institute of Standards and Technology.

The goal of this writeup, and the X Prize public phase, is to iterate over calling and unification methods to improve our algorithms and approaches. Rather than promoting or disparaging any particular technology or calling method, we're instead providing full transparency and a good-faith effort to combining approaches. Our hope is that this will help engage the community, encourage feedback, and result in a unbiased and accurate set of reference genomes for the competition.

For the August 15th public phase kickoff, we prepared a reference data set of NA19239 based on pooled sequencing of haploid fosmid clones. The callable regions of these clones totaled 129,513,026 total bases, covering ~4% of the 3.1 billion bases in the human genome. We use fosmid clones to obtain complete regional haplotype coverage and focus on partial genome coverage to achieve high coverage depth and accuracy for assessed regions.

Version 0.1 of the NA19239 reference set uses variant calls from two technologies: Illumina and SOLiD; and three callers: GATK's Unified Genotyper, FreeBayes and SAMtools. To move from these data to a unified call set we:

• Align to GRCh37 reference genome with Novoalign.
• Perform post-processing and indel realignment with GATK's IndelRealigner.
• Perform variant calling with GATK's UnifiedGenotyper, FreeBayes and samtools mpileup.
• Do pairwise comparisons between all technology/caller approaches.
• Generate the union of all possible calls and merge with initial GATK calls, recalling any no-call positions at expected sites.
• Use validation information on variants found in multiple technologies, plus metrics associated with common variants, to filter the full call set to a final set of trusted calls.

The challenging decisions begin when merging and filtering the final call set. This requires careful bookkeeping and variant representation to ensure identical variants are directly comparable, followed by setting cutoffs for variant inclusion.

To assess the quality of the unified call set, we compared to two public genomes:

• Complete Genomics's NA19239 variants from their public whole genome datasets,
• An Illumina whole genome dataset for NA19239 at 30x coverage.

This provides us with three independent call sets to assess variability between approaches. To provide a baseline, here is the comparison of the Illumina and Complete Genomics calls in our assessment regions:

We see familiar discordance rates: 13% of the Illumina calls and 7% of the Complete Genomics calls differ. Since it's diploid versus diploid, this comparison includes all heterozygous variant matches. As a result the numbers in this comparison will be higher, but it is a good order of magnitude approximation for looking at our fosmid reference set versus each individual technology.

The initial NA19239 public genome for the Genomics X Prize provides unified variant calls based on two sequencing technologies and three calling methods. I've delved into a lot of details on our approaches, challenges and goals with the hopes of encouraging suggestions from other researchers working on these problems. We're especially interested in feedback on these areas of ongoing research:

• Digging deeper into potential false positives and negatives: By combining comparison information between the unified callset and external resources, we can identify 17654 fosmid variants (10%) not found in both the Complete Genomics and Illumina datasets. These require additional in-depth analysis to classify as uniquely identified fosmid calls or potential false positives. Similarly, Illumina and Complete Genomics combine to call 1228 variants (0.7%) that are not in the fosmid call set. These need examination to classify as fosmid false negatives, or false positive calls in the individual technologies.
• Additional public genomes: We're actively working with teams like the Genome in a Bottle Consortium and Genome Research Consortium to compare with their reference sets and approaches. Our next target public genome is NA12878, used in both of these projects and widely studied.
• Improve variant representation and assessment: The variation software framework works hard to make variant representations as uniform as possible. Indels are especially challenging and we welcome practical examples of regions that need additional standardization.
• Refine approaches to unifying variant calls: What we learn from the additional inspection of discordant variants can help inform improved approaches to filtering. This is a great opportunity to develop generalized, reusable methods for combining variants from multiple approaches.

The call sets used here are available as public data folders on GenomeSpace:

• Public/chapmanb/xprize/NA19239-v0_1 – The combined final call set along with training true/false positives and Illumina/Complete Genomics comparison based potential false positives and negatives.
• Public/EdgeBio/PublicData/Release1 – All of the raw input data, including fastq files, BAM alignments and individual variant calls.

Combined with the open source code and configurations, we hope this will provided interested researchers with all the raw materials needed to reproduce and extend these analyses. Your feedback and suggestions are very welcome.