Data parallelism and statistical methods lie at the core of all DNA sequencing workloads because the physical process of decoding DNA is error-prone. These errors and uncertainties of errors are effectively mitigated by sequencing the same pieces of DNA many times over and calculating what is referred to as a “quality score”. This is an indication of how likely any given data point may be an error. At a very high level, the general process working with sequenced DNA can be broken down into three steps each with their own storage and compute requirements.
A standard genomics pipeline involves converting BCL files, written from sequencer to a Windows client. This process coverts raw data into millions of short reads or base calls. The data often lands on Network Attached Storage with limited, if any, benefit to running this on a parallel file system. Typical runs for a NGS system are in the 10’s to 100’s of MB/s and can last for a day or two.
Once the base calls are generated, they are mapped to a reference genome. This is referred to as the alignment process and determines how the base calls fit together. Variants, or reads that do not match are identified in a process called variant calling and saved in a separate dataset. This step requires significant capacity both in-flight and post processing. The amount of temporary capacity ranges between 8 and 10 times its input, and output data is typically 2 -3 times the size of the original dataset. This step scales linearly with cores and while the dataset is robust, does not take advantage of parallel IO.
Once the variant calling process is complete, the real science begins, taking the called variants and correlating those differences with scientific theory. The process often involves cross-referencing the called variants with other known variants or other analysis. Depending on the code and libraries used, the process takes significant advantage of cluster resources and high performance parallel file systems.
]]>A burst buffer consists of a combination of rapidly accessed persistent memory with its own processing power. The persistent memory is positioned between a set of processors with their non-persistent memory counterparts and a chunk of symmetric multi-processor compute through direct PCIe high bandwidth links. The back end of the burst buffer is comprised of slower large capacity storage systems.
In other words: Would you like to add a hyper-fast storage tier between the compute nodes and your parallel file system? Then you have come to the right place.
A little more detail on the specifics. The burst buffer’s purpose is to allow applications running on an SMP’s fast processor cores to perceive that the application data – data first residing in the SMP’s local and volatile memory – will be quickly saved on some persistent media. As far as the application is concerned, its data, once written into the burst buffer, had become persistent with a very low latency; the application did not need to wait long to learn that its data had been saved. If the power goes off after the completion of such a write, the data is assumed to be available for subsequent use.
Where would I use a Burst Buffer?
- Periodic bursts – use to flatten the latency curve and reduce IO spikes causing latency.
- Staging-in – Data that is utilized multiple times of large files can be staged in at the beginning of a job to maximize performance.
- Staging-in –application data can be staged at the beginning of a job to maximize performance.
- I/O improvements – to provide better IO for high IO applications.
Who is developing a Burst Buffer?
Within the HPC community, several vendors have released and are continuing to develop burst buffers including DDN, EMC and NetApp. The diagram below represents NERSC’s approach to creating a burst buffer architecture out of Cray DataWarp appliances. In total 1.5PB of burst buffer will front end over 100PB of capacity. The following is a conceptual diagram depicting the deployment.
The challenge in using the burst buffer at a smaller scale, say a DDN IME device front-ending GPFS-based GridScaler or a Lustre-based ExaScaler is to identify workloads and integrate scripts to take advantage of the buffer space that make sense.
Why Quobyte?
The value in Quobyte is that it’s a storage system of a new generation. Quobyte offers a solution with unified storage, simplified operations, and a complete set of management tools. It’s a software-defined storage solution that separates hardware from software. Quobyte makes data independent from the individual storage devices. Decoupling of logical and physical levels simplifies the management of infrastructure and enables the usage of hardware resources in a much more flexible way. It is virtualization-ready and has ultimate scalability on a per disk/per node with as dense of a footprint as you can get from the latest commodity servers.
As it is a full-featured POSIX file system, all UNIX applications can share the same storage – databases (like MySQL), email (like dovecot), VMs, etc. Interfaces for Hadoop, S3, NFS, and OpenStack are included. It supports parallel, sequential and small block random I/O workloads with near-hardware performance which keeps virtualization costs low. Because all data is available on any server or VM (shared storage), no extra NFS or CIFS servers are needed. Management and reconfiguration run transparently in the background.
Operations are simplified as Quobyte can manage all storage devices with one system. All applications share the same resources which enables oversubscription, dynamic reassignment and improves utilization. It decouples storage from host hardware and devices, with full split brain-safe fault tolerance, fast automatic failover and end-to-end checksums. Virtualization features yield low-touch operations because individual hardware failures are not relevant for the overall operation. If needed, broken servers and hard disks can replaced when convenient (even days and weeks later, given enough spare capacity) and there’s fast parallel automatic regeneration of any most or corrupt replica. There are simply no strings attached: switch off any server at any time! Because Quobyte runs on shared Linux servers, there are no special hardware or hardware redundancy features required and no RAID setup as plain formatted disks are used.
Along with the system comes a complete set of management tools:
- Full system scrub that verifies checksums
- Background rebalancer
- Backup interface that integrates with any file-based backup solution
- Convenient access via web console
- Integrated monitoring and alerting
- Fully programmable via JSON API and command line tools
To scale-out Quobyte appliances, you can mix and match nodes. For performance: 2 GB/s, NVMe and SSN disk options with NVMe/SSD for metadata. For general purpose: SAS disk options with SSD for metadata. For Archiving: NL-SAS disk options in dense chassis.
Here is an example of the underlying hardware architecture – hybrid:
Here is an example of the underlying hardware architecture – All-flash:
