The Schooner Perspective

Flash Memory Architecture Alternatives

Flash Memory Architecture Alternatives

Dr. John R Busch, CTO and Founder

Flash Chips: NOR vs NAND

Flash memory chips are constructed from different types of cells (NOR and NAND), and with different numbers of cells per memory location (single-level cell or SLC; and multi-level cell or MLC). These variations result in very different performance, cost, and reliability characteristics. NOR flash memory chips have much lower density, much lower bandwidth, much longer write and erase latencies, and much higher cost than NAND flash memory chips. For these reasons, NOR flash has minimal penetration in enterprise deployments; it is primarily used in consumer devices. Leading enterprise solid state drives (SSDs) are all designed with NAND flash.

Flash Chips: SLC vs MLC

Another distinction in flash memory is SLC versus MLC. MLC increases density by storing more than a single bit per memory cell. With their increased density, the cost of MLC flash chips is roughly half that of SLC, but the MLC write bandwidth is about 2 times worse than SLC, and MLC supports from 3 to 30 times fewer erase cycles than SLC. A new generation of SSDs incorporates special firmware that closes the performance and durability gap between SLC and MLC.

Flash Form Factor and Physical Interface: PCIe vs SSD, SATA/SAS

Flash memory can be installed into a server as PCIe flash or as SSDs. With PCIe flash the controller and flash chips are placed onto standard form factor PCIe cards that are plugged directly into the server’s PCIe slots. With SSDs, flash chips and controllers are placed into 2.5” or 1.5” cartridges which are installed into server hard disk drive slots and which interface through SATA or SAS controllers that are plugged into the server’s PCIe slots.

Because of the direct connection to PCIe, a single PCIe flash card has a lower latency and higher bandwidth than a single SSD (which is connected to PCIe through a controller card). However, in most workloads the latency difference is not significant, and any desired level of flash bandwidth can be achieved by using multiple SSDs. PCIe flash cards are significantly more expensive than SSDs on a $/GB basis.

In a typical 2U server, many more SSDs can be operated in parallel than PCIe cards. As a result a much higher total degree of flash parallelism, bandwidth, and capacity can be achieved through the use of parallel SSDs instead of PCIe flash memory subsystems. The SSD flash memory configuration can be adjusted to match workload capacity, bandwidth, and latency requirements with optimized controller/SSD configurations.

An SSD-based flash subsystem is also easier to maintain. It is much easier to replace an SSD than a PCIe flash memory card—somewhat similar to the difference between installing a memory stick into a USB port on a typical personal computer (PC) versus opening up a PC to install a graphics card.

An SSD flash subsystem allows hot swapping, resulting in lower downtime and higher serviceability than possible with a PCIe flash subsystem, since the latter requires a system to be taken out of service to add or replace a flash card.

Flash Space Management: Server-Based vs Device-Based

The flash memory management functions of write coalescing, space management, logical-to-physical mapping, wear leveling, and garbage collection require significant on-going computation and data movement. The first generation of enterprise-class flash technology was based on discrete logic, using FPGAs (field programmable gate arrays) for control. FPGAs have limited computational capability, so the flash interface to software was very low level, This forced the write coalescing, garbage collection, logical-to-physical mapping, and wear leveling all to be performed by special driver software executing in the server.

The new generation of enterprise flash contains advanced ASICs which provide the flash management functions very efficiently on the flash cards themselves, exploiting internal flash buses and device characteristics. This frees the server’s processor cores and DRAM for application use. This is very significant. For example, the first generation PCIe FPGA-based flash memory cards we evaluated perform these functions using the server’s resources. In our system-level benchmarking we measured that 25% of the server’s processor cores and 10 GB of the server DRAM were consumed for flash management overhead.

Selecting Flash Technology

The analysis of an appropriate flash subsystem configuration comes down to creating a balanced system for the target workload with required system uptime at the best price/performance. This requires workload characterization, system measurement, and TCO and availability modeling. Our Schooner Labs evaluation of flash technologies analyzes in this context.

3 Responses to “Flash Memory Architecture Alternatives”