## 3.6-rc3 Ext4 Scalability Measurements

This page contains ext4 performance and scalability data collected from the upstream 3.6-rc3 kernel, and a comparative reference baseline from 3.2. To permit comparisons with other Linux filesystems, the same measurements are made on ext3, xfs, and btrfs. Generally, each filesystem's default mkfs and mount options are used with a few exceptions intended to make for a more useful comparison. These are mentioned in the description of the test system configuration below. The primary goal of this work is to evaluate some of ext4's scalability properties as a function of core and thread count in both journaled and non-journaled configurations.

The benchmarks used are modified versions of the Boxacle ffsb workloads, with adjustments to the original ffsb profiles to run 1 thread, 48 threads (one per CPU), and 192 threads (four per CPU), among other things, to better match them to the 48 core test system.

The data shown in the graphs linked below are from lockstat runs. We collect two sets of lockstat data, and use the one which most closely matched the results from three additional runs made without lockstats. These additional runs help assess the impact of lock statistics collection and run to run variability. Lock statistics collection has a significant effect on most workloads at one thread, depressing throughput by 10 to 20%. It has a much less significant effect at 48 or 192 threads, where the effect is usually indistinguishable from run to run variability.

### Test System Configuration

- 48 core/48 CPU x86-64 server with 256 GB of memory - HP ProLiant DL785 G6
- One PCIe-attached RAID controller with 512 MB battery-backed writeback cache - HP P800 SmartArray
- One 7 TB hardware RAID0 volume consisting of twenty four 3 Gb/sec SAS disks
- e2fsprogs v1.42 on 3.2; not recorded for 3.6-rc3
- xfsprogs v3.1.7 on 3.2; not recorded for 3.6-rc3
- btrfs-progs v0.19 on 3.2; not recorded for 3.6-rc3
- deadline I/O scheduler (best results for this combination of hardware and workloads)
- lazy_itable_init=0 mkfs option for ext4 and ext4noj to avoid initialization overhead at start of runs that distorts results (up to 20% throughput improvement)
- inode64 mount option for xfs. Originally suggested as better than the 3.1 inode32 default for a filesystem of this size, it lead to throughput improvements of more than 20% for a number of these workloads. It became the default for xfs on systems able to support 64 bit inodes (naturally including x86_64) in 3.7.

### large_file_creates Workload

In the 3.2 release, there was a large scalability improvement for this workload on journaled ext4. The improvement bisected to Wu Fengguang's IO-less balance_dirty_pages() patch (143dfe8611a63030ce0c79419dc362f7838be557 in the 3.2 merge window). jbd2 tracepoint data (3.1, 3.2) suggested that the patch forced all journal I/O into the background in more frequently and regularly issued but smaller batches and in groups of more or less adjacent inodes. Prior to 3.2, inodes found in a single batch of journal I/Os tended to be scattered. Pressure on jbd2's j_state_lock, previously thought to be the limiting factor for this workload, was greatly reduced. The writeback code is now the biggest source of contention for ext4 both with and without a journal. Although the long worst case wait time when write locking j_state_lock in the journaled case may be worth investigating from a latency perspective, improvement may not greatly improve scalability given the modest gain observed from running without a journal at all.

- Graphs

  Throughput and CPU Cost

- ffsb Output

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- Lock Stats

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- ffsb Profiles

  1 thread, 48 threads, 192 threads

### random_writes Workload

ffsb reports much higher throughput rates for this workload as compared to the others. We think this may be due to a relatively low amount of generated I/O relative to the amount of cache on the RAID controller. The test system also has a lot of memory for page cache, and that should tend to convert a signficant portion of the generated random I/O into more sequential write patterns than we might want.

- Graphs

  Throughput and CPU Cost

- ffsb Output

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- Lock Stats

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- ffsb Profiles

  1 thread, 48 threads, 192 threads

### large_file_sequential_reads Workload

The strong throughput numbers btrfs delivers for this workload appear to be a function of an internally set 4 MB read ahead value. If a similarly aggressive read ahead value is set for the block devices used for ext4 or xfs filesystems, overriding the 128 KB default, similar throughput levels can be obtained.

- Graphs

  Throughput and CPU Cost

- ffsb Output

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- Lock Stats

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- ffsb Profiles

  1 thread, 48 threads, 192 threads

### random_reads Workload

This workload appears to be limited by the throughput of the storage system, so it doesn't reveal anything particularly interesting about ext4.

- Graphs

  Throughput and CPU Cost

- ffsb Output

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- Lock Stats

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- ffsb Profiles

  1 thread, 48 threads, 192 threads

### mail_server Workload

The mail server workload performs both reads and writes during execution. The read and write throughput results are broken out in separate graphs. Because it's not possible to isolate the CPU utilization for the read operations from the write operations, aggregate utilization (cost) is reported in a single graph rather than in separate graphs for the read and write cases.

Unlike the other workloads, the mail server workload historically (3.2 and earlier) tended to exhibit a lot of run to run variability. However, the 3.6-rc3 data collected don't show this, and it's also not particularly evident when comparing 3.6-rc3 data with the 3.2 results.

- Graphs

  Throughput and CPU Cost

- ffsb Output

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- Lock Stats

  1 thread - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  1 thread - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  48 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.2: ext4, ext4 nojournal, ext3, xfs, btrfs
  192 threads - 3.6-rc3: ext4, ext4 nojournal, ext3, xfs, btrfs

- ffsb Profiles

  1 thread, 48 threads, 192 threads