Overview

Despite continued innovations in disk design, input/output performance has not kept pace with concurrent increases in processor speeds. Much research has focused on developing algorithms to avoid input/output or hide input/output latency in an attempt to redress this widening gap. Many studies have shown that with advance knowledge of access patterns, file systems can improve input/output performance by selecting policies appropriate for the resource demands. Unfortunately, access patterns may be complex or data dependent, and therefore unknown a priori.

Our thesis is that the file system can automatically detect qualitative file access patterns both locally (per parallel program thread) and globally (per parallel program) and use this information to dynamically choose appropriate file system policies. We propose two complementary methods for automatic classification, based on neural networks and hidden Markov models, respectively. Global classifications are created from a combination of local classifications and additional access pattern information. We map qualititative classifications and quantitative statistics to file system policies shown to improve performance for those access patterns.

We have implemented this classification framework as extensions to the Portable Parallel File System (PPFS) testbed. Experimental results on sequential and parallel scientific applications demonstrate the utility of this approach.

More Information