A universal standard the whole ecosystem can depend on — a neural-net PDF for researchers to move networks between training frameworks and inferencing engines
The Khronos™ Group is about to release a new standard method of moving trained neural networks among frameworks, and between frameworks and inference engines. The new standard is the Neural Network Exchange Format (NNEF™); it has been in design for over a year and will be available to the public by the end of 2017.
The field of machine learning — especially using trained neural networks — has seen massive expansion in recent years driven by the availability of the huge data sets needed for successful implementations. This expansion has been accompanied by an equivalent rise in the number and variety of training frameworks, and we are on the cusp of a similar boom in hardware and software inference engines.
This diversity is a tremendous advantage for innovation but it comes at the cost of fragmentation. Moving networks among frameworks, and from training to inference, is hindered by the need to write converters, syntax checkers, and importers for the many-to-many mapping we see today. Already, we’re seeing point solutions coming into existence in acknowledgement of the problem, but what’s really needed is a universal standard that takes a broad view and provides a comprehensive, extensible, well supported solution that all parts of the ecosystem can depend on.
This is what NNEF is aiming to be: a “PDF for neural networks” that allows researchers to move networks between frameworks as their requirements change and they need access to new features. It it will allow network designers to easily and conveniently hand off their trained networks to implementation teams, whether they are targeting a CPU or GPU accelerated cloud environment or a low level, hardware accelerator at the edge. It’ll do this without imposing the need for the receiving environment to have detailed knowledge of the sender, and without imposing the limitations of the sender on the target environment.
NNEF will allow for the transmission of network structure alone, or structure plus weights and has facilities to include multiple alternate sets of weights in the same file structure. It uses a python-like language to describe well-defined data structures and operations, imposing strict type checking and limited, well-chosen functionality to make it easier to produce valid formats. All operations are defined without reference to the limitations of the originating framework, but with extensive guidance to the receiving parser so that details like quantization can be passed on for platform specific implementation and optimization.
The format is being designed to be hierarchical so that compact, human readable networks are easy to define, and also allows for a flat definition style that makes it easier to design lightweight, inline parsers that facilitate streaming implementations.
All Khronos standards are royalty-free and fully available for commercial use, regardless of membership in the organization. At launch, Khronos will provide a fully-functional parser and validator, along with a sample implementation of an inference engine as well as example exporters for a range of frameworks, including TensorFlow and Caffe. All these tools will be available as open source projects so that the machine learning community can take full advantage of them.
The success of Khronos standards is built on ecosystem and community involvement - they are standards built by the industry, for the industry. Khronos’ well proven standardisation process provides any company an equal voice in the evolution of this critical technology. We encourage the widest possible participation in standards definition through Khronos membership, our open source projects or participation in advisory panels that do not require taking out a fee-based membership. Learn more about Khronos participation here, and NNEF here.