All interprocess and device communication is provided in the caching model
by implementing it as an extension of the virtual memory system
using memory-based messaging [7].
With memory-based messaging,
threads communicate through the memory system by mapping a shared region
of physical memory into the sender and receiver address spaces,
as illustrated in Figure 3.
The sending thread writes a message into this region and then delivers the address of the new message to the receiving threads as an address-valued signal. That is, the virtual address corresponding to the location of the new message is passed to the receiving threads' signal function, translated from the virtual address of the sending thread (using the normal inverted page table support). On receiving the address-valued signal, the receiving thread reads the message at the designated location in the virtual memory region. While the thread is running in its signal function, additional signals are queued within the Cache Kernel.
To support memory-based messaging,
the page mappings described in the previous section are extended to
optionally specify a signal thread and also to specify that the page
is in message mode.
An application kernel interested in receiving signals for a given page
specifies a signal thread in the mapping for the page.
The signaling uses the same mapping data
structures as the rest of the virtual memory system. This extension
is simpler than the separate messaging mechanism for interprocess
communication that arises with other micro-kernels.
Also, the Cache Kernel is only involved in communication setup. The
performance-critical data transfer aspect of interprocess
communication is performed directly through the memory system.
Moreover, with suitable hardware support,
there is no software intervention even for signal
delivery
. Thus, communication performance
is limited primarily by the raw performance of the memory system, not
the software overhead of copying, queuing and delivering messages, as
arises with other micro-kernels.
Memory-based messaging is used for accessing devices controlled by the Cache Kernel. For example, the Ethernet device in our implementation is provided as memory-mapped transmission and reception memory regions. The client thread sends a signal to the Ethernet driver in the Cache Kernel to transmit a packet with the signal address indicating the packet buffer to transmit. On reception, a signal is generated to the receiving thread with the signal address indicating the buffer holding the new packet. This thread demultiplexes the data to the appropriate input stream, similar to conventional network protocol implementations.
Devices that fit into the memory-based messaging model directly require minimal driver code complexity of the Cache Kernel. They also provide the best performance. For example, our own network interface for a 266 Mb Fiber Channel interconnect is designed to fit into this memory-mapped model, and so requires relatively few (276) lines of code for the Cache Kernel driver. In particular, the driver only needs to support memory mapping the special device address space corresponding to the network interface. Data transfer and signaling is then handled using the general Cache Kernel memory-based messaging mechanism. The clock is also designed to fit this memory-based messaging model. In contrast, the Ethernet device requires a non-trivial Cache Kernel driver to implement the memory-based messaging interface because the Ethernet chip itself provides a conventional DMA interface.
An object-oriented RPC facility implemented on top of the memory-based messaging as a user-space communication library allows applications and services to use a conventional procedural communication interface to services. For instance, object writeback from the Cache Kernel to the owning application kernel uses a writeback channel implemented using this facility. This RPC facility is also used for high-performance communication between distributed application kernels, as described in Section 3. Memory-based messaging supports direct marshaling and demarshaling to and from communication channels with minimal copying and no protection boundary crossing in software. The implementation in user space allows application kernels control over communication resource management and exception handling, by, for example, overriding functions in the communication library.