.\" Copyright (c) 2017 Rick Macklem .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd July 2, 2017 .Dt PNFS 4 .Os .Sh NAME .Nm pNFS .Nd NFS Version 4.1 Parallel NFS Protocol .Sh DESCRIPTION The NFSv4.1 client and server provides support for the .Tn pNFS specification; see .%T "Network File System (NFS) Version 4 Minor Version 1 Protocol RFC 5661" . A pNFS service separates the Read/Write operations from all other NFSv4.1 operations, which are referred to as Metadata operations. The Read/Write operations are performed directly on the Data Server (DS) where the file's data resides, bypassing the NFS server. All other operations are performed on the NFS server, which is referred to as a Metadata Server (MDS). NFS clients that do not support .Tn pNFS perform Read/Write operations on the MDS, which acts as a proxy for the appropriate DS. .Pp A FreeBSD pNFS service consists of a single MDS server plus one or more DS servers, all of which are FreeBSD systems. .Tn pNFS clients mount the MDS as they would a single NFS server. .Pp A .Tn pNFS client must be running the .Xr nfscbd 8 daemon and use the mount options ''nfsv4,minorversion=1,pnfs''. .Pp A pNFS DS server must be configured as a NFSv4.1 server, where there is an exported directory with subdirectories named ds0, ds1, ..., ds created in it. For example, if the exported directory is /ds and the number of subdirectories is 20, the subdirectories are named /ds/ds0, /ds/ds1, ..., /ds/ds19. This exported directory is the one that the MDS will mount via NFSv4.1 to use as a DS. The subdirectories are created simply to reduce the size of the directories by spreading the data storage files across them. If the .Tn pNFS service will be storing a large number of files, the service should be configured with a large number of subdirectories. There really is no disadvantage in having a large number of subdirectories, so sysadmins should err on the side of creating many of them. Each of these subdirectories must be owned by the that the -maproot .Xr exports 5 option maps to, since the MDS accesses these directories as . These directories should have file mode 0700, so that only the mapped for has access to them. See .Xr exports 5 for more information on this. These subdirectories must be created by the sysadmin on all DS servers before the .Tn pNFS service is started. .Pp The sysctl .sp .Bd -literal -offset indent vfs.nfsd.dsdirsize .Ed .Pp defines the number of subdirectories named ds0, ds1, ... , ds, where N is vfs.nfsd.dsdirsize - 1, with the default set to 20. The number of subdirectories can be increased after the server has been running, but only when the .Xr nfsd 8 daemon is not running. .Pp When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size zero. Each of these files will also have two extended attributes in the system attribute name space: .sp .Bd -literal -offset indent pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file on a DS for this file. pnfsd.dsattr - This extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file. .Ed .Pp For each regular (VREG) file, the MDS creates a data storage file on one of the DSs, in one of the ds subdirectories of the exported DS directory. The name of this file is the file handle of the file on the MDS in hexadecimal at time of creation. .Pp For .Tn pNFS clients, the service generates File Layout layouts and associated DeviceInfo. For NFS clients that do not support NFSv4.1 pNFS, there will be a performance hit, since the I/O RPCs will be proxied by the MDS for the DS server the data storage file resides on. .Pp Configuration of a DS is done exactly as any other NFS server is configured, with the data storage directory exported to the MDS. .Pp The MDS is configured to mount the data storage directories of the DSs. For example, if there are 2 DSs named nfsv4-ds0 and nfsv4-ds1 and both of these have a /ds directory exported to the MDS, the .Xr fstab 5 entries might be: .sp .Bd -literal -offset indent nfsv4-ds0:/ds /ds0 nfs rw,nfsv4,minorversion=1 0 0 nfsv4-ds1:/ds /ds1 nfs rw,nfsv4,minorversion=1 0 0 .Ed .Pp The MDS will require a "-p" flag option specifying the DSs. For the above mounts, the nfs_server_flags entry in .Xr rc.conf 5 might be: .sp .Bd -literal -offset indent nfs_server_flags="-u -t -n 256 -p nfsv4-ds0:/ds0,nfsv4-ds1:/ds1" .Ed .Pp Note that the "-p" flag requires the "mounted-on" directory path on the MDS. .Sh SEE ALSO .Xr pnfsdsfile 1 , .Xr nfsv4 4 , .Xr exports 5 , .Xr fstab 5 , .Xr rc.conf 5 , .Xr nfscbd 8 , .Xr nfsd 8 , .Xr nfsuserd 8 .Sh BUGS At this time, there is no support for DS mirroring. As such, the MDS plus all DS servers are single points of failure for the .Tn pNFS service.