Merge pull request #1321 from trapexit/allowmmap

Add support for 'direct-io-allow-mmap' if supported by kernel

README.md

@@ -89,7 +89,12 @@ start with one of the following option sets.
 
 #### You need `mmap` (used by rtorrent and many sqlite3 base software)
 
-`cache.files=partial,dropcacheonclose=true,category.create=mfs`
+`cache.files=auto-full,dropcacheonclose=true,category.create=mfs`
+
+or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode
+that allows shared mmap when `cache.files=off`. To be sure of the best
+performance between `cache.files=off` and `cache.files=auto-full`
+you'll need to do your own benchmarking but often `off` is faster.
 
 #### You don't need `mmap`
 
@@ -97,11 +102,11 @@ start with one of the following option sets.
 
 ### Command Line
 
-`mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
+`mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
 
 ### /etc/fstab
 
-`/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0`
+`/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0`
 
 ### systemd mount
 
@@ -116,7 +121,7 @@ Type=simple
 KillMode=none
 ExecStart=/usr/bin/mergerfs \
   -f \
-  -o cache.files=partial \
+  -o cache.files=auto-full \
   -o dropcacheonclose=true \
   -o category.create=mfs \
   /mnt/hdd0:/mnt/hdd1 \
@@ -162,6 +167,11 @@ These options are the same regardless of whether you use them with the
   longer need the data and it can drop its cache. Recommended when
   **cache.files=partial|full|auto-full|per-process** to limit double
   caching. (default: false)
+* **direct-io-allow-mmap=BOOL**: On newer kernels (>= 6.6) it is
+  possible to disable file page caching while still allowing for
+  shared mmap support. mergerfs will enable this feature if available
+  but an option is provided to turn it off for testing and debugging
+  purposes. (default: true)
 * **symlinkify=BOOL**: When enabled and a file is not writable and its
   mtime or ctime is older than **symlinkify_timeout** files will be
   reported as symlinks to the original files. Please read more below
@@ -388,8 +398,8 @@ The above line will use all mount points in /mnt prefixed with **hdd** and **ssd
 To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
 
 ```
-# <file system>        <mount point>  <type>         <options>             <dump>  <pass>
-/mnt/hdd*:/mnt/ssd    /media          fuse.mergerfs  minfreespace=16G      0       0
+# <file system>        <mount point>  <type>    <options>             <dump>  <pass>
+/mnt/hdd*:/mnt/ssd    /media          mergerfs  minfreespace=16G      0       0
 ```
 
 **NOTE:** the globbing is done at mount or when updated using the
@@ -682,21 +692,13 @@ is empty is opened for writing.
 In theory this flag should not be exposed to the end user. It is a
 low-level FUSE flag which indicates whether or not the kernel can send
 certain kinds of messages to it for the purposes of using with
-NFS. mergerfs does support these messages but due to a possible bug in
-the Linux kernel this option needs to be configurable. As it turns out
-at least certain versions of NFS can still work when disabled and no
-longer triggers the bug.
-
-While it may be safe to default this setting to `false` in an
-abundance of caution it is being left enabled by default.
+NFS. mergerfs does support these messages but due bugs and quirks
+found in the kernel and mergerfs this option is provided just in case
+it is needed for debugging.
 
 Given that this flag is set when the FUSE connection is first
 initiated it is not possible to change during run time.
 
-See [Kernel Issues &
-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
-
 
 # FUNCTIONS, CATEGORIES and POLICIES
 
@@ -1178,8 +1180,8 @@ mount. If you plan on changing settings with the new mount you should
 ```
 $ sudo mount /mnt/mergerfs
 $ mount | grep mergerfs
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
 ```
 
 A problem with this approach is that the underlying instance will
@@ -1876,16 +1878,17 @@ more details.
 
 #### rtorrent fails with ENODEV (No such device)
 
-Be sure to set `cache.files=partial|full|auto-full|per-processe` or
-turn off `direct_io`. rtorrent and some other applications use
-[mmap](http://linux.die.net/man/2/mmap) to read and write to files and
-offer no fallback to traditional methods. FUSE does not currently
-support mmap while using `direct_io`. There may be a performance
-penalty on writes with `direct_io` off as well as the problem of
-double caching but it's the only way to get such applications to
-work. If the performance loss is too high for other apps you can mount
-mergerfs twice. Once with `direct_io` enabled and one without it. Be
-sure to set `dropcacheonclose=true` if not using `direct_io`.
+Be sure to set
+`cache.files=partial|full|auto-full|per-processe`. rtorrent and some
+other applications use [mmap](http://linux.die.net/man/2/mmap) to read
+and write to files and offer no fallback to traditional methods. FUSE
+does not currently support mmap while using `direct_io`. There may be
+a performance penalty on writes with `direct_io` off as well as the
+problem of double caching but it's the only way to get such
+applications to work. If the performance loss is too high for other
+apps you can mount mergerfs twice. Once with `direct_io` enabled and
+one without it. Be sure to set `dropcacheonclose=true` if not using
+`direct_io`.
 
 
 #### Plex doesn't work with mergerfs
@@ -2047,17 +2050,19 @@ remote filesystem on Unix/POSIX systems. Due to how NFS works there
 are some settings which need to be set in order for mergerfs to work
 with it.
 
-It should be noted that NFS and FUSE (the technology mergerfs uses)
-do not work perfectly with one another due to certain design
-choices in FUSE (and mergerfs.) Due to these issues it is
-generally recommended to use SMB when possible till situations
-change. That said issues should still be reported. NFS is not really
-recommended but it isn't unsupported.
+It should be noted that NFS and FUSE (the technology mergerfs uses) do
+not work perfectly with one another due to certain design choices in
+FUSE (and mergerfs.) Due to these issues it is generally recommended
+to use SMB when possible till situations change. That said mergerfs
+should generally work as an export of NFS and issues discovered should
+still be reported.
+
+To ensure compatibility between mergerfs and NFS use the following
+settings.
 
 mergerfs settings:
 * noforget
 * inodecalc=path-hash
-* export-support=false (only available in v2.40.0 and above)
 
 NFS export settings:
 * fsid=UUID
@@ -2080,32 +2085,17 @@ calculation algorithm then it is possible that if you changed a file
 or updated a directory the file mergerfs will use will be on a
 different branch and therefore the inode would change. This isn't an
 ideal solution and others are being considered but it works for most
-situations. It might be possible to leave the default value of
-`hybrid-hash` if `export-support=false` but it hasn't been fully
-tested yet.
-
-`export-support=false` is a low-level FUSE option that is needed as a
-workaround to a Linux kernel bug. Technically this option should be
-set to `true` when expecting to use mergerfs with NFS but it turns out
-that at least some versions of NFS work when not enabled. At the
-moment it is unclear if this has any negative side effects but when
-disabled NFS appears to work alright and the kernel no longer sends
-invalid requests. When the issue is resolved in the kernel these docs
-will be updated with appropriate details.
-
-Also see [Kernel Issues &
-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
+situations.
 
 `fsid=UUID` is needed because FUSE filesystems don't have different
 `st_dev` values which can cause issues when exporting. The easiest
 thing to do is set each mergerfs export `fsid` to some random
 value. An easy way to generate a random value is to use the command
-line tool `uuidgen` or through a website such as
+line tool `uuid` or `uuidgen` or through a website such as
 [uuidgenerator.net](https://www.uuidgenerator.net/).
 
 `no_root_squash` is not strictly necessary but can lead to confusing
-permission and ownership issues.
+permission and ownership issues if root squashing is enabled.
 
 
 ## SMB / CIFS

libfuse/include/fuse_common.h

@@ -108,23 +108,26 @@ struct fuse_file_info_t
  * FUSE_CAP_IOCTL_DIR: ioctl support on directories
  * FUSE_CAP_CACHE_SYMLINKS: cache READLINK responses
  */
-#define FUSE_CAP_ASYNC_READ        (1 << 0)
-#define FUSE_CAP_POSIX_LOCKS       (1 << 1)
-#define FUSE_CAP_ATOMIC_O_TRUNC    (1 << 3)
-#define FUSE_CAP_EXPORT_SUPPORT    (1 << 4)
-#define FUSE_CAP_BIG_WRITES        (1 << 5)
-#define FUSE_CAP_DONT_MASK         (1 << 6)
-#define FUSE_CAP_FLOCK_LOCKS       (1 << 10)
-#define FUSE_CAP_IOCTL_DIR         (1 << 11)
-#define FUSE_CAP_READDIR_PLUS      (1 << 13)
-#define FUSE_CAP_READDIR_PLUS_AUTO (1 << 14)
-#define FUSE_CAP_ASYNC_DIO         (1 << 15)
-#define FUSE_CAP_WRITEBACK_CACHE   (1 << 16)
-#define FUSE_CAP_PARALLEL_DIROPS   (1 << 18)
-#define FUSE_CAP_POSIX_ACL         (1 << 19)
-#define FUSE_CAP_CACHE_SYMLINKS    (1 << 20)
-#define FUSE_CAP_MAX_PAGES         (1 << 21)
-#define FUSE_CAP_SETXATTR_EXT      (1 << 22)
+#define FUSE_CAP_ASYNC_READ           (1ULL << 0)
+#define FUSE_CAP_POSIX_LOCKS          (1ULL << 1)
+#define FUSE_CAP_ATOMIC_O_TRUNC       (1ULL << 3)
+#define FUSE_CAP_EXPORT_SUPPORT       (1ULL << 4)
+#define FUSE_CAP_BIG_WRITES           (1ULL << 5)
+#define FUSE_CAP_DONT_MASK            (1ULL << 6)
+#define FUSE_CAP_FLOCK_LOCKS          (1ULL << 10)
+#define FUSE_CAP_IOCTL_DIR            (1ULL << 11)
+#define FUSE_CAP_READDIR_PLUS         (1ULL << 13)
+#define FUSE_CAP_READDIR_PLUS_AUTO    (1ULL << 14)
+#define FUSE_CAP_ASYNC_DIO            (1ULL << 15)
+#define FUSE_CAP_WRITEBACK_CACHE      (1ULL << 16)
+#define FUSE_CAP_PARALLEL_DIROPS      (1ULL << 18)
+#define FUSE_CAP_POSIX_ACL            (1ULL << 19)
+#define FUSE_CAP_CACHE_SYMLINKS       (1ULL << 20)
+#define FUSE_CAP_MAX_PAGES            (1ULL << 21)
+#define FUSE_CAP_SETXATTR_EXT         (1ULL << 22)
+#define FUSE_CAP_DIRECT_IO_ALLOW_MMAP (1ULL << 23)
+#define FUSE_CAP_CREATE_SUPP_GROUP    (1ULL << 24)
+
 
 /**
  * Ioctl flags
@@ -150,56 +153,17 @@ struct fuse_file_info_t
  * indicate the value requested by the filesystem.  The requested
  * value must usually be smaller than the indicated value.
  */
-struct fuse_conn_info {
-  /**
-   * Major version of the protocol (read-only)
-   */
+struct fuse_conn_info
+{
   unsigned proto_major;
-
-  /**
-   * Minor version of the protocol (read-only)
-   */
   unsigned proto_minor;
-
-  /**
-   * Maximum size of the write buffer
-   */
   unsigned max_write;
-
-  /**
-   * Maximum readahead
-   */
   unsigned max_readahead;
-
-  /**
-   * Capability flags, that the kernel supports
-   */
-  unsigned capable;
-
-  /**
-   * Capability flags, that the filesystem wants to enable
-   */
-  unsigned want;
-
-  /**
-   * Maximum number of backgrounded requests
-   */
+  uint64_t capable;
+  uint64_t want;
   unsigned max_background;
-
-  /**
-   * Kernel congestion threshold parameter
-   */
   unsigned congestion_threshold;
-
-  /**
-   * Max pages
-   */
   uint16_t max_pages;
-
-  /**
-   * For future use.
-   */
-  unsigned reserved[22];
 };
 
 struct fuse_session;

libfuse/lib/debug.c

@@ -158,9 +158,9 @@ open_flag_to_str(const uint64_t offset_)
 static
 const
 char*
-fuse_flag_to_str(const uint32_t offset_)
+fuse_flag_to_str(const uint64_t offset_)
 {
-  switch(1 << offset_)
+  switch(1ULL << offset_)
     {
       FUSE_INIT_FLAG_CASE(ASYNC_READ);
       FUSE_INIT_FLAG_CASE(POSIX_LOCKS);
@@ -189,6 +189,16 @@ fuse_flag_to_str(const uint32_t offset_)
       FUSE_INIT_FLAG_CASE(NO_OPENDIR_SUPPORT);
       FUSE_INIT_FLAG_CASE(EXPLICIT_INVAL_DATA);
       FUSE_INIT_FLAG_CASE(MAP_ALIGNMENT);
+      FUSE_INIT_FLAG_CASE(SUBMOUNTS);
+      FUSE_INIT_FLAG_CASE(HANDLE_KILLPRIV_V2);
+      FUSE_INIT_FLAG_CASE(SETXATTR_EXT);
+      FUSE_INIT_FLAG_CASE(INIT_EXT);
+      FUSE_INIT_FLAG_CASE(INIT_RESERVED);
+      FUSE_INIT_FLAG_CASE(SECURITY_CTX);
+      FUSE_INIT_FLAG_CASE(HAS_INODE_DAX);
+      FUSE_INIT_FLAG_CASE(CREATE_SUPP_GROUP);
+      FUSE_INIT_FLAG_CASE(HAS_EXPIRE_ONLY);
+      FUSE_INIT_FLAG_CASE(DIRECT_IO_ALLOW_MMAP);
     }
 
   return NULL;
@@ -200,7 +210,7 @@ static
 void
 debug_open_flags(const uint32_t flags_)
 {
-  fprintf(stderr,"%s,",open_accmode_to_str(flags_));
+  fprintf(stderr,"%s, ",open_accmode_to_str(flags_));
   for(int i = 0; i < (sizeof(flags_) * 8); i++)
     {
       const char *str;
@@ -212,7 +222,7 @@ debug_open_flags(const uint32_t flags_)
       if(str == NULL)
         continue;
 
-      fprintf(stderr,"%s,",str);
+      fprintf(stderr,"%s, ",str);
     }
 }
 
@@ -717,28 +727,31 @@ debug_fuse_fallocate_in(const void *arg_)
 void
 debug_fuse_init_in(const struct fuse_init_in *arg_)
 {
+  uint64_t flags;
+
+  flags = (((uint64_t)arg_->flags) | ((uint64_t)arg_->flags2) << 32);
   fprintf(g_OUTPUT,
           "FUSE_INIT_IN: "
           " major=%u;"
           " minor=%u;"
           " max_readahead=%u;"
-          " flags=0x%08X (",
+          " flags=0x%016lx (",
           arg_->major,
           arg_->minor,
           arg_->max_readahead,
-          arg_->flags);
-  for(uint64_t i = 0; i < (sizeof(arg_->flags)*8); i++)
+          flags);
+  for(uint64_t i = 0; i < (sizeof(flags)*8); i++)
     {
       const char *str;
 
-      if(!(arg_->flags & (1ULL << i)))
+      if(!(flags & (1ULL << i)))
         continue;
 
       str = fuse_flag_to_str(i);
       if(str == NULL)
         continue;
 
-      fprintf(g_OUTPUT,"%s,",str);
+      fprintf(g_OUTPUT,"%s, ",str);
     }
   fprintf(g_OUTPUT,")\n");
 }
@@ -748,7 +761,10 @@ debug_fuse_init_out(const uint64_t              unique_,
                     const struct fuse_init_out *arg_,
                     const uint64_t              argsize_)
 {
+  uint64_t flags;
   const struct fuse_init_out *arg = arg_;
+
+  flags = (((uint64_t)arg->flags) | ((uint64_t)arg->flags2) << 32);
   fprintf(g_OUTPUT,
           /* "unique=0x%016"PRIx64";" */
           /* " opcode=RESPONSE;" */
@@ -758,27 +774,27 @@ debug_fuse_init_out(const uint64_t              unique_,
           " major=%u;"
           " minor=%u;"
           " max_readahead=%u;"
-          " flags=0x%08X ("
+          " flags=0x%016lx ("
           ,
           /* unique_, */
           /* sizeof(struct fuse_out_header) + argsize_, */
           arg->major,
           arg->minor,
           arg->max_readahead,
-          arg->flags);
+          flags);
 
-  for(uint64_t i = 0; i < (sizeof(arg->flags)*8); i++)
+  for(uint64_t i = 0; i < (sizeof(flags)*8); i++)
     {
       const char *str;
 
-      if(!(arg->flags & (1ULL << i)))
+      if(!(flags & (1ULL << i)))
         continue;
 
       str = fuse_flag_to_str(i);
       if(str == NULL)
         continue;
 
-      fprintf(g_OUTPUT,"%s,",str);
+      fprintf(g_OUTPUT,"%s, ",str);
     }
 
   fprintf(g_OUTPUT,

libfuse/lib/fuse.c

@@ -1375,7 +1375,7 @@ set_path_info(struct fuse             *f,
     return -ENOMEM;
 
   e->ino        = node->nodeid;
-  e->generation = f->nodeid_gen.generation;
+  e->generation = ((e->ino == FUSE_ROOT_ID) ? 0 : f->nodeid_gen.generation);
 
   pthread_mutex_lock(&f->lock);
   update_stat(node,&e->attr);
@@ -1386,6 +1386,15 @@ set_path_info(struct fuse             *f,
   return 0;
 }
 
+/*
+  lookup requests only come in for FUSE_ROOT_ID when a "parent of
+  child of root node" request is made. This can happen when using
+  EXPORT_SUPPORT=true and a file handle is used to keep a reference to
+  a node which has been forgotten. Mostly a NFS concern but not
+  excluslively. Root node always has a nodeid of 1 and generation of
+  0. To ensure this set_path_info() explicitly ensures the root id has
+  a generation of 0.
+ */
 static
 int
 lookup_path(struct fuse             *f,

libfuse/lib/fuse_lowlevel.c

@@ -294,6 +294,11 @@ fuse_reply_entry(fuse_req_t                     req,
 
   fill_entry(&arg, e);
 
+#ifdef NDEBUG
+  // TODO: Add checks for cases where a node could be marked bad by
+  // the kernel.
+#endif
+
   return send_reply_ok(req, &arg, size);
 }
 
@@ -1092,6 +1097,11 @@ do_init(fuse_req_t             req,
   struct fuse_init_in *arg = (struct fuse_init_in *) &hdr_[1];
   struct fuse_ll *f = req->f;
   size_t bufsize = fuse_chan_bufsize(req->ch);
+  uint64_t inargflags;
+  uint64_t outargflags;
+
+  inargflags = 0;
+  outargflags = 0;
 
   if(f->debug)
     debug_fuse_init_in(arg);
@@ -1122,40 +1132,49 @@ do_init(fuse_req_t             req,
 
   if(arg->minor >= 6)
     {
+      inargflags = arg->flags;
+      if(inargflags & FUSE_INIT_EXT)
+        inargflags |= (((uint64_t)arg->flags2) << 32);
+
       if(arg->max_readahead < f->conn.max_readahead)
         f->conn.max_readahead = arg->max_readahead;
-      if(arg->flags & FUSE_ASYNC_READ)
+
+      if(inargflags & FUSE_ASYNC_READ)
         f->conn.capable |= FUSE_CAP_ASYNC_READ;
-      if(arg->flags & FUSE_POSIX_LOCKS)
+      if(inargflags & FUSE_POSIX_LOCKS)
         f->conn.capable |= FUSE_CAP_POSIX_LOCKS;
-      if(arg->flags & FUSE_ATOMIC_O_TRUNC)
+      if(inargflags & FUSE_ATOMIC_O_TRUNC)
         f->conn.capable |= FUSE_CAP_ATOMIC_O_TRUNC;
-      if(arg->flags & FUSE_EXPORT_SUPPORT)
+      if(inargflags & FUSE_EXPORT_SUPPORT)
         f->conn.capable |= FUSE_CAP_EXPORT_SUPPORT;
-      if(arg->flags & FUSE_BIG_WRITES)
+      if(inargflags & FUSE_BIG_WRITES)
         f->conn.capable |= FUSE_CAP_BIG_WRITES;
-      if(arg->flags & FUSE_DONT_MASK)
+      if(inargflags & FUSE_DONT_MASK)
         f->conn.capable |= FUSE_CAP_DONT_MASK;
-      if(arg->flags & FUSE_FLOCK_LOCKS)
+      if(inargflags & FUSE_FLOCK_LOCKS)
         f->conn.capable |= FUSE_CAP_FLOCK_LOCKS;
-      if(arg->flags & FUSE_POSIX_ACL)
+      if(inargflags & FUSE_POSIX_ACL)
         f->conn.capable |= FUSE_CAP_POSIX_ACL;
-      if(arg->flags & FUSE_CACHE_SYMLINKS)
+      if(inargflags & FUSE_CACHE_SYMLINKS)
         f->conn.capable |= FUSE_CAP_CACHE_SYMLINKS;
-      if(arg->flags & FUSE_ASYNC_DIO)
+      if(inargflags & FUSE_ASYNC_DIO)
         f->conn.capable |= FUSE_CAP_ASYNC_DIO;
-      if(arg->flags & FUSE_PARALLEL_DIROPS)
+      if(inargflags & FUSE_PARALLEL_DIROPS)
         f->conn.capable |= FUSE_CAP_PARALLEL_DIROPS;
-      if(arg->flags & FUSE_MAX_PAGES)
+      if(inargflags & FUSE_MAX_PAGES)
         f->conn.capable |= FUSE_CAP_MAX_PAGES;
-      if(arg->flags & FUSE_WRITEBACK_CACHE)
+      if(inargflags & FUSE_WRITEBACK_CACHE)
         f->conn.capable |= FUSE_CAP_WRITEBACK_CACHE;
-      if(arg->flags & FUSE_DO_READDIRPLUS)
+      if(inargflags & FUSE_DO_READDIRPLUS)
         f->conn.capable |= FUSE_CAP_READDIR_PLUS;
-      if(arg->flags & FUSE_READDIRPLUS_AUTO)
+      if(inargflags & FUSE_READDIRPLUS_AUTO)
         f->conn.capable |= FUSE_CAP_READDIR_PLUS_AUTO;
-      if(arg->flags & FUSE_SETXATTR_EXT)
+      if(inargflags & FUSE_SETXATTR_EXT)
         f->conn.capable |= FUSE_CAP_SETXATTR_EXT;
+      if(inargflags & FUSE_DIRECT_IO_ALLOW_MMAP)
+        f->conn.capable |= FUSE_CAP_DIRECT_IO_ALLOW_MMAP;
+      if(inargflags & FUSE_CREATE_SUPP_GROUP)
+        f->conn.capable |= FUSE_CAP_CREATE_SUPP_GROUP;
     }
   else
     {
@@ -1186,44 +1205,57 @@ do_init(fuse_req_t             req,
   if(f->op.init)
     f->op.init(f->userdata, &f->conn);
 
-  if((arg->flags & FUSE_MAX_PAGES) && (f->conn.want & FUSE_CAP_MAX_PAGES))
+  outargflags = outarg.flags;
+  if((inargflags & FUSE_MAX_PAGES) && (f->conn.want & FUSE_CAP_MAX_PAGES))
     {
-      outarg.flags     |= FUSE_MAX_PAGES;
+      outargflags     |= FUSE_MAX_PAGES;
       outarg.max_pages  = f->conn.max_pages;
 
       msgbuf_set_bufsize(outarg.max_pages + 1);
     }
 
   if(f->conn.want & FUSE_CAP_ASYNC_READ)
-    outarg.flags |= FUSE_ASYNC_READ;
+    outargflags |= FUSE_ASYNC_READ;
   if(f->conn.want & FUSE_CAP_POSIX_LOCKS)
-    outarg.flags |= FUSE_POSIX_LOCKS;
+    outargflags |= FUSE_POSIX_LOCKS;
   if(f->conn.want & FUSE_CAP_ATOMIC_O_TRUNC)
-    outarg.flags |= FUSE_ATOMIC_O_TRUNC;
+    outargflags |= FUSE_ATOMIC_O_TRUNC;
   if(f->conn.want & FUSE_CAP_EXPORT_SUPPORT)
-    outarg.flags |= FUSE_EXPORT_SUPPORT;
+    outargflags |= FUSE_EXPORT_SUPPORT;
   if(f->conn.want & FUSE_CAP_BIG_WRITES)
-    outarg.flags |= FUSE_BIG_WRITES;
+    outargflags |= FUSE_BIG_WRITES;
   if(f->conn.want & FUSE_CAP_DONT_MASK)
-    outarg.flags |= FUSE_DONT_MASK;
+    outargflags |= FUSE_DONT_MASK;
   if(f->conn.want & FUSE_CAP_FLOCK_LOCKS)
-    outarg.flags |= FUSE_FLOCK_LOCKS;
+    outargflags |= FUSE_FLOCK_LOCKS;
   if(f->conn.want & FUSE_CAP_POSIX_ACL)
-    outarg.flags |= FUSE_POSIX_ACL;
+    outargflags |= FUSE_POSIX_ACL;
   if(f->conn.want & FUSE_CAP_CACHE_SYMLINKS)
-    outarg.flags |= FUSE_CACHE_SYMLINKS;
+    outargflags |= FUSE_CACHE_SYMLINKS;
   if(f->conn.want & FUSE_CAP_ASYNC_DIO)
-    outarg.flags |= FUSE_ASYNC_DIO;
+    outargflags |= FUSE_ASYNC_DIO;
   if(f->conn.want & FUSE_CAP_PARALLEL_DIROPS)
-    outarg.flags |= FUSE_PARALLEL_DIROPS;
+    outargflags |= FUSE_PARALLEL_DIROPS;
   if(f->conn.want & FUSE_CAP_WRITEBACK_CACHE)
-    outarg.flags |= FUSE_WRITEBACK_CACHE;
+    outargflags |= FUSE_WRITEBACK_CACHE;
   if(f->conn.want & FUSE_CAP_READDIR_PLUS)
-    outarg.flags |= FUSE_DO_READDIRPLUS;
+    outargflags |= FUSE_DO_READDIRPLUS;
   if(f->conn.want & FUSE_CAP_READDIR_PLUS_AUTO)
-    outarg.flags |= FUSE_READDIRPLUS_AUTO;
+    outargflags |= FUSE_READDIRPLUS_AUTO;
   if(f->conn.want & FUSE_CAP_SETXATTR_EXT)
-    outarg.flags |= FUSE_SETXATTR_EXT;
+    outargflags |= FUSE_SETXATTR_EXT;
+  if(f->conn.want & FUSE_CAP_CREATE_SUPP_GROUP)
+    outargflags |= FUSE_CREATE_SUPP_GROUP;
+  if(f->conn.want & FUSE_CAP_DIRECT_IO_ALLOW_MMAP)
+    outargflags |= FUSE_DIRECT_IO_ALLOW_MMAP;
+
+  if(inargflags & FUSE_INIT_EXT)
+    {
+      outargflags |= FUSE_INIT_EXT;
+      outarg.flags2 = (outargflags >> 32);
+    }
+
+  outarg.flags = outargflags;
 
   outarg.max_readahead = f->conn.max_readahead;
   outarg.max_write = f->conn.max_write;
@@ -1242,6 +1274,9 @@ do_init(fuse_req_t             req,
       outarg.congestion_threshold = f->conn.congestion_threshold;
     }
 
+  if(f->conn.proto_minor >= 23)
+    outarg.time_gran = 1;
+
   size_t outargsize;
   if(arg->minor < 5)
     outargsize = FUSE_COMPAT_INIT_OUT_SIZE;

man/mergerfs.1

@@ -106,16 +106,22 @@ If you don\[cq]t already know that you have a special use case then just
 start with one of the following option sets.
 .SS You need \f[C]mmap\f[R] (used by rtorrent and many sqlite3 base software)
 .PP
-\f[C]cache.files=partial,dropcacheonclose=true,category.create=mfs\f[R]
+\f[C]cache.files=auto-full,dropcacheonclose=true,category.create=mfs\f[R]
+.PP
+or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode
+that allows shared mmap when \f[C]cache.files=off\f[R].
+To be sure of the best performance between \f[C]cache.files=off\f[R] and
+\f[C]cache.files=auto-full\f[R] you\[cq]ll need to do your own
+benchmarking but often \f[C]off\f[R] is faster.
 .SS You don\[cq]t need \f[C]mmap\f[R]
 .PP
 \f[C]cache.files=off,dropcacheonclose=true,category.create=mfs\f[R]
 .SS Command Line
 .PP
-\f[C]mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media\f[R]
+\f[C]mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media\f[R]
 .SS /etc/fstab
 .PP
-\f[C]/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0\f[R]
+\f[C]/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0\f[R]
 .SS systemd mount
 .PP
 https://github.com/trapexit/mergerfs/wiki/systemd
@@ -130,7 +136,7 @@ Type=simple
 KillMode=none
 ExecStart=/usr/bin/mergerfs \[rs]
   -f \[rs]
-  -o cache.files=partial \[rs]
+  -o cache.files=auto-full \[rs]
   -o dropcacheonclose=true \[rs]
   -o category.create=mfs \[rs]
   /mnt/hdd0:/mnt/hdd1 \[rs]
@@ -185,6 +191,13 @@ Recommended when
 caching.
 (default: false)
 .IP \[bu] 2
+\f[B]direct-io-allow-mmap=BOOL\f[R]: On newer kernels (>= 6.6) it is
+possible to disable file page caching while still allowing for shared
+mmap support.
+mergerfs will enable this feature if available but an option is provided
+to turn it off for testing and debugging purposes.
+(default: true)
+.IP \[bu] 2
 \f[B]symlinkify=BOOL\f[R]: When enabled and a file is not writable and
 its mtime or ctime is older than \f[B]symlinkify_timeout\f[R] files will
 be reported as symlinks to the original files.
@@ -516,8 +529,8 @@ tools use \f[B]/etc/fstab\f[R].
 .IP
 .nf
 \f[C]
-# <file system>        <mount point>  <type>         <options>             <dump>  <pass>
-/mnt/hdd*:/mnt/ssd    /media          fuse.mergerfs  minfreespace=16G      0       0
+# <file system>        <mount point>  <type>    <options>             <dump>  <pass>
+/mnt/hdd*:/mnt/ssd    /media          mergerfs  minfreespace=16G      0       0
 \f[R]
 .fi
 .PP
@@ -847,20 +860,12 @@ In theory this flag should not be exposed to the end user.
 It is a low-level FUSE flag which indicates whether or not the kernel
 can send certain kinds of messages to it for the purposes of using with
 NFS.
-mergerfs does support these messages but due to a possible bug in the
-Linux kernel this option needs to be configurable.
-As it turns out at least certain versions of NFS can still work when
-disabled and no longer triggers the bug.
-.PP
-While it may be safe to default this setting to \f[C]false\f[R] in an
-abundance of caution it is being left enabled by default.
+mergerfs does support these messages but due bugs and quirks found in
+the kernel and mergerfs this option is provided just in case it is
+needed for debugging.
 .PP
 Given that this flag is set when the FUSE connection is first initiated
 it is not possible to change during run time.
-.PP
-See Kernel Issues &
-Bugs (https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
 .SH FUNCTIONS, CATEGORIES and POLICIES
 .PP
 The POSIX filesystem API is made up of a number of functions.
@@ -1577,8 +1582,8 @@ apply those before mounting the new version.
 \f[C]
 $ sudo mount /mnt/mergerfs
 $ mount | grep mergerfs
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
-media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
+media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
 \f[R]
 .fi
 .PP
@@ -2404,8 +2409,8 @@ Take a look at the section on NFS in the #remote-filesystems for more
 details.
 .SS rtorrent fails with ENODEV (No such device)
 .PP
-Be sure to set \f[C]cache.files=partial|full|auto-full|per-processe\f[R]
-or turn off \f[C]direct_io\f[R].
+Be sure to set
+\f[C]cache.files=partial|full|auto-full|per-processe\f[R].
 rtorrent and some other applications use
 mmap (http://linux.die.net/man/2/mmap) to read and write to files and
 offer no fallback to traditional methods.
@@ -2588,11 +2593,13 @@ It should be noted that NFS and FUSE (the technology mergerfs uses) do
 not work perfectly with one another due to certain design choices in
 FUSE (and mergerfs.) Due to these issues it is generally recommended to
 use SMB when possible till situations change.
-That said issues should still be reported.
-NFS is not really recommended but it isn\[cq]t unsupported.
+That said mergerfs should generally work as an export of NFS and issues
+discovered should still be reported.
 .PP
-mergerfs settings: * noforget * inodecalc=path-hash *
-export-support=false (only available in v2.40.0 and above)
+To ensure compatibility between mergerfs and NFS use the following
+settings.
+.PP
+mergerfs settings: * noforget * inodecalc=path-hash
 .PP
 NFS export settings: * fsid=UUID * no_root_squash
 .PP
@@ -2618,35 +2625,17 @@ mergerfs will use will be on a different branch and therefore the inode
 would change.
 This isn\[cq]t an ideal solution and others are being considered but it
 works for most situations.
-It might be possible to leave the default value of \f[C]hybrid-hash\f[R]
-if \f[C]export-support=false\f[R] but it hasn\[cq]t been fully tested
-yet.
-.PP
-\f[C]export-support=false\f[R] is a low-level FUSE option that is needed
-as a workaround to a Linux kernel bug.
-Technically this option should be set to \f[C]true\f[R] when expecting
-to use mergerfs with NFS but it turns out that at least some versions of
-NFS work when not enabled.
-At the moment it is unclear if this has any negative side effects but
-when disabled NFS appears to work alright and the kernel no longer sends
-invalid requests.
-When the issue is resolved in the kernel these docs will be updated with
-appropriate details.
-.PP
-Also see Kernel Issues &
-Bugs (https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
-for more details.
 .PP
 \f[C]fsid=UUID\f[R] is needed because FUSE filesystems don\[cq]t have
 different \f[C]st_dev\f[R] values which can cause issues when exporting.
 The easiest thing to do is set each mergerfs export \f[C]fsid\f[R] to
 some random value.
 An easy way to generate a random value is to use the command line tool
-\f[C]uuidgen\f[R] or through a website such as
+\f[C]uuid\f[R] or \f[C]uuidgen\f[R] or through a website such as
 uuidgenerator.net (https://www.uuidgenerator.net/).
 .PP
 \f[C]no_root_squash\f[R] is not strictly necessary but can lead to
-confusing permission and ownership issues.
+confusing permission and ownership issues if root squashing is enabled.
 .SS SMB / CIFS
 .PP
 SMB (https://en.wikipedia.org/wiki/Server_Message_Block) is a protocol

src/config.cpp

@@ -58,6 +58,7 @@ namespace l
     IFERT("branches-mount-timeout");
     IFERT("cache.symlinks");
     IFERT("cache.writeback");
+    IFERT("direct-io-allow-mmap");
     IFERT("export-support");
     IFERT("fsname");
     IFERT("fuse_msg_size");
@@ -94,6 +95,7 @@ Config::Config()
     cache_symlinks(false),
     category(func),
     direct_io(false),
+    direct_io_allow_mmap(true),
     dropcacheonclose(false),
     export_support(true),
     flushonclose(FlushOnClose::ENUM::OPENED_FOR_WRITE),
@@ -150,6 +152,7 @@ Config::Config()
   _map["category.create"]        = &category.create;
   _map["category.search"]        = &category.search;
   _map["direct_io"]              = &direct_io;
+  _map["direct-io-allow-mmap"]   = &direct_io_allow_mmap;
   _map["dropcacheonclose"]       = &dropcacheonclose;
   _map["export-support"]         = &export_support;
   _map["flush-on-close"]         = &flushonclose;

src/config.hpp

@@ -117,6 +117,7 @@ public:
   ConfigBOOL     cache_symlinks;
   Categories     category;
   ConfigBOOL     direct_io;
+  ConfigBOOL     direct_io_allow_mmap;
   ConfigBOOL     dropcacheonclose;
   ConfigBOOL     export_support;
   FlushOnClose   flushonclose;

src/fuse_init.cpp

@@ -144,14 +144,15 @@ namespace FUSE
     l::want_if_capable(conn_,FUSE_CAP_ATOMIC_O_TRUNC);
     l::want_if_capable(conn_,FUSE_CAP_BIG_WRITES);
     l::want_if_capable(conn_,FUSE_CAP_CACHE_SYMLINKS,&cfg->cache_symlinks);
+    l::want_if_capable(conn_,FUSE_CAP_DIRECT_IO_ALLOW_MMAP,&cfg->direct_io_allow_mmap);
     l::want_if_capable(conn_,FUSE_CAP_DONT_MASK);
     l::want_if_capable(conn_,FUSE_CAP_EXPORT_SUPPORT,&cfg->export_support);
     l::want_if_capable(conn_,FUSE_CAP_IOCTL_DIR);
     l::want_if_capable(conn_,FUSE_CAP_PARALLEL_DIROPS);
     l::want_if_capable(conn_,FUSE_CAP_POSIX_ACL,&cfg->posix_acl);
     l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS,&cfg->readdirplus);
-    //    l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS_AUTO);
     l::want_if_capable(conn_,FUSE_CAP_WRITEBACK_CACHE,&cfg->writeback_cache);
+    //    l::want_if_capable(conn_,FUSE_CAP_READDIR_PLUS_AUTO);
     l::want_if_capable_max_pages(conn_,cfg);
     conn_->want &= ~FUSE_CAP_POSIX_LOCKS;
     conn_->want &= ~FUSE_CAP_FLOCK_LOCKS;

tools/preload.c

@@ -107,7 +107,7 @@ open(const char *pathname_,
   LOAD_FUNC(open);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -119,7 +119,7 @@ open(const char *pathname_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -155,7 +155,7 @@ open64(const char *pathname_,
   LOAD_FUNC(open64);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -167,7 +167,7 @@ open64(const char *pathname_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -204,7 +204,7 @@ openat(int         dirfd_,
   LOAD_FUNC(openat);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -216,7 +216,7 @@ openat(int         dirfd_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)
@@ -253,7 +253,7 @@ openat64(int         dirfd_,
   LOAD_FUNC(openat64);
 
   mode = 0;
-  if(flags_ & O_CREAT)
+  if(flags_ & (O_CREAT|O_TMPFILE))
     {
       va_list args;
       va_start(args,flags_);
@@ -265,7 +265,7 @@ openat64(int         dirfd_,
   if(fd == -1)
     return -1;
 
-  if(flags_ & (O_DIRECTORY|O_PATH))
+  if(flags_ & (O_DIRECTORY|O_PATH|O_TMPFILE))
     return fd;
   rv = fstat(fd,&st);
   if(rv == -1)